robots.txt Generator

Block the paths you want, point crawlers at your sitemap, and choose whether to block AI training bots, without blocking AI search.

Instant Runs in your browser. Instant, no quota.

Result

Results appear here Fill in the fields and hit Generate. It runs in your browser.

Generate a robots.txt that doesn’t hide you from AI

How it works, what it checks and why it matters.

  1. 1Fill in the fields

    Or hit “Try example” to see how it works.

  2. 2Generate instantly

    Everything runs in your browser. Nothing is sent to a server.

  3. 3Copy and ship

    Copy the output straight into your site.

Why it matters

Training and search are separate bots

Blocking GPTBot, ClaudeBot or Google-Extended keeps you out of training data. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from AI answers. This generator only blocks the first kind.

Point crawlers at your sitemap

A Sitemap: line helps Google, Bing and other crawlers find the URLs you want indexed.

State your AI preferences

Content Signals let you say search=yes, ai-input=yes, ai-train=no in plain terms. It’s a preference, not a block, and about 4% of top domains already publish one.

What it checks

Disallow rules
One Disallow line per path you enter, under User-agent: *.
Training-bot block
Optional groups that block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.
AI search left open
Search and user bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot get no group of their own, so they follow your general rules.
Content Signals
An optional Content-Signal line with your search, ai-input and ai-train choices.
Sitemap
A Sitemap: line with your sitemap URL.

Questions

Does robots.txt stop a page being indexed?

No, it stops crawling. A blocked URL can still appear in results if other pages link to it. Use noindex instead, and leave the page crawlable so Google can see the tag.

Will blocking GPTBot remove me from ChatGPT?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which this generator leaves on your general rules.

Do I need a separate group for Googlebot?

No. Googlebot follows the User-agent: * group unless you give it its own. A bot with its own group ignores the * rules entirely, so repeat any rules it needs.

Is it free?

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

More in Generators

Tools that pair well with this one

All tools