robots.txt Checker

Fetch a robots.txt and see exactly which crawlers can reach a given path, plus every sitemap it declares.

Live Fetches and analyzes a real URL on our servers.

Result

Results appear here Enter your input and hit Analyze. We fetch live data, read-only.

Test any path against Googlebot and AI bots

How it works, what it checks and why it matters.

  1. 1Enter a URL

    A domain or page address. No install, nothing to add to your site.

  2. 2We fetch it live

    Our servers request the page read-only, the way a crawler would.

  3. 3Read and fix

    Get the results in seconds, with what each one means.

Why it matters

Blocking the wrong bot hides you

Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those AI answers. Blocking GPTBot or ClaudeBot only opts you out of training.

robots.txt doesn’t remove pages

Google says a URL blocked by robots.txt can still be indexed from links, just without its content. Use noindex to keep a page out.

Specific groups override *

A crawler follows only the most specific User-agent group that names it. A bot with its own group ignores your * rules entirely.

What it checks

File and location
Fetches /robots.txt from the site’s origin and counts its lines.
Sitemaps
Every Sitemap: line the file declares.
Search bots
Googlebot and Bingbot, tested against your path.
AI search and user bots
OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot.
AI training bots
GPTBot, ClaudeBot and Google-Extended.
Deciding rule
The exact rule behind each result, using RFC 9309 matching: the longest rule wins and Allow wins a tie.

Questions

Does Google-Extended affect AI Overviews?

No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls use for Gemini training and grounding outside Search.

Do AI bots obey robots.txt?

The search crawlers do. OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and Perplexity-User, because a person asked for the page. Anthropic says all three of its bots honour it.

Why is a path “allowed” with no rule?

If no rule in the matching group covers the path, it’s allowed by default. The same applies when no group matches the bot at all.

Is it free?

Yes. Every tool is free with no signup.

Do you store what I enter?

Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.

How is this different from the site audit?

This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.

More in Crawling & indexing

Tools that pair well with this one

All tools