How to Check Whether a Site Blocks AI Crawlers in robots.txt

To check whether a site blocks AI crawlers, fetch its /robots.txt and work out which rule group each crawler obeys. Doing that by eye gets messy once a file has several groups, wildcards and exceptions. The robots.txt checker does it for one site or a list: each site comes back as Open to all, Blocks AI crawlers, Blocks search engines, No robots.txt or Unreachable, with the rule that decides for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others.

The distinction that matters, and the one people miss, is between crawlers that train AI models and crawlers that let a site appear in AI search answers. They are different tokens, and you can treat them differently.

How a crawler reads robots.txt

RFC 9309 describes the rules that checkers and crawlers follow:

• The file is split into groups, each starting with one or more User-agent lines.
• A crawler obeys every group that names its product token (matching is case-insensitive). It falls back to the * group only when no group names it.
• Within the groups that apply, the longest matching Allow or Disallow path wins; on a tie, Allow wins.
• * matches any characters and a trailing $ anchors the end of the path.
• A missing file (a 4xx answer) means everything may be fetched. A 5xx answer or no answer means crawlers should assume everything is disallowed until it comes back.

That second bullet is why this goes wrong by eye. A file with User-agent: * followed by Disallow: /private/ and a separate User-agent: GPTBot group with Disallow: / blocks GPTBot completely, and GPTBot ignores the * group.

Step 1: enter the site

Open the robots.txt checker and paste one website, or a list, one per line: up to 100 a run, or 500 after a free Google sign-in. The tool fetches each site's /robots.txt over HTTPS (plain HTTP if HTTPS doesn't answer) and follows up to five redirects.

Step 2: read the verdict and the per-crawler rows

For each crawler you see Allowed or Blocked, with the rule and group that decided it, plus any Sitemap lines in the file. The crawlers it reads:

Token	Operator	What it is for
GPTBot	OpenAI	Training
OAI-SearchBot	OpenAI	ChatGPT search results
ClaudeBot	Anthropic	Crawling for Anthropic
Claude-SearchBot	Anthropic	Search
PerplexityBot	Perplexity	Perplexity's search index
CCBot	Common Crawl	The Common Crawl dataset
Google-Extended	Google	A control token for Gemini, not a crawler
Applebot-Extended	Apple	A control token for Apple's AI, not a crawler
Googlebot, Bingbot	Search engines	Search indexing

The operators document their tokens: see OpenAI's crawler overview, Google's common crawlers, Anthropic's page on its crawler and Perplexity's crawler docs. Tokens can change, and crawlers that aren't listed aren't shown.

Step 3: separate training from search

This is the decision the file encodes, so look at it deliberately:

• Blocking a training token such as GPTBot says you don't want new crawls used to train a model.
• Blocking a search token such as OAI-SearchBot decides whether your site can be fetched for, and linked from, that product's search answers.

A publisher that wants to appear in AI answers but not in training sets would allow the search tokens and block the training ones. The checker's verdicts show where a site stands, so you can see whether an old blanket Disallow: / for * or a copied block list has shut out more than you meant to.

Step 4: catch accidental blocks

The most common serious find has nothing to do with AI: a stray Disallow: / left over from a staging site that blocks Googlebot after launch. The checker flags it as Blocks search engines. Run the check after every site launch or migration, together with the sitemap checker to see that the sitemap listed in the file loads.

Step 5: check a whole list

Agencies auditing a client portfolio and researchers measuring how many sites in a list block AI crawlers can paste up to 500 sites per run when signed in, then download a CSV with one column per crawler. For more than 500, run batches and download after each.

What robots.txt can't do

Be careful what you conclude from the file:

• It is a request, not a lock. Well-behaved crawlers follow it; others don't have to. To keep content private, use authentication.
• It doesn't remove indexed pages. A blocked page can still be indexed from links elsewhere. Use noindex for that.
• The checker tests the home page path only. Rules for other paths are listed under each group, so you can see what else is blocked.
• It reads robots.txt only, not meta robots tags, X-Robots-Tag headers or firewall rules that block crawlers by IP.
• It reads up to 500 KiB, the minimum size RFC 9309 says crawlers must parse. Rules after that are ignored, as crawlers may do.

If you're checking sites you plan to contact

If you are checking websites as part of lead research, a robots.txt result says something about the site's attitude to crawlers, not about whether the people behind it can be emailed. Check the contact addresses separately on the verify page. See also check if a domain accepts email.

Check your own site

Paste your domain into the robots.txt checker and read the per-crawler rows. If you also have an address list from the same research, the first 100 email checks on the verify page are free with a Google sign-in, no card; each address comes back valid, risky or invalid with a reason. See pricing for larger lists.