Robots.txt checker: which search and AI crawlers may this site's pages be read by?
In short: Paste websites and see, for each, whether robots.txt lets Googlebot, Bingbot and the main AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot...) fetch the home page, plus its Sitemap lines. Limit: It tests the home page path only; other paths are listed, not tested.
A robots.txt checker fetches a site's /robots.txt and works out what each crawler may fetch. Paste one site or up to 100 (500 after a Google sign-in): each gets Open to all, Blocks AI crawlers, Blocks search engines, No robots.txt or Unreachable, with the rule that decides for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others, matched as RFC 9309 says.
Price: this tool is free. Checking a whole email list is from $9/mo, and the first 100 checks are free with Google, no card. See pricing
Sample result
globex.example lets Googlebot in but blocks GPTBot, ClaudeBot and CCBot.
- Website
- globex.example
- Googlebot
- Allowed
- GPTBot
- Disallow: /
What this means for you
Know which search engines and AI crawlers each site lets in, and the rule that decides.
- Googlebot, Bingbot and 8 AI tokens
- Longest-match rule shown for each
- Sitemap lines and crawl-delay read
- 100 sites a run, 500 when signed in
- robots.txt
- AI crawlers
- GPTBot
By SimpleVerifier Team · Last updated
Checking a whole list of sites?
- Over 500 sites? Run them in batches of 500 when signed in and download the CSV after each.
- Check that each site's sitemap loads with the sitemap checker.
- Emailing the owners of these sites? Check the list at /verify. The first 100 email checks are free with a Google sign-in, no card.
How do I check a site's robots.txt for AI crawlers?
- Paste one website or a list, one per line: up to 100 a run, or 500 after a free Google sign-in.
- Press Check robots.txt. Each site's /robots.txt is fetched over HTTPS (plain HTTP if HTTPS doesn't answer), following up to five redirects.
- Read the verdict and, for each crawler, Allowed or Blocked with the rule and group that decided it, plus any Sitemap lines.
- Download the CSV with one column per crawler.
Example
- You enter
- Three websites:
acme.example,globex.exampleandinitech.example. - You get
acme.exampleOpen to all: no group blocks /.globex.exampleBlocks AI crawlers: GPTBot, ClaudeBot and CCBot are disallowed by their own groups, Googlebot and OAI-SearchBot are allowed.initech.exampleNo robots.txt (404), so every crawler may fetch everything. Each crawler's deciding rule is in the CSV.- Limits
- Up to 100 websites a run without an account and 500 when signed in with Google. The home page path is tested.
How does a crawler read robots.txt?
As RFC 9309 describes: the file is split into groups, each starting with one or more User-agent lines. A crawler obeys every group naming its product token (case-insensitive) and only falls back to the * group when none does. Within those groups the longest matching Allow or Disallow path wins, and on a tie Allow wins. * matches any characters and a trailing $ anchors the end.
No robots.txt (a 4xx answer) means every crawler may fetch everything. A 5xx answer or no answer at all means crawlers should assume everything is disallowed until it comes back, which the checker reports as Unreachable.
Which AI crawlers does it check?
GPTBot (OpenAI training) and OAI-SearchBot (ChatGPT search), per OpenAI's crawler docs; ClaudeBot and Claude-SearchBot from Anthropic; PerplexityBot; CCBot from Common Crawl; and two control tokens that aren't crawlers but are read from robots.txt: Google-Extended for Gemini and Applebot-Extended for Apple. Blocking a training crawler and blocking a search crawler are different choices: the search ones decide whether the site can appear and be linked in AI answers.
Who uses this tool, and for what?
- SEO teams
- Catch a stray Disallow: / that blocks Googlebot after a site launch.
- Publishers
- See which AI training and search crawlers each property allows.
- Agencies
- Audit robots.txt across a client portfolio in one run.
- Researchers
- Measure how many sites in a list block AI crawlers.
What can’t this tool tell you?
- It tests the home page path only; other paths are listed, not tested.
- It reads robots.txt only, not meta robots tags, X-Robots-Tag headers or firewall rules that block crawlers by IP.
- Crawler tokens are from each operator's docs and can change; unlisted crawlers aren't shown.
- Up to 100 sites a run without an account and 500 signed in.
Common questions
- Does robots.txt stop a crawler?
- No. It's a request that well-behaved crawlers follow, not access control. A page blocked in robots.txt can still be indexed from links elsewhere; use noindex or authentication to keep it out.
- Which path is tested?
- The home page, /. Rules for other paths are listed in each group so you can see what else is blocked.
- Does blocking GPTBot keep my site out of ChatGPT?
- It stops OpenAI training on new crawls. ChatGPT search results use OAI-SearchBot, a separate token; the checker shows both.
- How much of the file is read?
- Up to 500 KiB, the minimum RFC 9309 says crawlers must parse. Rules after that are ignored, as crawlers may do.
- Can I check many sites at once?
- Yes. 100 sites a run without an account and 500 after a free Google sign-in, with a CSV.
- Is the robots.txt checker free?
- Yes, it's a free tool with no card. One GET of each site's /robots.txt.
Sources
You audited one domain. Cold outreach means thousands.
This tool checks a domain you type in. It will not tell you which addresses at that domain are real.
- Bulk mailbox verification with real SMTP checks
- The email finder — name plus company to a verified address
- Website enrichment for emails, owner names and socials
Plans from $9 a month, cancel anytime, no contract. Sign in with Google and your first 100 email checks are free, no card. The free tools stay free either way.