Bulk noindex checker: which websites tell search engines to stay away?
In short: Paste a list of websites and check each home page for a noindex meta tag, an X-Robots-Tag noindex header and a robots.txt that disallows everything, with the exact signal, as a CSV. Limit: The home page and robots.txt only; other pages can differ.
A bulk noindex checker looks at each home page for signals that ask search engines to stay away. Paste up to 100 websites, or 500 after a free Google sign-in; each row shows a noindex robots meta tag, an X-Robots-Tag noindex header, or a robots.txt Disallow: / for all crawlers, with the exact text. Only the home page is read. Download the CSV.
Price: this tool is free. Checking a whole email list is from $9/mo, and the first 100 checks are free with Google, no card. Check your list free nowSee pricing
Sample result
acme.example's home page carries a noindex tag; globex.example's robots.txt blocks every crawler.
- Site
- acme.example
- Signal
- Meta robots: noindex, follow
- Also checked
- X-Robots-Tag, robots.txt
What this means for you
Both are requests to search engines, and they are different: noindex stops listing, robots.txt stops crawling.
- Robots, googlebot and bingbot tags read
- X-Robots-Tag header read
- robots.txt Disallow: / checked
- Home page only
- noindex
- X-Robots-Tag
- robots.txt
By SimpleVerifier Team · Last updated
Checking a list of sites before a campaign or a link-building push?
- Paste the sites here: 100 a run, or 500 after a free Google sign-in. Filter to the rows with a blocking signal and download the CSV.
- Upload or paste it at /verify. Every address gets a live mailbox check, and the first 100 email checks are free with a Google sign-in, no card.
How do I check many websites for noindex?
- Paste up to 100 sites, or 500 after a free Google sign-in, one per line; email addresses and URLs work too, and the domain is taken from each. You can also upload a CSV.
- Press Check for noindex. Each home page is requested once, over HTTPS and then HTTP, and its robots.txt once, five sites at a time.
- Read each row: Blocking signal found, with the signal (a robots, googlebot or bingbot meta tag, an X-Robots-Tag header, or robots.txt disallowing everything) and its text, or No blocking signal.
- Filter to the flagged rows and download the CSV.
Example
- You enter
- Three sites:
acme.example(a noindex meta tag left from launch),globex.example(robots.txt with Disallow: / for all crawlers) andinitech.example(neither). - You get
- acme.example: Blocking signal found, a robots meta tag with noindex. globex.example: robots.txt blocks all crawlers, noted as different from noindex. initech.example: No blocking signal. Download the CSV.
- Limits
- Up to 100 a run without an account and 500 when signed in with Google, counted over a minute. 10 sites a request; one GET of the home page from our server, no cookies, nothing stored.
What do noindex and robots.txt do?
A noindex directive, in a robots meta tag or an X-Robots-Tag header, asks search engines to leave a page out of their results (Google's robots meta tag guide). A robots.txt rule, defined in RFC 9309, asks crawlers not to fetch a path. Disallow: / under User-agent: * asks all crawlers to stay away from the whole site.
They are different. A page blocked in robots.txt can still appear in results if other sites link to it, and a crawler can't see a noindex on a page it was told not to fetch. This tool shows which signal it found.
Why would a site have one?
Staging sites, sites under construction and members-only areas often carry one on purpose. A live business site that has one by mistake, often left over from a launch, loses search traffic until it is removed. A site that is closed to search is also a hint that it isn't ready for outreach.
Only the home page is read, so a noindex on other pages isn't seen. For a full robots.txt, use the robots.txt checker; for who hosts a site, use the website host checker.
What do you do with the flagged rows?
If the sites are yours or a client's, fix the tag or header and check again. If they are prospects, a blocked home page is a reason to look closer before you write, and a talking point if you sell SEO.
Before you email anyone on a list, verify the addresses. A site that is hidden from search may also be unfinished, and its addresses may not exist yet.
Who uses this tool, and for what?
- SEO agencies
- Spot sites left on noindex after a launch before a pitch.
- Site owners
- Check several domains for a stray noindex after a migration.
- Link builders
- Skip sites that tell search engines to ignore them.
- Sales teams
- Flag prospect sites that look unfinished or hidden before outreach.
What can’t this tool tell you?
- The home page and robots.txt only; other pages can differ.
- Signals are requests to search engines, not proof of what is or isn't indexed.
- Meta tags added by JavaScript aren't seen.
- Up to 100 sites a run without an account, 500 signed in.
Common questions
- How do I check if a website has noindex?
- Paste the site into a bulk noindex checker like this one. It reads the home page's robots meta tags and X-Robots-Tag header and checks robots.txt for a rule that blocks everything.
- Is this noindex checker free?
- Yes. 100 sites a run without an account, 500 after a free Google sign-in, with no card, and a CSV download.
- Is a robots.txt block the same as noindex?
- No. robots.txt asks crawlers not to fetch pages; noindex asks search engines not to list them. The tool names which one it found.
- Does it check every page of a site?
- No, only the home page and the site's robots.txt.
- Why does a site show no blocking signal?
- No noindex was on its home page and robots.txt doesn't disallow everything. Other pages can still differ.
- Does it check whether a page is actually in Google?
- No. It reads the signals a site sends. Whether a page is in Google's index is something only Google's own tools can tell you.
Sources
A tidy list can still be full of dead addresses.
Extracting, deduplicating and syntax checks all work on the text of an address. None of them can tell whether the mailbox exists. Only the server that hosts it can answer that, through an SMTP check, and that is what the paid product does.
- Upload the cleaned file as it is: CSVs up to 100,000 rows
- A real SMTP mailbox check on every row, with catch-all detection
- Small list? Starter at $9/mo covers 2,500 emails a month, cancel anytime
Plans from $9 a month, cancel anytime, no contract. Sign in with Google and your first 100 email checks are free, no card. The free tools stay free either way.