How to Check Whether Many Websites Are Set to Noindex

To check whether many websites are set to noindex, read each home page's robots tags and headers, and each site's robots.txt. The bulk noindex checker does that for up to 100 sites at a time (500 after a free Google sign-in) and names the signal it found.

Agencies use it to spot sites left closed after a launch before a pitch; sales teams use it to flag prospects that look unfinished.

The by-hand way

Three places can ask search engines to stay away:

1. A meta tag. View the page source and search for <meta name="robots" with noindex (or none) in its content. Tags named googlebot and bingbot count too. Google's robots meta tag guide covers them.
2. A response header. Run curl -I https://example.com and look for X-Robots-Tag: noindex. See checking the HTTP headers of a website.
3. robots.txt. Open /robots.txt and look under User-agent: * for Disallow: /, which asks all crawlers to stay out of the whole site. The format is defined in RFC 9309.

Step 1: paste the websites

Paste one site or a list, one per line. Email addresses and URLs work too, and the domain is taken from each. You can also upload a CSV.

Step 2: run the checker

Each home page is requested once, and its robots meta tags, the X-Robots-Tag header and the site's robots.txt are read. No JavaScript is run.

Step 3: read the rows

Row says	What it means	What to do
Meta robots tag with noindex	The home page asks not to be listed	Check whether that is intended
X-Robots-Tag header	The server sends the same request	Same
robots.txt blocks everything	All crawlers are asked to stay away	A different signal from noindex
No blocking signal	None found on the home page or in robots.txt	Other pages can still differ

Noindex and robots.txt are different

A page blocked in robots.txt can still appear in results if other sites link to it, and a crawler cannot see a noindex on a page it was told not to fetch. That is why the tool shows which signal it found rather than a single yes or no. Staging sites and members-only areas carry one on purpose. A live business site with one is often a launch leftover, and loses search traffic until it is removed. For the full file, use the robots.txt checker and read checking robots.txt for AI crawlers.

What this check cannot do

• It reads the home page and robots.txt only; other pages can differ.
• Signals are requests to search engines, not proof of what is or is not indexed.
• Meta tags added by JavaScript are not seen.
• The limit is 100 sites a run without an account and 500 signed in.

Found their emails? Check they exist before you send

A site hidden from search may be unfinished, and its addresses may not exist yet. Pull the published addresses from each site, then check them on the verify page: the first 100 email checks are free with a Google sign-in, no card. Each comes back valid, risky or invalid with a reason, and nothing is sent to anyone on the list. For larger lists, see bulk email verification and pricing.

Check your list for noindex now

Paste your sites into the bulk noindex checker.