How to check if a URL is indexable

A URL is indexable when it returns a 200 status, isn't redirected, has no noindex in its HTML or X-Robots-Tag header, isn't blocked by robots.txt for Googlebot, and its canonical points to itself. If any of these fails, Google won't index that URL, however good the content is. Indexable doesn't mean indexed, but it's the minimum.

The six checks at a glance

1. What status code does the URL return?

Ask for the headers only, without following redirects, and pretend to be Googlebot so you see what it sees:

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://site.com/page/

The first line shows the status. You want HTTP/2 200. A 404 or 410 means the page doesn't exist for Google. A 403 or 401 means access is blocked, often by a firewall or bot protection that treats Googlebot differently from a browser. A 5xx means the server failed.

Watch for soft 404s: a page that returns 200 but shows "not found" or an empty listing. Status-wise it passes, but Google will likely treat it as an error page.

2. Does the URL redirect?

If the status is 301, 302, 307 or 308, look at the location header. To see the whole chain:

curl -sIL https://site.com/page/ | grep -iE "^(HTTP|location)"

A redirecting URL won't be indexed under its own address. That's fine for old URLs you've moved on purpose. It's a problem when your sitemap or internal links point at the redirecting version, for example http:// instead of https://, or a missing trailing slash. Update them to the final URL.

3. Is there a noindex in the HTML?

Search the page source for the robots meta tag:

curl -s https://site.com/page/ | grep -io '<meta[^>]*name="\(robots\|googlebot\)"[^>]*>'

Anything containing noindex blocks indexing. Both name="robots" and name="googlebot" apply to Google. Common culprits are SEO plugins with "discourage search engines" switched on, staging settings copied to production and templates that noindex certain page types.

If the site builds pages with JavaScript, also check the rendered HTML in Search Console. A script that adds noindex after load can still block indexing.

4. Is there a noindex in the HTTP headers?

This one is easy to miss because it never appears in the page source:

curl -sI https://site.com/page/ | grep -i x-robots-tag

If you see X-Robots-Tag: noindex, the server, a CDN rule or a plugin is telling Google not to index the page. It's often set on purpose for PDFs or staging hosts and then accidentally left on for the whole site.

5. Does robots.txt allow crawling?

Open the file and read the rules that apply to Googlebot:

curl -s https://site.com/robots.txt

A few things to know when reading it:

• Google follows the most specific User-agent group that matches. If there's a User-agent: Googlebot group, the * group doesn't apply to Googlebot at all.
• Rules match by prefix. Disallow: /blog blocks /blog/, /blog-news/ and /blog.html.
• When an Allow and a Disallow both match, the longer rule wins.
• * matches any sequence of characters and $ marks the end of the URL.

Keep in mind that robots.txt controls crawling, not indexing. A blocked URL can occasionally appear in results without a description if other sites link to it. And a page you want removed should not be blocked in robots.txt, because Google then can't see its noindex.

6. Where does the canonical point?

Look for the canonical in the HTML and in the headers:

curl -s https://site.com/page/ | grep -io '<link[^>]*rel="canonical"[^>]*>'
curl -sI https://site.com/page/ | grep -i '^link:.*canonical'

The canonical should be the URL you're checking, exactly: same protocol, same host, same trailing slash. If it points somewhere else, Google will usually index that other URL and report this one as an alternate. Also note that Google treats canonicals as a hint. If your pages are near-duplicates, it may pick a different canonical than the one you declared.

Gotchas that fool manual checks

Your own checks can look clean while Googlebot sees something else. Watch for these:

• Different responses by user agent or IP. Some firewalls let your browser through and challenge anything that says Googlebot, or the other way round. Faking the user agent with curl isn't the same as coming from Google's IP ranges, so confirm with URL Inspection's live test.
• Cached responses. A CDN may serve an old version with an old noindex or canonical. Check the cache headers, and purge after fixing.
• Trailing slash and case. /Page, /page and /page/ are different URLs. Check exactly the one you're submitting.
• Mobile vs desktop. Google crawls mostly as a smartphone. If the mobile version hides content or sets different directives, that's the version that counts.
• Language versions. With hreflang, each version should be canonical to itself, not to the main language.

To check a short list by hand, loop over it in the terminal:

while read u; do
  printf "%s  " "$u"
  curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "$u"
done < urls.txt

Checking in Search Console

If you have access to the property, URL Inspection does most of this in one go. Use Test live URL to see the current state rather than the last crawl. It shows whether the page is reachable, whether indexing is allowed, the user-declared and Google-selected canonical, and the rendered HTML. The one thing it won't do is check many URLs at once.

Checking many URLs at once

For a whole sitemap, running commands by hand doesn't scale. A desktop crawler can export status codes, directives and canonicals for every URL. Or send the list to @IndexForgeBot: it runs these same six checks on each link as Googlebot, shows how many passed and lists the rest grouped by reason. The check is free. You only pay $0.01 for links you choose to send for indexing.

Indexable isn't the same as indexed

Passing all six checks means nothing technical stops Google. Whether the page actually gets indexed then depends on whether Google has crawled it yet and whether it thinks the page is worth keeping. If the page is indexable but still missing, the not-indexed checklist covers what to look at next.

Checklist

• Status 200 when requested as Googlebot
• No redirect, or you're checking the final URL
• No noindex in meta robots or meta googlebot
• No noindex in X-Robots-Tag
• Path allowed in robots.txt for Googlebot
• Canonical points exactly to this URL
• Rendered HTML contains the main content

Common questions

Is a 302 redirect worse than a 301 for indexing?

Either way, the redirecting URL itself won't be indexed. A 301 tells Google the move is permanent and helps it switch the indexed URL to the destination sooner. Use 301 for permanent moves.

Does nofollow on internal links stop indexing?

No, nofollow is not a noindex. But if the only links to a page are nofollow, Google has a weaker path to it, so use normal links internally.

What about a page blocked in robots.txt that also has noindex?

Google can't see the noindex because it isn't allowed to fetch the page. If you want the page out of the index, allow crawling and keep the noindex until it drops out.