noindex vs robots.txt: which to use and when

Use noindex when you want a page crawled but kept out of search results. Use robots.txt Disallow when you want Google not to fetch URLs at all, usually to save crawl time on low-value sections. Don't combine them on the same page: if robots.txt blocks crawling, Google can't see the noindex, and the URL can still appear in results without a description.

What each one does

When to use noindex

• Thank-you pages, internal search results, thin tag archives.
• Pages users need but searchers don't, like account or checkout steps.
• Duplicate variants you can't consolidate with a canonical.
• Removing a page from Google that's already indexed.

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

When to use robots.txt

• Infinite URL spaces: faceted filters, sort orders, session IDs, calendars.
• Admin, cart and API endpoints that have no business being crawled.
• Large sections of near-duplicate URLs eating crawl time.

User-agent: *
Disallow: /search
Disallow: /*?sort=

Why combining them backfires

A common mistake: adding noindex to a page and also blocking it in robots.txt "to be sure". Google obeys robots.txt and doesn't fetch the page, so it never sees the noindex. If other sites link to that URL, Google can still list it in results with no snippet.

To remove a page from Google, keep it crawlable and add noindex. Once it has dropped out, you can block it in robots.txt if you also want to stop the crawling.

Mistakes that block good pages

How to check a URL

Look at the meta robots tag in the source, the X-Robots-Tag response header and the robots.txt rules that match the path. Our free checker does all three for up to 10 URLs, and the guide on checking indexability shows the commands.

For backlinks on other sites

Hosts use both mechanisms on profile pages, forums and user content. A backlink on a noindexed or robots-blocked page can't count, however many crawl requests you send. That's why @IndexForgeBot checks both before sending a link, and doesn't charge for the ones that fail.

What about nofollow?

Nofollow is a third, separate signal, and it applies to links, not to pages. A rel="nofollow" on a link asks Google not to pass credit through it; it doesn't stop the target page from being crawled or indexed. A nofollow value inside the meta robots tag applies to all links on that page. Neither one removes a page from the index. For that, only noindex works.

A robots.txt syntax refresher

• Rules apply per User-agent group. Google follows the most specific group that matches Googlebot.
• Disallow and Allow match by URL prefix.
• When both match, the longer (more specific) rule wins.
• * matches any characters, $ marks the end of the URL.
• robots.txt must be at the root of the host and is case-sensitive in paths.

Other meta robots directives

X-Robots-Tag for files

PDFs, images and other files can't carry a meta tag. Use the X-Robots-Tag HTTP header instead, set in the server configuration for those file types. It's also the cleanest way to noindex a whole staging host.

Removing a section from Google safely

1. Add noindex to every page in the section and keep it crawlable.
2. Remove the section from your sitemap.
3. Wait until the pages drop out of the index (check in Search Console).
4. Only then, if you want to stop crawling too, add a robots.txt rule.

Faceted navigation: a common case

Online shops generate thousands of filter combinations. A common approach is to let a few valuable filter pages be indexed, noindex the rest, and block parameter patterns that create infinite combinations in robots.txt. The exact mix depends on which filters people actually search for.

How to see what Google sees

URL Inspection shows whether a page is blocked by robots.txt, whether indexing is allowed, and what the rendered HTML looks like. The live test reflects the current state, which matters right after you change a rule. For pages on other sites, fetch them with a Googlebot user agent and read the headers and robots.txt directly.

Testing robots.txt rules before publishing

• List the URLs you want blocked and the ones you want open.
• Check each against the rules, remembering prefix matching and the longest-match rule.
• Pay special attention to rules without a trailing slash.
• After deploying, check the robots.txt report in Search Console.

Case: a staging site leaked into Google

A staging copy without protection gets crawled and indexed, competing with the real site. The fix: add X-Robots-Tag: noindex to every response on the staging host and keep it crawlable until the pages drop out. Then protect it with a password. Blocking it in robots.txt first would leave the URLs in results.

Case: cleaning up a blog archive

A blog with thousands of thin tag pages wants them out. Noindex the tag archives, remove them from the sitemap, keep internal links to the useful ones. Once they're out of the index, decide whether to block crawling as well. Most sites can leave them crawlable: Google will simply visit less often.

Quick decision table

Why noindex doesn't remove a page instantly

Google has to recrawl the page to see the new tag. Rarely crawled pages can stay in results for weeks after you add noindex. For urgent cases, the Removals tool hides a URL for about six months while the noindex takes effect.

How the two interact with canonicals and redirects

Canonicals and redirects only work if Google can crawl the page. A canonical tag on a URL blocked in robots.txt is never read. A redirect on a blocked URL is never followed. That's another reason to keep duplicate and old URLs crawlable until Google has processed the signal: block crawling only when there's nothing left for Google to learn from the URL.

Which one hurts more when it's wrong

A wrong robots.txt rule can block whole sections instantly, without changing any page, and it's easy to miss because the pages look fine in a browser. A wrong noindex usually sits in a template and spreads to every page using it. Both are worth checking after every deploy. A short list of your most important URLs, run through an indexability check after each release, catches both mistakes before Google acts on them.

robots.txt on other sites and your backlinks

When a host blocks a section in robots.txt, every link inside it is invisible to Google, even if the page loads fine in a browser. Profile pages, user directories and "go" redirect paths are blocked often. Check the host's robots.txt for the path of your link before paying for the placement or asking for crawling.

Testing whether a page is blocked

Search Console's robots.txt report shows the file Google last fetched for your properties. For other sites, open /robots.txt on the host, find the group for Googlebot or *, and compare the rules with the URL path, remembering that the longest matching rule wins.

Common questions

Does Google support noindex in robots.txt?

No. Google stopped supporting noindex rules inside robots.txt in 2019. Use the meta tag or the header.

How long does noindex take to remove a page?

Until Google recrawls the page. For faster removal of urgent pages, use the Removals tool in Search Console as well.

Should I noindex pagination?

Usually not. Paginated pages help Google discover deeper items. Let them be crawled and indexed, or simply not ranked.

Is a disallowed page bad for SEO?

Not by itself. It's only a problem when you block something you actually want indexed.