Two delivery methods for the same indexing rules
The robots meta tag and the X-Robots-Tag HTTP header deliver page-level indexing and display rules, and Google obeys a noindex once it has crawled the page, while rel=canonical is only a hint and a robots.txt block hides both from Google. Use noindex or a canonical pointing to another URL, never both, and keep any noindexed URL crawlable.
The robots meta tag is an HTML element, <meta name="robots" content="...">. The X-Robots-Tag is a response header that carries the same rules, which makes it the only option for files that have no HTML to put a tag in, such as PDFs and images. Google's documentation now calls the values "rules"; older material, and most of the industry, still says "directives". Common informal names include meta robots, the noindex tag and the X-Robots header.
Three signals often get lumped together, and they work at different stages. robots.txt governs crawling, meaning whether Googlebot fetches the URL at all. The robots meta tag and X-Robots-Tag govern indexing and serving, meaning whether a fetched page is stored and how its result looks. rel=canonical governs consolidation, meaning which of several similar URLs represents the group. Most mistakes on this subject come from expecting one of these to do another's job.
The rules Google supports, and the ones it retired
Google's robots meta tag specification lists the rules below. Listing all changes nothing, since it is the default.
| Rule | Effect in Google Search |
|---|---|
noindex | Keeps the page, media or resource out of results |
nofollow | Google does not follow links on that page |
none | Shorthand for noindex, nofollow |
nosnippet | No text snippet or video preview; also withholds the content as direct input to AI Overviews and AI Mode (a static thumbnail may still appear) |
max-snippet:[number] | Caps snippet length in characters; 0 means none, -1 lets Google decide; also limits AI Overviews and AI Mode input |
max-image-preview:[setting] | none, standard or large |
max-video-preview:[number] | Longest video preview, in seconds |
indexifembedded | Lets content embedded through an iframe be indexed despite noindex; does nothing without noindex |
notranslate | No translation offered in results |
noimageindex | Images on the page are not indexed |
unavailable_after:[date/time] | Drops the page after the date; RFC 822, RFC 850 and ISO 8601 formats are accepted |
Three values have moved to Google's list of historical and unused rules. noarchive stopped controlling anything in Google Search after the cached-link feature went away; the cache: operator documentation was removed on September 24, 2024, and the rule was reclassified on October 2, 2024. nocache is not used by Google Search. nositelinkssearchbox was archived on November 29, 2024, along with the sitelinks search box itself. Google has said there is no need to strip noarchive out, because other services may read it, and Bing is one of them.
Writing the tag and the header correctly
The meta tag goes in the HTML <head>. Its name targets all crawlers with robots or a specific one by user agent:
<meta name="robots" content="noindex, nofollow">
<meta name="googlebot" content="nosnippet">
<meta name="googlebot-news" content="nosnippet">The header form accepts a user agent prefix and can repeat:
X-Robots-Tag: noindex
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollowFor file types, set the header at the server. These examples apply it to every PDF:
# Apache
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
# NGINX
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}A few behaviors are worth knowing before debugging anything. Names and values are case-insensitive in both forms. When rules conflict, Google applies the more restrictive one, so nosnippet beats max-snippet:50. When a page has rules for several crawlers, Google adds up the negative rules: robots with nofollow plus googlebot with noindex leaves Googlebot with both. Google will honor a robots meta tag found in the body, but the HTML standard places <meta name> in the head and other engines may be stricter, so the body is not a place to rely on. To hide part of a page from snippets rather than the whole page, the boolean data-nosnippet attribute works on span, div and section elements.
Why robots.txt makes both signals invisible
A robots meta tag or header is only read when the crawler fetches the URL. If robots.txt disallows the URL, Google never sees the rule, and its block-indexing guide is explicit that a noindex page must not be blocked by robots.txt for the rule to take effect. The same applies to a canonical tag on that page: it is never read either.
The blocked URL does not disappear as a result. Google can still index it from links elsewhere and show it without a description, which is exactly how URLs end up in the Page indexing report as "Indexed, though blocked by robots.txt". Disallowing a URL and adding noindex to it is therefore self-defeating: the disallow ensures the rule is never read.
Two related dead ends. A Noindex: line inside robots.txt was never an official rule, and Google stopped honoring unsupported robots.txt rules, including noindex, nofollow and crawl-delay, on September 1, 2019. And a page that ships noindex in its initial HTML cannot count on JavaScript to remove it, because Google may skip rendering once it sees that rule. The version of the page that matters is the one in the raw response.
How noindex and rel=canonical interact
The two signals ask for opposite outcomes. A cross-page canonical on URL B nominates URL A as the representative and invites Google to combine B's signals into A. A noindex on B says B should not be in the index at all. Because the exclusion is a rule and the canonical is a hint, the rule tends to win, and the consolidation the canonical was meant to produce may not happen. Mueller put the underlying point this way in 2018: signals get forwarded and combined with canonicalization, but noindex alone and robots.txt disallow are not clear signs for it.
Google's duplicate-URL documentation says it does not recommend noindex for steering canonical selection within one site, since it removes the page from Search completely. Staff statements on combining the two have shifted over time, which is why conflicting advice still circulates: in 2021 Mueller said both could be used and some signals might be forwarded, while in 2024 his advice was to pick one. The later, simpler guidance is the one to follow.
| Goal | Use | Avoid |
|---|---|---|
| Fold a duplicate into a preferred URL | rel=canonical, or a 301 if the duplicate can go | noindex on the duplicate |
| Keep a page out of Search | noindex alone, crawlable | A canonical pointing to another URL |
| Stop crawling of an infinite URL space | robots.txt disallow | Expecting noindex or a canonical there to be read |
A self-referencing canonical on a noindex page is generally harmless; the exclusion governs. The combination that should never ship is a canonical that points at a noindexed URL, which nominates as canonical a page the site has asked Google not to index. The same thinking extends to hreflang clusters: Gary Illyes has said that where Google treats cluster pages as duplicates, one noindex can affect the whole cluster, so each page meant to be removed should carry its own.
Templates are the usual source of the conflict. Paginated and faceted archives in some plugins emit both noindex and a canonical to page one or to an unfiltered listing. Decide what each template should do and let it emit one signal.
Diagnosing what Google actually received
URL Inspection in Search Console shows the HTML Googlebot fetched during its crawl, along with the user-declared and Google-selected canonical, so it answers both questions at once: did Google see the exclusion rule, and which URL did it choose. The Page indexing report groups affected URLs under "URL marked 'noindex'", "URL blocked by robots.txt" and "Indexed, though blocked by robots.txt"; the first is expected for intentionally hidden pages, while the third usually signals the self-defeating pattern described above.
Headers are the blind spot. Viewing the page source shows the meta tag but never an X-Robots-Tag, so a noindex added by a CDN, a server rule or a framework can go unnoticed for months. Fetch the headers directly:
curl -I https://www.example.com/whitepaper.pdfSite crawlers such as Screaming Frog and Sitebulb report the meta robots and X-Robots-Tag values in separate columns and flag URLs that are both canonicalized elsewhere and noindexed, which makes them the fastest way to find template-level conflicts across a large site.
Edge cases: long-term noindex, other crawlers and Bing
noindex, follow does not stay that way. Mueller said in 2017 that when Google sees noindex on a page for long enough, it concludes the page is not meant for Search and drops it completely; that has been widely reported as long-term noindex, follow behaving like noindex, nofollow. Illyes added in 2020 that a noindexed page never reaches the serving index, although Google keeps the fetched copy for things such as link graph calculation. Neither statement makes a noindexed page a dependable route for passing value.
noindex is aimed at search crawlers. Crawlers outside Search, such as AdsBot-Google, may need their own targeted rules. And nofollow in a robots meta tag covers only the links on that one page; it is not a way to redistribute PageRank around a site.
Bing reads the same tag and header, with differences. Bing added max-snippet, max-image-preview and max-video-preview in April 2020 and kept NOSNIPPET. Unlike Google, Bing still gives meaning to NOARCHIVE and NOCACHE, repurposed in September 2023 for its AI chat answers: NOCACHE limits what appears to the URL, title and snippet, and NOARCHIVE keeps the content out of those answers entirely, while both leave the page in regular results. Removing noarchive because Google ignores it gives up that control in Bing. Whether Bing reads a robots tag placed in the body is not documented.
Frequently asked questions
Can I use noindex and a canonical tag on the same page?
Not with a canonical that points to a different URL. The canonical asks Google to consolidate the page into another URL while noindex asks Google to drop it, and because the latter is a rule and the canonical a hint, the consolidation may not happen. A self-referencing canonical on a noindexed page is generally harmless.
What is the difference between the robots meta tag and X-Robots-Tag?
They carry the same rules through different channels. The meta tag sits in a page's HTML, while X-Robots-Tag is an HTTP response header, which also works for PDFs, images and other non-HTML files. The header is invisible in view-source, so check it with curl -I.
Does noindex work if the page is blocked in robots.txt?
No. Google has to crawl the page to read the rule, so a robots.txt disallow hides the rule completely. The URL can still be indexed from external links and appear without a description, reported as "Indexed, though blocked by robots.txt".
Does Google still support noarchive?
No. Google moved noarchive to its list of historical rules on October 2, 2024, after the cached-page feature was removed. Bing still uses it to control whether content appears in its AI chat answers, so there is a reason to keep it if Bing matters to the site.
Can I add or remove noindex with JavaScript?
Removing it is unreliable. When Google finds noindex in the initial HTML it may skip rendering, so a script that later deletes the tag may never run for Googlebot. Serve the intended value in the raw HTML or in the header.
Does a robots meta tag have to be in the head?
Google says it will respect a robots meta tag in the body, but the HTML standard expects <meta name> in the head and other search engines may not be as lenient. Place it in the head to be safe with every crawler.