Five tools, five different jobs
Duplicate content is controlled with five main signals, and they do not compete to do the same job: a 301 redirect removes a duplicate, rel=canonical consolidates it, noindex excludes a page from Search, robots.txt controls crawling rather than indexing, and hreflang chooses which language or regional version to show. Teaching them one tag at a time hides that division of labor, which is the most useful thing to know about them.
| Signal | Job | How binding | URL still reachable? |
|---|---|---|---|
| 301 or 308 redirect | Removes the duplicate and passes it to the target | A "strong" canonical signal, and the old URL no longer serves content | No |
| rel=canonical | Consolidates a duplicate into a preferred URL | A "strong" signal, but "a hint, not a rule" | Yes |
| noindex (meta robots or X-Robots-Tag) | Excludes the page from Search | A rule, once Google can crawl the page | Yes |
| robots.txt disallow | Stops crawling, not indexing | Respected for crawling; a blocked URL can still be indexed | Yes |
| hreflang | Chooses which locale's URL to show | Advisory; no ranking effect | Yes |
Read the job column as a set of verbs: remove, consolidate, exclude, don't fetch, localize. Most duplicate content errors come from asking one verb to do another's work, such as using noindex to consolidate, robots.txt to deindex, or a canonical to tie language versions together.
The pipeline model: crawl, index, select, serve
Duplicate content signals are easier to reason about once each is placed on the path a URL takes through a search engine.
- Crawl. robots.txt decides whether Googlebot fetches the URL. Redirects are met here too: the crawler follows them and ignores the redirecting URL's content.
- Index. Once a page is fetched, noindex in the HTML or the
X-Robots-Tagheader decides whether it may enter the index. - Select. Among indexable duplicates, Google clusters the URLs and picks a representative, weighing rel=canonical, redirects, sitemaps, HTTPS and internal links.
- Serve. At query time, hreflang lets Google swap in the URL for the searcher's language or region.
The model explains the most important interaction in one line: a signal at an earlier stage hides every signal at a later one. If robots.txt blocks a URL, Google never reads its noindex or its consolidation hint. Google's robots meta tag specification puts it directly: for a disallowed page, "any information about indexing or serving rules will not be found and will therefore be ignored." The blocked URL can still turn up in results without a description if other pages link to it. In the same way, noindex acts before selection, which is why Google advises against using it to influence which URL becomes canonical within a site.
What Google treats as duplicate, and what it does not
Every tool here assumes the duplicate content was identified correctly, and many mistakes start earlier than the choice of tag. Google scopes canonicalization to "duplicate or very similar pages," and its list of common causes includes protocol and host variants, device and regional versions, and the output of sorting and filtering on category pages. Three kinds of page are regularly misfiled as duplicates:
- Translations. Google counts language versions as duplicates "only if the primary content is in the same language." A French page and its English source are alternates, each self-canonical and linked with hreflang. Same-language regional versions with near-identical content are different: Google staff say it is common for one to be chosen as the representative, while hreflang can still swap the displayed URL.
- Paginated series. Page 2 of a category lists different items from page 1, so each page stands as its own preferred URL.
- Different pages on one topic. Two distinct articles ranking for the same query are not duplicates. In 2025 John Mueller said "pages aren't duplicates just because they happen to appear in the same search results page." This is the ground usually labeled keyword cannibalization, and its remedies are merging with a redirect, differentiating intent, or leaving both pages alone, not a consolidation hint that Google is likely to ignore.
A quick way to tell which side of the line a pair of URLs falls on is the Page indexing report. If Google already reports one URL as a duplicate of the other, it is a canonicalization question. If both are indexed and both earn impressions, it is a content question.
Rules, hints and what can be overridden
The second axis is how binding each signal is. Google's vocabulary is precise on this point and worth adopting.
- noindex is a rule. Once Google can crawl the page, it complies, and where robots rules conflict "the more restrictive rule applies." That firmness is why noindex prevails when it shares a page with a canonical: the page drops out, and the consolidation the tag asked for may never happen.
- robots.txt governs crawling only. Google "always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page." It manages crawl load; it does not remove anything.
- Redirects and rel=canonical are strong signals. Google applies the same word to both and treats every such preference as a hint. In practice a permanent redirect leaves less room for reinterpretation, because the duplicate stops serving content, whereas a canonical keeps both URLs live.
- Sitemaps are a weak signal. They support agreement but seldom decide anything alone.
- hreflang is advisory. Google's specification does not call it a hint, but Mueller has said "hreflang doesn't guarantee indexing," and Gary Illyes has said it brings "more targeted traffic" rather than a ranking benefit, with signals not passed between cluster members.
The working rule follows from this. Use a rule when the outcome must be certain, a strong signal when the goal is consolidation, and never expect a hint on a URL to outweigh a rule on the same URL.
Combinations that cancel each other out
Each signal is safe on its own. Certain pairs are not, and they recur in audits often enough to learn by sight.
| Combination | What happens | Resolution |
|---|---|---|
| robots.txt disallow plus noindex | The noindex is never read; the URL can appear as "Indexed, though blocked by robots.txt" | Allow crawling so the noindex can be seen |
| robots.txt disallow plus canonical | The tag is never read, and inbound signals to the blocked URL cannot be consolidated | Do not use robots.txt for canonicalization |
| noindex plus a canonical to another URL | "Drop me" contradicts "fold me into that URL"; noindex tends to win | Choose one: canonical to consolidate, noindex to exclude |
| Canonical aimed at a noindexed, redirecting or 404 URL | Nominates a URL Google cannot cleanly index | Point every tag at a 200, indexable URL |
| Cross-language canonical plus hreflang | One says index only one version, the other says index each; Google may ignore both | Self-canonical on every locale |
| noindex on one member of an hreflang cluster | If the cluster is treated as duplicates, "one noindex can affect the whole cluster" | Add noindex only to every page you actually intend to remove |
| Canonical to the clean URL, tracking parameters on internal links | Mueller called this "mixed signals" | Keep parameters off internal links |
| Sitemap lists URL A; A canonicalizes to B | No documented precedence decides it | List only preferred URLs in sitemaps |
A decision table: from situation to signal
Start from what should happen to the duplicate URL, not from whichever tag is most familiar.
| Situation | Use | Why not the others |
|---|---|---|
HTTP, wrong host, index.html or other variants nobody needs | 301 to the preferred form | A canonical would leave a pointless URL live |
| Page moved permanently or merged into another | 301 | Bing: "Moving content? Think 301, not rel=canonical" |
| Tracking or campaign parameters | Canonical to the clean URL | robots.txt would hide the tag and strand inbound signals |
| Sort orders | Canonical to the default sort, or robots.txt if crawl volume is large | noindex still costs crawling |
| Filter combinations with no search demand | robots.txt disallow or URL fragments | Google calls canonicals less effective long term for crawl control |
| Filtered views that should rank | Crawlable, self-canonical, stable URLs linked internally | Blocking would hide them |
| Pages users need but Search does not, such as internal search results | noindex, with no canonical to another URL | A canonical would not keep the page out of Search |
| Syndicated copy on a partner site | noindex on the partner's copy | Google stopped recommending cross-domain canonicals for syndication in 2023 |
| Language or regional versions | hreflang, with each version self-canonical | A cross-language canonical would suppress the translations |
| Two distinct articles with the same intent | Merge and 301 | Canonicals apply to duplicate or very similar pages |
| PDF duplicating an HTML page | Canonical in an HTTP Link header | A PDF has no head for a link element |
| Paginated category pages | Self-canonical on each page | Canonicalizing to page 1 hides items linked from deeper pages |
Retired controls that still appear in old advice
Several duplicate content controls recommended by older guides no longer exist or no longer work for Google, and following them wastes effort.
- The URL Parameters tool in Search Console went offline on April 26, 2022. There is no replacement setting; Google points to robots.txt rules instead. Bing still offers parameter normalization.
- The preferred domain setting was retired in 2019. Host preference is now expressed through redirects, canonicals, sitemaps and links.
Noindex:lines in robots.txt were never supported and have been ignored since September 1, 2019.- rel=prev/next stopped mattering to Google for indexing; in March 2019 Google said it "hasn't [used them] for years." Bing still uses them as a discovery hint, so leaving them in place does no harm.
- The International Targeting report was removed from Search Console in September 2022, so hreflang errors now have to be found with crawlers and URL Inspection.
Reading the outcome in Search Console
Each duplicate content signal leaves a recognizable trace in the Page indexing report, which is the quickest way to confirm that the tool you chose is doing its job.
| Status | Signal behind it | Usual reading |
|---|---|---|
| "Page with redirect" | 301 or 308 | Working as intended |
| "Alternate page with proper canonical tag" | rel=canonical accepted | Working as intended |
| "Duplicate, Google chose different canonical than user" | rel=canonical overridden | Other signals disagree, or the target is weak |
| "Duplicate without user-selected canonical" | Nothing declared | Google chose on its own |
| "URL marked 'noindex'" | noindex | Excluded as instructed; confirm it was intended |
| "Indexed, though blocked by robots.txt" | robots.txt | Crawling blocked, but the URL was indexed from links |
Two cautions apply when reading the numbers. Performance data for all variations is assigned to the URL Google selects, so a duplicate can vanish from reports with no change in rankings. And no Search Console report validates hreflang any more; locale problems surface only indirectly, through the Google-selected canonical of each version, checked against Google's localized versions documentation.
Frequently asked questions
Should I use noindex or a canonical tag for duplicate pages?
Use a canonical when the duplicate should fold into another URL and pass its signals along; use noindex when a page must exist for visitors but should not appear in Search at all. Google does not recommend noindex for choosing a canonical within a site, because it removes the page from Search entirely. Using both on the same page sends contradictory instructions, and noindex tends to win.
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, and Google says a disallowed URL can still be indexed if other pages link to it, appearing without a description. Because the page is never fetched, any noindex or canonical on it goes unread. To keep a page out of Search, allow crawling and use noindex.
Can a page have both a canonical tag and hreflang?
Yes, and in a multilingual setup it should. Each language or regional version carries a canonical pointing to itself and hreflang annotations listing every version, including itself, with return links. The conflict arises only when a canonical points to a different language version, which asks Google not to index the page that hreflang asks it to serve.
Is a 301 redirect stronger than a canonical tag?
Google labels both a "strong" signal and does not rank one above the other. They do different jobs: a redirect removes the duplicate for users and crawlers, while a canonical keeps both URLs live and asks Google to index one. Use a redirect when the old URL has no reason to stay reachable, and a canonical when it does.
How should faceted navigation duplicates be handled?
Treat it as a crawl problem first. For filter combinations with no search value, Google's faceted navigation guidance favors robots.txt disallow rules or URL fragments, and describes canonicals as generally less effective over the long term. Filtered views that deserve to rank should get stable, crawlable, self-canonical URLs linked from the site, and empty filter combinations should return a 404.