Noindex vs Disallow in Robots.txt: Which One Should You Use?
Noindex controls indexing, disallow controls crawling. Here is when to use each, and why combining them backfires.
Both a noindex tag and a robots.txt Disallow rule are about keeping pages out of Google, so people mix them up. They do different jobs. Choosing the wrong one is a common reason pages stay in search results when they should not โ or disappear when they should stay.
The difference in one line
- Disallow (robots.txt) = "do not crawl this URL". It controls crawling.
- Noindex (meta robots tag or X-Robots-Tag header) = "you may crawl this, but do not show it in search". It controls indexing.
Why disallow does not remove pages from Google
If a URL is blocked in robots.txt, Google cannot read it โ but it can still index the bare URL if other pages link to it. That is why Search Console sometimes reports "Indexed, though blocked by robots.txt". The result may show with no description. Disallow saves crawling effort; it does not guarantee a page stays out of results.
When to use noindex
- Thank-you and order confirmation pages.
- Internal search result pages.
- Thin tag or filter pages you want visitors to use but not to rank.
- Duplicate printer-friendly or campaign landing pages.
- Pages you want removed from Google that must remain online.
Add <meta name="robots" content="noindex"> โ the meta tag generator explains each tag โ or send an X-Robots-Tag header for files like PDFs.
When to use disallow
- Admin areas, scripts and backend endpoints.
- Endless parameter and filter combinations that waste crawl time.
- Large private file folders.
- Staging or test copies โ though a password is safer.
Build the rules safely with the robots.txt generator.
Never combine them on the same page
This is the classic mistake. If you add noindex to a page and block it in robots.txt, Google cannot crawl the page, so it never sees the noindex tag. The page can stay indexed. To remove a page: allow crawling, add noindex, wait until it drops out of the index, and only then block it if you still want to.
Removing a page from Google quickly
- Add noindex (or delete the page so it returns 404/410).
- Make sure robots.txt does not block it.
- For urgent cases, use the Removals tool in Search Console as a temporary measure while Google recrawls.
Check that your pages send the signals you intend with the bulk indexability checker. For the bigger picture, read what a robots.txt file is and what a canonical tag is.
Quick decision table
- Page must disappear from Google but stay online โ noindex.
- Page is deleted โ return 404 or 410 (or 301 to a replacement).
- Duplicate of another page โ canonical tag or 301 redirect.
- Thousands of low-value URLs waste crawling โ disallow (after they are out of the index).
- Private content โ password protection, not robots.txt.
When in doubt, prefer noindex for indexing problems and disallow for crawling problems.
Related guides
More free tools: browse all our free SEO tools, no sign-up needed.
Frequently asked questions
Can I put noindex in robots.txt?
No. Google does not support noindex rules in robots.txt. Use the meta tag or X-Robots-Tag header.
Which is better for duplicate pages?
Usually a canonical tag or a 301 redirect, not disallow. Noindex works for duplicates you do not want indexed at all.
How long does noindex take to work?
Until Google recrawls the page, which can take days or weeks. The URL Inspection tool can request a recrawl.
Does disallow save crawl budget?
Yes, it stops crawling of those URLs, which helps large sites with many low-value URLs.