Blocked by Robots.txt in Search Console: How to Fix It
Find which robots.txt rule blocks your pages, decide whether it should, and fix it safely.
"Blocked by robots.txt" in Search Console's Page indexing report means Google found these URLs but did not crawl them, because a rule in your robots.txt file told it not to. Sometimes that is exactly what you want. Sometimes a rule is blocking pages that should rank.
Step 1: Decide whether the block is intended
Open the report and look at the example URLs.
- Intended โ admin pages, internal search results, cart and checkout, filter parameters. Leave them blocked.
- Unintended โ product, service, blog or category pages. These need fixing.
Step 2: Find the rule
Open yourdomain.com/robots.txt and look for Disallow lines that match the blocked URLs. Common culprits:
Disallow: /left over from a staging site โ blocks everything.- Wildcards that are too broad, such as
Disallow: /*?blocking every URL with a parameter, including pages you want indexed. - A folder rule like
Disallow: /blogthat also blocks/blog-guides/because rules match the start of the path. - Rules added automatically by a plugin, host or security tool.
The robots.txt report in Search Console shows which version of the file Google last fetched and whether it had errors.
Step 3: Fix the rule safely
Edit the file so the important pages are allowed and the junk stays blocked. Make rules as specific as possible, and add an Allow rule if you need to open a path inside a blocked folder. Generate the file with the robots.txt generator to avoid syntax mistakes, and read what a robots.txt file is for how rules are matched.
Step 4: Confirm the fix
- Check the live robots.txt file loads with your changes.
- Run the affected URLs through the bulk indexability checker โ they should no longer be blocked.
- In Search Console, inspect a fixed URL, then click "Validate fix" on the issue.
Google usually re-reads robots.txt within about a day.
Want a page out of Google instead?
Blocking crawling does not reliably remove a page from search results. If your goal is removal, allow crawling and use a noindex tag โ see noindex vs disallow.
A worked example
A news site noticed new articles were not appearing in Google. The report showed hundreds of article URLs "Blocked by robots.txt". A rule Disallow: /*? had been added to block tracking parameters โ but the site's article URLs used ?id=. The team replaced it with specific rules for the tracking parameters only, and articles were crawled and indexed again within days.
Safe habits for robots.txt
- Keep a copy of the working file before every change.
- Change one rule at a time and test the affected URLs.
- Comment your rules with a line starting with # explaining why each exists.
- Check after deployments โ staging files sometimes overwrite the live one.
- Do not use robots.txt to hide private content; the file is public and anyone can read it. Protect private areas with a login.
A short monthly glance at the Page indexing report catches accidental blocks before they cost much traffic.
Related guides
More free tools: browse all our free SEO tools, no sign-up needed.
Frequently asked questions
Can blocked pages still appear in Google?
Yes. Google may index the URL without its content if other pages link to it.
How quickly does a robots.txt change work?
Google typically refreshes its copy within about 24 hours.
Should I block CSS and JavaScript files?
No. Google needs them to render your pages properly.
Is this status an error?
Not necessarily. It is only a problem when important pages are blocked.