What Is Crawl Budget in SEO? (And When It Actually Matters)
What crawl budget means, why most small sites can ignore it, and how large sites stop wasting it.
Crawl budget is the number of pages a search engine is willing and able to crawl on your site in a given period. Google describes it as a combination of how much your server can handle (crawl capacity) and how much Google wants to crawl your pages (crawl demand).
Does your site need to worry?
Usually not. Google has said crawl budget is mainly a concern for very large sites โ hundreds of thousands or millions of URLs โ or sites that add many pages quickly. A blog with a few hundred posts, or a shop with a few thousand products, is normally crawled fine if it is well built.
It matters when you see signs like:
- New pages take weeks to be crawled.
- Search Console shows many "Discovered โ currently not indexed" pages. See what that status means.
- Crawl stats show bots spending most time on parameter or filter URLs.
What wastes crawl budget
- Endless filter and sort combinations on category pages.
- Session IDs and tracking parameters in internal links.
- Redirect chains and links to redirected URLs.
- Soft 404s and thin duplicate pages.
- Slow server responses, which make crawlers slow down.
- Infinite spaces, such as calendars that link to every future date.
How to use crawl budget well
- Keep your sitemap clean โ only canonical, indexable URLs. Generate it with the XML sitemap generator.
- Link to clean URLs internally, not parameter versions.
- Block true junk (internal search results, endless filters) in robots.txt โ but never block pages you want indexed.
- Fix redirect chains โ read how to fix redirect chains.
- Speed up the server so crawlers can fetch more pages without strain.
- Remove or merge low-value pages that nobody needs.
How to check crawl activity
In Search Console, open Settings โ Crawl stats. You can see total requests, average response time, response codes and which file types and purposes (discovery or refresh) take up crawling. Server logs give even more detail, showing exactly which URLs bots request.
A worked example
A European fashion marketplace had 40,000 products but over two million crawlable URLs, because each colour, size, brand and price filter created a new address. Crawl stats showed most requests going to filter combinations, while new products took weeks to appear. The team made filter links non-crawlable except for a few popular combinations with real search demand, cleaned the sitemap and fixed redirect chains. New products began appearing in search within days.
For smaller sites, focus on content quality and internal links instead โ see how to find orphan pages.
Related guides
More free tools: browse all our free SEO tools, no sign-up needed.
Frequently asked questions
Does crawl budget affect rankings?
Not directly. It affects how quickly pages are discovered and refreshed.
Can I ask Google to crawl more?
Not directly. A faster server, cleaner URLs and more valuable content raise crawl demand and capacity naturally.
Does noindex save crawl budget?
Not much โ Google still has to crawl a page to see the noindex.
Is crawl budget the same on Bing?
The idea is similar. Bing also supports IndexNow, which tells it directly when pages change.