What is Crawl Budget and When Should You Worry About It

You might have read that you should (or perhaps, don’t need to) be concerned about crawl budget. But is this correct? Does it apply to your business? Do you understand why crawl budget can be important? Or when you should make crawl budget a priority?

In the first of this two-part series, we’ll answer these questions and prepare you for part 2: How to Optimize for Crawl Budget, when we get into the specifics and offer some expert tips to help you maximize crawl budget and ensure that all of the key pages on your website are being indexed.

Let’s get started with some key definitions.

What Is Crawl Budget?

Crawl Budget is the number of pages a search engine (e.g., Google) crawls and indexes on a website within a given timeframe. Search engines calculate crawl budget based on crawl limit and crawl demand.

Crawl Limit = the amount of crawling a website can handle. Crawlers want to avoid overloading your web server and so will err on the side of caution if they notice a lot of URL request timeouts or server errors.

Crawl Demand = how inclined Google or another search engine is to crawl your pages. If your pages are popular (have many trusted links and rank for many queries) and fresh (content is up to date) then Google will be more inclined to crawl them more often.

Why Is Crawl Budget Important for SEO?

Crawl budget limits how often a website will be crawled by search engine crawlers, which in turn limits the number of pages that will be indexed by Google. If Google doesn’t index a page, it’s not going to rank for anything. So if your number of pages exceeds  your site’s crawl budget, there will be pages on your site that aren’t indexed.

When Do You Have to Worry About Crawl Budget?

Perhaps you have been under the impression that you don’t need to worry about crawl budget (?). Our experience in this area has given us a few ideas of our own about this topic.

In August 2020, Gary Illyes of Google suggested that crawl budget shouldn’t be a concern for the “majority” of businesses. He said:

“We’ve been pushing back on the crawl budget, historically, typically telling people that you don’t have to care about it. And I stand my ground and I still say that most people don’t have to care about it. We do think that there is a substantial segment of the ecosystem that has to care about it … but I still believe that – I’m trying to reinforce this here – that the vast majority of the people don’t have to care about it.”

Although Google has played down the importance of crawl budget, in our own experience here at adaptive, crawl budget has been an issue in a number of SEO audits that we have carried out for new clients.

How do you know if your business falls into the “substantial segment of the ecosystem that has to care about it”? We’ve found that in general, when crawl budget becomes an issue, it is  due to one or more of these three circumstances:

1. You run a very large website e.g. an ecommerce site with upwards of 10,000 pages then Google may have trouble finding them all, particularly if your categorization and site structure are not logical and well crafted
2. You have just recently added a new section with hundreds of new pages of content then Google may have difficulty crawling and indexing them all quickly
3. Your site has lots of redirects. Redirecting one URL to another is appropriate in many situations. However, if redirects are done incorrectly, it can lead to disastrous results. Long redirect chains and infinite loops make it difficult for Google  to crawl your site and wastes your crawl budget

‍

If you think crawl budget issues may be affecting your site’s indexability and rankings, find out how to optimize for crawl budget in our article: "How to Optimize for Crawl Budget" or get in touch to discuss an SEO audit.