You might have read that you should (or perhaps, don’t need to) be concerned about crawl budget. But is this correct? Does it apply to your business? Do you understand why crawl budget can be important? Or when you should make crawl budget a priority?
In the first of this two-part series, we’ll answer these questions and prepare you for part 2: How to Optimize for Crawl Budget, when we get into the specifics and offer some expert tips to help you maximize crawl budget and ensure that all of the key pages on your website are being indexed.
Let’s get started with some key definitions.
Crawl Budget is the number of pages a search engine (e.g., Google) crawls and indexes on a website within a given timeframe. Search engines calculate crawl budget based on crawl limit and crawl demand.
Crawl Limit = the amount of crawling a website can handle. Crawlers want to avoid overloading your web server and so will err on the side of caution if they notice a lot of URL request timeouts or server errors.
Crawl Demand = how inclined Google or another search engine is to crawl your pages. If your pages are popular (have many trusted links and rank for many queries) and fresh (content is up to date) then Google will be more inclined to crawl them more often.
Crawl budget limits how often a website will be crawled by search engine crawlers, which in turn limits the number of pages that will be indexed by Google. If Google doesn’t index a page, it’s not going to rank for anything. So if your number of pages exceeds your site’s crawl budget, there will be pages on your site that aren’t indexed.
Perhaps you have been under the impression that you don’t need to worry about crawl budget (?). Our experience in this area has given us a few ideas of our own about this topic.
In August 2020, Gary Illyes of Google suggested that crawl budget shouldn’t be a concern for the “majority” of businesses. He said:
“We’ve been pushing back on the crawl budget, historically, typically telling people that you don’t have to care about it. And I stand my ground and I still say that most people don’t have to care about it. We do think that there is a substantial segment of the ecosystem that has to care about it … but I still believe that – I’m trying to reinforce this here – that the vast majority of the people don’t have to care about it.”
Although Google has played down the importance of crawl budget, in our own experience here at adaptive, crawl budget has been an issue in a number of SEO audits that we have carried out for new clients.
How do you know if your business falls into the “substantial segment of the ecosystem that has to care about it”? We’ve found that in general, when crawl budget becomes an issue, it is due to one or more of these three circumstances:
If you think crawl budget issues may be affecting your site’s indexability and rankings, find out how to optimize for crawl budget in our article: "How to Optimize for Crawl Budget" or get in touch to discuss an SEO audit.