Crawl Budget

Definition

Crawl budget is the number of pages a search engine bot will crawl on your site within a given timeframe. It is determined by both the crawl rate (how fast the bot can crawl without overloading your server) and crawl demand (how many pages Google considers worth crawling).

Why It Matters for SEO

For large sites with thousands of pages, crawl budget becomes a limiting factor. If Googlebot spends its budget on low-value pages, your important content may not get indexed promptly. Clean site architecture and proper robots.txt management help allocate crawl budget efficiently.