robots.txt

Definition

robots.txt is a text file placed in the root directory of a website that tells search engine crawlers which pages or sections of the site they should not request. It follows the Robots Exclusion Protocol and is one of the first files crawlers check when visiting a site.

Why It Matters for SEO

A properly configured robots.txt helps manage crawl budget by preventing bots from wasting time on pages that don't need to be indexed (admin areas, search result pages, staging environments). Misconfigured robots.txt can accidentally block important pages from being crawled.