Robots.txt and sitemaps are separate files that do related jobs. Robots.txt tells crawlers what they can and cannot access. A sitemap tells them what you want indexed. When these two files contradict each other, you end up with disallowed URLs appearing in the sitemap, which confuses crawlers and wastes crawl budget. Getting both files right is not complicated, but it does require knowing which rules apply where. This guide walks through the robots.txt file format, common configuration mistakes, and how to use a visual generator to avoid the most frequent errors. Photo by panumas nikhomkhai on Pexels What Robots.txt Does (and Does Not Do) A robots.txt file sits at the root of your domain and provides crawling directives to well-behaved bots. The two most common directives are User-agent (which crawler the rule applies to) and Disallow (which paths are off-limits for that crawler). Crucially, robots.txt is not a security mechanism. It relies on crawlers choosing to comply. Malicious bots routinely ignore it.…