robots.txt unreachable is not usually a content problem. It is usually a fetch, routing, DNS, CDN, middleware, firewall, redirect, or cache problem. That distinction matters because teams often waste time editing pages when Google is really saying: "I could not reliably fetch the file that tells me what I am allowed to crawl." Here is the debugging order I use. 1. Confirm the file exists at the root Open: https://example.com/robots.txt Enter fullscreen mode Exit fullscreen mode It should return a plain-text response from the same public host Google crawls. 2. Check the HTTP status Use: curl -I https://example.com/robots.txt Enter fullscreen mode Exit fullscreen mode You want a stable 200 OK . Watch for: 403 from bot protection 404 from routing 5xx from hosting or edge functions long redirect chains HTML being returned instead of plain text 3. Check middleware and auth rules This is especially easy to miss in modern app routers.…