Google may expand the list of unsupported robots.txt rules in its documentation based on analysis of real-world robots.txt data collected through HTTP Archive. Gary Illyes and Martin Splitt described the project on the latest episode of Search Off the Record . The work started after a community member submitted a pull request to Google’s robots.txt repository proposing two new tags be added to the unsupported list. Illyes explained why the team broadened the scope beyond the two tags in the PR: “We tried to not do things arbitrarily, but rather collect data.” Rather than add only the two tags proposed, the team decided to look at the top 10 or 15 most-used unsupported rules. Illyes said the goal was “a decent starting point, a decent baseline” for documenting the most common unsupported tags in the wild. How The Research Worked The team used HTTP Archive to study what rules websites use in their robots.txt files.…