For most of the web's history, content platforms operated on a simple binary: open or blocked. Bots that followed the rules, like search engine crawlers and legitimate aggregators, were welcomed. Bots that misbehaved were added to a blocklist. It was an imperfect system, but it was more or less functional. Then generative AI changed everything. The explosion of LLMs created enormous commercial demand for high-quality training data, and the web became the most convenient source of that data. AI crawlers began hitting content sites at unprecedented scale to extract data for model training. The reciprocal traffic loop that once underpinned the internet's content economy began to collapse. "With the rise of AI products looking to take data for model training, we found ourselves in a position in the last year or so to revisit that approach," said Janice Manningham, Strategic Product Leader at Stack Overflow, on the Leaders of Code podcast.…