Menu

📰
0

Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure

Diff·@Eureka-WMF·2 months ago
#5lQgBC
Reading 0:00
15s threshold

One year ago, the Wikimedia Foundation reported a significant increase in bot traffic to the Wikimedia projects, largely coming from crawlers who extract content to train generative AI systems. We shared about the impact of these crawlers, and introduced our action plan to ensure a fairer use of our resources. Let’s take a look at the progress we’ve made on protecting our infrastructure, what we’ve learned along the way, and next steps. ## **Recap: High demand, increased strain, less visibility** As generative AI increasingly draws from high-quality, human-created content, automated traffic has risen sharply on Wikimedia sites. While Wikimedia content is free, the infrastructure that serves it is not. Crawlers tend to access every part of the Wikimedia ecosystem – articles, media files, and developer platforms – exposing risks of overloading the systems and impacting the experience of our readers and contributors.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More