Menu

Anthropic’s Claude Is Crawling the Web at Unprecedented Scale — and Website Owners Are Scrambling to Respond
🖼️
0

Anthropic’s Claude Is Crawling the Web at Unprecedented Scale — and Website Owners Are Scrambling to Respond

WebProNews·Name·about 1 month ago
#eNhZ3HDM
Reading 0:00
15s threshold

Anthropic, the artificial intelligence company behind the Claude chatbot, has dramatically increased its web crawling activity in recent months, raising alarm among website operators, SEO professionals, and publishers who say the aggressive data harvesting is straining their infrastructure and ignoring established protocols designed to control automated access. According to reporting by Search Engine Land , Anthropic’s bots have been observed crawling websites at rates that far exceed what most site owners consider reasonable, with some reporting hundreds of thousands of requests in short time periods. The activity has prompted a growing backlash from the webmaster community, which is now grappling with how to manage AI crawlers that behave very differently from traditional search engine bots like Googlebot. A Flood of Requests That Servers Can’t Handle The scale of the crawling has caught many website operators off guard.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More