Menu

Post image 1
Post image 2
Post image 3
1 / 3
0

The rise of the AI crawler

Vercel News·Giacomo Zecchini·4 days ago
#k0euTkfS
Reading 0:00
15s threshold

Real-world data from MERJ and Vercel shows distinct patterns from top AI crawlers. AI crawlers have become a significant presence on the web. OpenAI's GPTBot generated 569 million requests across Vercel's network in the past month, while Anthropic's Claude followed with 370 million. For perspective, this combined volume represents about 20% of Googlebot's 4.5 billion requests during the same period. After analyzing how Googlebot handles JavaScript rendering with MERJ , we turned our attention to these AI assistants. Our new data reveals how Open AI’s ChatGPT, Anthropic’s Claude, and other AI tools crawl and process web content. We uncovered clear patterns in how these crawlers handle JavaScript, prioritize content types, and navigate the web, which directly impact how AI tools understand and interact with modern web applications. Link to heading Data collection process Our primary data comes from monitoring nextjs.org and the Vercel network for the past few months.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More