Menu

How I tracked which AI bots actually crawl my site
πŸ“°
0

How I tracked which AI bots actually crawl my site

DEV CommunityΒ·Joseph HernandezΒ·about 1 month ago
#pyFnhu57
#seo#ai#user#cloudfront#robots#hits
Reading 0:00
15s threshold

I launched a new domain two weeks ago and wanted to know which AI bots were actually showing up β€” not theoretically, but in my CloudFront logs. So I built a small tracker that parses access logs from S3 and reports hits per bot per URL. After 5 days, here's what the data shows. The setup The site is easerva.com β€” static HTML on S3 + CloudFront, zero JavaScript, JSON-LD on every page, sitemap submitted to GSC and Bing Webmaster Tools, IndexNow integrated. I enabled CloudFront standard logging (free, writes gzipped logs to S3 every few minutes), then wrote a script that filters by user-agent string for the bots that matter: Googlebot , Bingbot , OAI-SearchBot , ChatGPT-User , GPTBot , PerplexityBot , ClaudeBot , Claude-User , Applebot .…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More