Menu

Post image 1
Post image 2
1 / 2
0

75% of Sites Blocking AI Bots Still Get Cited. Here Is Why Blocking Does Not Work.

DEV Community·Searchless·about 1 month ago
#4hPHK0eF
#ai#seo#chatgpt#webdev#citation#robots
Reading 0:00
15s threshold

Seventy-five percent of websites that actively block AI crawlers through robots.txt, meta tags, or server-level rules still appear in AI-generated answers from ChatGPT, Perplexity, and Gemini. Blocking does not stop citations. It stops you from controlling them. That number comes from new cross-platform citation analysis published by Position Digital in April 2026, and it dismantles the most common instinct brands have when they discover AI engines are using their content: shut the door. Why Brands Block AI Bots The logic feels sound. OpenAI, Google, Anthropic, and Perplexity all send crawlers across the web to ingest content. Their bots have user-agent strings like ChatGPT-User , Googlebot , CCBot , and PerplexityBot . You can add them to your robots.txt file and tell them to stay out. Many sites did exactly that. After the AI training data controversies of 2023-2024, publishers ranging from major news outlets to niche SaaS blogs added Disallow rules targeting known AI user agents.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More