Menu

Post image 1
Post image 2
Post image 3
1 / 3
0

AI Search Crawlers Are Curl From 1998, Not Chrome. Your SPA Is Invisible and Here Is the Mechanism.

DEV Community·Cihangir Bozdogan·29 days ago
#TcPrBgtt
#ai#seo#javascript#html#user#crawlers
Reading 0:00
15s threshold

There is a meeting that happens at almost every web team I have audited where someone says "we render fine, our React app renders fine for crawlers." Most of the time this is wrong. The kind of crawler the team has in mind is Googlebot, which has shipped a JavaScript-rendering second-pass since around 2019. The kind of crawler that actually decides whether a site appears in ChatGPT, Claude, Perplexity, or any of the AI search products is not Googlebot. It is GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and these things behave like curl from 1998. They issue an HTTP GET, parse the HTML they get back, and that is the entire interaction. JavaScript is not executed. Hydration does not happen. The single-page-app shell with <div id="root"></div> is what gets indexed. I worked through this the slow way.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More