JavaScript SEO for AI crawlers: what Google renders and most AI crawlers never see

Google renders JavaScript: it crawls the page, renders it in a headless Chromium and then indexes it. Most AI crawlers do not. In the Vercel and MERJ measurement from December 2024, the crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity read only the HTML sent by the server: whatever JavaScript paints afterwards does not exist for them.

To check it on your own site: AI visibility audit

The problem is invisible in the browser, which runs everything, and almost always in Google, which does too. That is why so many audits miss it. It is one of the access blocks in AI search optimization: a well-written answer is no use if the crawler that should cite it receives an empty <div>.

How does Google process a JavaScript page?

Rendering means running a page's JavaScript to obtain the final HTML the user sees. Google does it in three phases, according to its JavaScript SEO guide1:

1. Crawling. Googlebot fetches the URL, checks that robots.txt allows it and extracts the links from the HTML it receives.
2. Rendering. Pages that return a 200 status code join a queue. When resources allow, a headless Chromium runs the JavaScript. Google says a page may spend a few seconds in that queue, but it can take longer.
3. Indexing. Google uses the rendered HTML to discover more links and to index the content.

Even so, the same guide recommends server-side rendering or pre-rendering: it makes the site faster for users and crawlers, and not all bots can run JavaScript1. It adds a warning worth reading twice: if Google finds noindex in the original HTML, it may skip rendering. Removing that tag with JavaScript does not work.

Do AI crawlers run JavaScript?

The most widely cited public source is "The rise of the AI crawler", which Vercel published on 17 December 2024 with data from the consultancy MERJ2. It is worth knowing exactly what it measured:

• What data. One month of crawler traffic across Vercel's network, including nextjs.org and several sites built on different stacks.
• How much traffic. In that month, GPTBot made 569 million requests, Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million, against 4.5 billion for Googlebot.
• What it concluded. None of the major AI crawlers from OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider) and Perplexity (PerplexityBot) executed JavaScript. ChatGPT fetched JavaScript files in 11.50% of its requests and Claude in 23.84%, but neither executed them.
• Which ones do. Gemini relies on Googlebot's infrastructure and renders; so does AppleBot, with a browser-based crawler.
• What it left out. Microsoft Copilot, because it has no distinct user agent to track.

Two limits worth adding. The study does not explain how it told fetching a file apart from executing it. And it is almost two years old: crawlers change, and one may have started rendering since. The crawler pages from OpenAI, Anthropic and Perplexity explain what each one is for, but do not say whether they run JavaScript345. The prudent conclusion is still the one Vercel gives: anything important has to arrive in the server's HTML.

What does a crawler without JavaScript receive from a SPA?

A single-page application (SPA) built with client-side rendering sends an almost empty HTML file and a JavaScript bundle that fills it in the browser. This is what a crawler that does not run JavaScript receives when it requests a service page:

<!-- Client-side rendering: what the server sends -->
<!doctype html>
<html lang="en">
<head>
  <title>Loading…</title>
  <script type="module" src="/assets/app.4f9c.js"></script>
</head>
<body>
  <div id="root"></div>
  <!-- All the text is inserted by app.4f9c.js in the browser -->
</body>
</html>

And this is what the same crawler receives if the page is rendered on the server or generated as static HTML:

<!-- Server-side or static rendering: the text is already inside -->
<!doctype html>
<html lang="en">
<head>
  <title>Kitchen renovations in Leeds: prices and timescales</title>
  <meta name="description" content="What a kitchen renovation includes, what it costs and how long it takes.">
</head>
<body>
  <main>
    <h1>Kitchen renovations in Leeds</h1>
    <p>We renovate complete kitchens with a fixed quote and a written timescale…</p>
  </main>
  <!-- The JavaScript arrives afterwards and only adds interactivity -->
  <script type="module" src="/assets/app.4f9c.js"></script>
</body>
</html>

In the first case, OAI-SearchBot or PerplexityBot have no title, no heading and no sentence to cite. In the second, the text is there even if the JavaScript never runs.

Server-side rendering, static rendering and hydration: which to choose

web.dev, Google's site for web developers, defines each technique as follows6:

The practical rule: if the text you want cited is in the HTML before anything runs, it does not matter which crawler turns up. Current frameworks (Next.js, Nuxt, Astro, SvelteKit, Angular with SSR) offer server-side or static rendering; the work is usually switching it on for the templates that matter, not rebuilding the site.

There is a fourth route best avoided: dynamic rendering, which serves rendered HTML to bots and the JavaScript version to people. Google says it was a workaround and not a long-term solution, and recommends server-side rendering, static rendering or hydration instead7. It also only works for the bots your list recognises, and new AI crawlers appear every few months, as the guide to AI crawlers and robots.txt shows.

How to check what a crawler sees

Three tests, from quickest to most thorough.

1. View the page source. In the browser, Ctrl+U (or view-source: before the URL) shows the HTML as the server sends it. Search it with Ctrl+F for a sentence from your main text. Careful: "Inspect element" shows the DOM after JavaScript has run, which is exactly what you do not want to look at.

2. Request the page with curl. Useful for automation and for using a crawler's user agent:

# Is the sentence in the served HTML? 1 or more = yes; 0 = only JavaScript paints it
curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0)" https://www.example.com/services/ \
  | grep -c "fixed quote"

# How many words of text arrive without JavaScript? Compare with what you see on screen
curl -s https://www.example.com/services/ \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' | wc -w

The word count is approximate (multi-line scripts slip through), but it is enough to tell 40 words from 900.

3. The URL Inspection tool in Search Console. It tells you how Google sees the page: the live test shows a screenshot of the rendered page as its inspection tool sees it, along with the HTML, the HTTP headers and the JavaScript console messages8. If the screenshot is blank or half-built, Google does not see your content either. If the screenshot looks right, remember that it tells you about Google, not about AI crawlers.

Common failures that only show without JavaScript

• Tabs and accordions that load their content on click. If the text is not in the HTML until someone clicks, no crawler sees it. If it is in the HTML and only hidden with CSS, there is no access problem.
• FAQs assembled with JavaScript. It is exactly the format that gets cited most, and the one most often fetched from an API.
• Price and availability requested from an API. Google warns that structured data generated with JavaScript can make its Shopping crawls less frequent and less reliable9. An AI crawler simply will not see it.
• URLs with #. Google says it cannot reliably resolve URLs that load content through fragments and asks you to use the History API1.
• noindex in the initial HTML that JavaScript later removes: Google may never render the page1.

What gets cited also has to be extractable, not just accessible; how to write it is covered in the guide on how to get cited by AI.

What SmoothSeen checks

SmoothSeen counts the words in the HTML the server sends and in the same page after running its JavaScript in a browser, and warns you when much of the text depends on the browser. You will find it in the AI search half of the analysis, alongside crawler access in robots.txt. It also flags FAQs marked up with FAQPage that do not appear in the HTML.

Frequently asked questions

Does Google index content generated with JavaScript?

Yes. Google crawls the page, places it in a rendering queue, runs the JavaScript in a headless Chromium and indexes the resulting HTML. Even so, it recommends server-side rendering or pre-rendering because the page is faster and because not every bot runs JavaScript. Bear in mind that rendering can be delayed and that an initial noindex can prevent it altogether.

Does ChatGPT read content loaded by JavaScript?

According to the measurement Vercel published in December 2024, no: OAI-SearchBot, ChatGPT-User and GPTBot sometimes fetched JavaScript files but did not execute them. OpenAI does not document how its crawlers process pages, so the prudent approach is for the text you want ChatGPT to cite to arrive in the server's HTML rather than depend on the browser.

Do I need to rebuild my site if it is a SPA?

Usually not. Most current frameworks can render on the server or generate static HTML per page, and it is enough to switch that on for the templates you want cited: services, product pages, articles and FAQs. Private areas, such as the user dashboard or the basket, can keep rendering in the browser without affecting your visibility.

Is dynamic rendering a good solution?

Google no longer recommends it: it describes it as a workaround, not a long-term solution, and suggests server-side rendering, static rendering or hydration instead. It also depends on recognising each bot by its user agent, and the list of AI crawlers changes every few months. If a new crawler is not on your list, it will receive the empty page.

What to do next

Open the source of your most important page and search for a sentence from the main text. If it is not there, switch on server-side or static rendering for that template before touching anything else. Once the text arrives in the HTML, carry on with the other blocks in the guide to AI search optimization.