Do AI crawlers read JavaScript? What to do about it

Do AI crawlers execute JavaScript?

No, with two exceptions. In December 2024 Vercel and MERJ reported that GPTBot generated 569 million requests across Vercel’s network in a month and Anthropic’s crawler 370 million, and that none of the major AI crawlers they tested executed JavaScript. The bots did download script files, 11.5% of ChatGPT’s requests and 23.84% of Claude’s, but never ran them.

A September 2026 review by Lantern lists the agents that read raw HTML only: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot and Claude-User from Anthropic; PerplexityBot and Perplexity-User; and Meta’s and ByteDance’s crawlers. The exceptions are Googlebot, which feeds Gemini and AI Overviews, and Applebot.

How can you tell what an AI crawler sees on your site?

Turn JavaScript off and reload the page. Whatever is left is roughly what GPTBot or ClaudeBot receives. You can do the same from a terminal by fetching the URL with curl and reading the HTML: if your headings, body copy and links are not in that response, they are invisible to most AI crawlers.

Do this for one page of each template: home, a product or service page, a blog post, a case study and any tool page. Templates fail together, so checking one of each finds most problems in under an hour. Write down what is missing and fix it at the template level rather than page by page.

• Disable JavaScript in your browser’s developer tools and reload key pages.
• Fetch the page with curl and search the HTML for a sentence from the body copy.
• Check the rendered HTML in Google Search Console separately; Google renders, the others do not.
• Look at server logs for AI user agents and the status codes they receive.

What should you render on the server?

Everything you would want quoted: the page title and headings, the main copy, prices, product facts, author and date, internal links and structured data. Vercel’s own recommendation is to prioritise server-side rendering for essential content and keep client-side rendering for dynamic enhancements. Interactive widgets, animations and personalisation can stay in JavaScript as long as the words around them are in the HTML.

You do not need to rebuild the site. Static generation or prerendering at build time is enough for most marketing pages, and many frameworks support it per route.

How should robots.txt treat AI crawlers?

Decide per agent. OpenAI documents three: OAI-SearchBot surfaces sites in ChatGPT search results, GPTBot collects content that may be used to train models, and ChatGPT-User fetches pages when a person asks ChatGPT to, where robots.txt rules may not apply. OpenAI says robots.txt changes take about 24 hours to take effect.

If you want to be cited, allow the search agents. Blocking a training crawler is a separate business decision and does not have to cost you search visibility. Whatever you choose, write the rules explicitly by user agent instead of relying on a wildcard.

What about single-page apps and site builders?

Single-page apps built with client-side frameworks are the most exposed, because the first HTML response is often just a root element and a script tag. If you run one for marketing pages, move those routes to static generation or server rendering; most modern frameworks let you do this route by route, so the logged-in app can stay as it is.

Site builders vary. Some ship server-rendered HTML by default; others render blocks in the browser. Check yours with the JavaScript-off test rather than assuming. Watch especially for content loaded after the page appears: tabs, accordions and “load more” lists whose contents are fetched on click are invisible to a crawler that does not click.

• Put text inside tabs and accordions in the HTML, hidden with CSS rather than fetched on click.
• Render the first page of any list on the server.
• Avoid content that only appears after scrolling or interaction.
• Keep important links as normal anchor tags with real URLs.

How do you monitor AI crawler access over time?

Look at your server or CDN logs by user agent. A weekly view of requests from GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot, with the status codes they received, will show quickly if a release starts returning errors or empty pages to one of them. Vercel’s study also noted that AI crawlers hit a notable share of failing URLs, so redirects and broken links cost you twice.

Pair the logs with a monthly JavaScript-off check of your most important pages. It takes minutes and catches the regressions that logs alone will not, such as a template change that moves copy into a script.

What about tools, calculators and other interactive pages?

Give every interactive page a server-rendered explanation: what the tool does, how to use it, the formula it applies and a few worked examples or answered questions. The widget can load afterwards. That way the page has something to cite even for a crawler that never runs the script, and people who land on it from an AI answer still find the tool working.

This site follows the same rule: each free tool page ships its description, steps and questions as static HTML, and the calculator itself runs in the browser.