AI crawlers read your HTML, not your JavaScript

If you want to know why an AI assistant never cites your documentation, start one step earlier than most audits do. Before asking whether a model chose your page, ask whether it could read it.

The crawlers that feed AI assistants fetch a URL and parse the bytes that come back. They are not browsers. They do not wait for a bundle to download, hydrate and paint. Whatever your page renders after JavaScript runs is, for that fetch, invisible.

The check takes one command #

Ask for the page the way a crawler asks for it, then look at what actually arrived:

curl -sA "GPTBot/1.0" https://example.com/pricing \
  | tr -d '\n' \
  | sed 's/<[^>]*>/ /g' \
  | tr -s ' ' \
  | wc -c

That strips the tags and counts the readable characters in the HTML as served. Run it against a page you believe is rich in content. If a 40 KB page returns a few hundred characters, the content is not in the document — it is in the bundle, and it arrives too late.

Two follow-ups are worth the extra minute: check that a real <h1> is present in that same raw HTML, and check that your primary claim appears as text rather than as an image or a canvas.

What the fetch does not tell you #

Being readable is necessary, not sufficient. A page can be perfectly crawlable and still never be cited, because retrieval, ranking and citation are separate decisions made after the fetch. Treat the curl check as a floor, not a score.

A page that cannot be read cannot be cited. A page that can be read might still be ignored.

It also matters which crawler you are looking at, because they do different jobs and blocking them has different consequences.

Each vendor publishes its own list and its own rules — OpenAI documents its bots here and Google documents its crawlers here. Read them before you write a blanket Disallow, because a single careless line can remove you from the assistant you most wanted to appear in.

Where this leaves measurement #

Rendering is the part you control and can verify yourself in a minute. Being chosen is the part you cannot verify by inspection, because the answer differs by prompt, by model and by day.

That second part is the one we built an endpoint for: POST /v1/geo/answer-landscape reports share of answer across the SERP, the AI Overview and several LLMs for a query you choose, as structured JSON you can chart yourself. It is a measurement instrument, not a verdict — it tells you who was cited, and you decide what to do about it. The API documentation has the request shape and the credit cost per call.

Fix the floor first. It is the cheapest work in this entire field, and no amount of measurement compensates for a page that was never legible.