Which AI Assistants Can Read Your API Docs? robots.txt and AI Citations in 2026

Paste a link to your API reference into an assistant and ask it a question about the page. Something fetches that page. Whether it read your robots.txt first depends on which assistant you asked, and for most of them the honest answer is that robots.txt was never meant to apply.

A small public test posted on 2026-10-09 made that visible. The author gave five assistants each their own unlinked URL, asked each one in a fresh chat to open it and read back a code, and logged every request. The author is plain that it was a few runs per assistant on one day. Read at that size, it is still a clear picture of what happens between a user's prompt and your server.

This post is for developer-tool and API companies that want ChatGPT, Claude, Gemini, Perplexity and Grok to read and cite their docs. It covers what the test observed per assistant, what each vendor's own documentation says about user-triggered fetches, and what a docs team should actually decide in robots.txt.

The test separated one question from all the noise around it: when a user asks an assistant to open a page, which requests arrive, and does any of them read robots.txt first? Each assistant got a secret URL linked from nowhere, so anything that requested it had to be that assistant acting on the prompt.

The setup was simple on purpose. A logging script sat in front of the site and recorded every request: path, user agent, address and time. The author then asked each assistant to open its link and report a code printed on the page. Across the five assistants in the 2026-10-09 post, 3 of 5 requested the planted page without reading robots.txt first, and one, Perplexity, never requested it at all.

Size matters here. The author says it was a few runs each, on one day, and every figure in this post from that test should be read as "this happened" rather than "this is the rate". For a rate, the closer source is the academic study posted on 2026-07-19, which ran ten assistants through four robots.txt conditions across 200 trials and reported substantial variation between them.

Claude was the only assistant that read robots.txt before touching the site and then honoured it. When the page was disallowed it refused and said it would not try to work around the rule. ChatGPT, Gemini and Grok, 3 of the 5, requested the planted page without reading robots.txt first, per the test write-up of 2026-10-09.

ChatGPT went straight to the disallowed page on its first visit to a brand new site without reading robots.txt. About 76 seconds later, OAI-SearchBot arrived and read robots.txt. The next time the author asked, ChatGPT refused the page without sending a request at all. In a reply on the same thread on 2026-10-09, the author reads this as ChatGPT reusing the crawler's saved copy of the rules, and says that cannot be proven from logs.

Gemini arrived with a user agent that read only "Google" and never requested robots.txt. The author reported that the addresses were not on Google's published lists. Perplexity sent its crawler to read robots.txt and the homepage, never fetched the planted page, and told the author the link might be phishing.

Grok is where the logs got strange. Every request brought 9 to 12 different addresses, most within a few seconds, all claiming to be Chrome or Safari. Over 6 requests the author counted 59 addresses and no reuse, and none of them requested robots.txt, per the same 2026-10-09 post. The addresses came from hosting companies and small ISPs in Brazil, Mexico and Poland, with a couple inside a commercial proxy network's registered range.

The vendor pages explain most of the test before you reach the logs. The key distinction is between a crawler, which visits on a schedule, and a user-triggered fetcher, which reads one page because a person asked. We read each first-party page on 2026-10-14 and recorded the exact wording for the fetcher.

OpenAI's crawler page, read 2026-10-14, says of ChatGPT-User that because these actions are initiated by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to decide what appears in search, and it publishes 4 IP range files, one per agent. So the ChatGPT first fetch in the test is the documented behaviour, not a violation.

Google's user-triggered fetchers page, last updated 2026-08-19, says these fetchers generally ignore robots.txt rules because the fetch was requested by a user. It lists Google-Agent and the Gemini Notebook fetcher and publishes 3 IP range files. The bare "Google" user agent in the test does not match the strings on that page, which is the part that is still unexplained.

Perplexity's bot page, read 2026-10-14, says Perplexity-User generally ignores robots.txt rules because a user requested the fetch, and that a change can take up to 24 hours to apply. It publishes 2 IP range files. Anthropic's article, dated 2026-04-07, says its bots honour robots.txt and that disabling Claude-User stops Claude retrieving your content for a user query. It publishes 1 IP list.

For xAI we found no crawler or fetcher documentation in its developer docs. We read the xAI developer sitemap on 2026-10-14: 0 of its 236 URLs mention crawling, robots, fetching or user agents, while a control path matched 25. That is absence of documentation, not a statement from xAI, but it means there is no stated rule to hold Grok's behaviour against.

A page gets cited in an answer because a search crawler indexed it earlier, not because a fetcher read it once. So the robots.txt rules that matter for AI citations are the ones aimed at OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. We covered the OpenAI half of that in detail in which crawler decides whether ChatGPT can cite you, and the same split into search, fetch and training agents holds at every vendor page we read on 2026-10-14.

There is a sign that the index matters more each month. One site owner reported on 2026-10-10 that ChatGPT-User fetches fell from about 433k in June to under 70k in September, while referrals from ChatGPT held at about 1,900 in both months. That is one site, but it fits the test: ChatGPT refusing a page from a stored copy of the rules, and answering from an index more often than from a live read.

Usually not. Allow the search crawlers, allow the user fetchers, and decide training on its own terms. That is the whole policy for a documentation site that wants to be cited, and it follows from the vendor pages rather than from any one test.

Allow the search crawlers because they are the only agents that decide whether a page can appear as a citation. Allow the user fetchers because a block there mostly affects the one vendor that honours it, Anthropic per its 2026-04-07 article, and leaves the others unaffected. Treat GPTBot, ClaudeBot and Google-Extended as a separate licensing question about future models, not a switch for today's answers.

The common failure is not a deliberate block. A read of 500 small B2B startups posted on 2026-10-13 found that of the 471 that answered, 3 blocked an AI answer crawler, and one of those 3 disallowed every agent with a single wildcard rule. The more likely problem on a docs host is an edge or firewall default that returns a 403 to an agent your robots.txt allows.

If you want a shape to start from, this is the structure with agent names read from each vendor page on 2026-10-14. Re-read those pages before you copy any name, because rosters change.

# Search crawlers: these decide whether docs can be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# User-triggered fetchers: a person asked for this page
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

# Training crawlers: a separate licensing decision
# (change Allow to Disallow if you decide against training)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

At most vendors, no. robots.txt is a request, read voluntarily, by agents that choose to identify themselves. It cannot stop a user-triggered fetch at the three vendors whose pages, read 2026-10-14, document one as exempt (OpenAI, Google, Perplexity), and it cannot stop an agent that never requests the file, which is what Grok did in all 6 requests in the 2026-10-09 test.

It also cannot override your edge. If a firewall returns a 403, the robots.txt line that says Allow is never consulted. And it cannot remove a page from a model that was already trained on it. A page that must stay private needs authentication, not a disallow rule. A page that must be cited needs to be reachable by the search crawlers at the network level as well as in the file.

For docs teams, the practical reading is reassuring. Your API reference is public, you want it read, and the agents that matter for citation are the polite, documented ones. The unidentified traffic is a bandwidth and analytics question, not a citation one.

You do not need to trust anyone's logs, including the ones in this post. The test costs one afternoon and a few unlinked pages, and it answers the question for your own host, your own edge rules and today's versions of each assistant.

Mint one unlinked URL per assistant with a short code on each page. Do not open the links yourself, because the test author did exactly that on the first run (r/SEO, 2026-10-09) and the only visitor in the logs was their own browser. Ask each assistant in a fresh chat to open its link and report the code, note the time, and match requests to prompts by timing rather than by user agent.

Then repeat it on another day. Answers moved between identical runs in our own measurement too: the same 12 queries changed outcome on 7 of them across 5 repeats, as published on 2026-07-27 in why a single AI visibility score is noise. Assistant output moves between runs, so one afternoon is one draw.

Being readable is the first link in a chain, not the result. A docs page also has to say something an assistant can lift, and the assistant has to reach for it when a buyer asks a question that does not contain your name. Our posts on whether llms.txt moves AI citations and making your docs agent-readable cover the next two links.

What we sell is the measured version of that chain for developer tools and APIs. Our answer engine optimization work starts with a crawlability and citability check of exactly the kind this post describes, then measures citation share against a held-out set of queries left unworked, so a change in access is judged by what it did to the answers.

Every behaviour attributed to an assistant in this post is the test author's report from the r/SEO post of 2026-10-09 and its comment replies, read in full on 2026-10-14. We did not rerun the test. Every statement about what a vendor's agent is supposed to do comes from that vendor's own page, read on 2026-10-14, with the page date where one is shown.

The limits are the test's limits. It is one person, a few runs per assistant, one day, and the author says the data on hand could not group the Grok addresses any further. The 2026-07-19 study of ten assistants is the better guide to how much behaviour varies. Nothing in this post measures whether allowing or blocking an agent changes citation share, which is a separate question with its own design.