GPTBot, OAI-SearchBot, and Which Crawler Actually Decides Whether ChatGPT Can Cite You

On 2026-08-29 somebody pulled the robots.txt file from 72 companies and counted which OpenAI crawlers each one named. GPTBot appeared in 18 of them. OAI-SearchBot appeared in 11. OAI-AdsBot appeared in none.

That gap is the whole subject of this post, because those two crawlers do different jobs, and only one of them has anything to do with whether ChatGPT can cite you.

We read all ten pages ranking for oai-searchbot in the United States on 2026-09-08, plus the top ten for gptbot robots txt, block gptbot and openai crawlers. That is 40 organic results across 34 distinct domains. They fall into three groups. Bot directories tell you the user-agent string and the IP range. Block-it guides answer the question "should you block AI crawlers" as though the crawlers were one undifferentiated group. First-party OpenAI documentation is accurate, complete, and deliberately silent about the marketing consequence.

Not one of the 40 connects the crawler choice to the citation outcome. This post does that, and it also corrects the version of the story that is currently spreading, which is dramatic, memorable, and pointed in the wrong direction.

No. GPTBot governs whether your content can be collected to train future models. OAI-SearchBot governs whether ChatGPT search can show your site in an answer. They are separate user agents doing separate jobs, and a robots.txt rule against one has no effect on the other.

This is not an inference from practitioner testing. It is what OpenAI says in its own crawler documentation, which we read at the live page on 2026-09-09 rather than quoting somebody else's quotation of it.

The consequence of the other switch is stated just as plainly, and it is the sentence that makes OAI-SearchBot the expensive one to get wrong.

Read those two statements together and the practical picture is simple. If you disallow GPTBot, ChatGPT search can still find you, retrieve you and cite you. If you disallow OAI-SearchBot, it cannot show you in search answers at all, though your URL can still surface as a navigational link.

OpenAI runs four named user agents. Most robots.txt files know about one or two of them.

OAI-SearchBot is for search. OpenAI's description is that it is used to surface websites in search results in ChatGPT's search features. This is the citation crawler. If you want to appear in ChatGPT search answers, this is the one that must be able to reach you.

GPTBot is for training. Allowing it indicates that crawled content may be used to train OpenAI's generative AI foundation models. Disallowing it is a statement about training data and nothing else.

ChatGPT-User is a live fetch. When a person asks ChatGPT or a Custom GPT a question that requires visiting a specific page, ChatGPT may visit it with this agent. OpenAI states that it is not used for crawling the web in an automatic fashion, and adds an important caveat: because these actions are initiated by a user, robots.txt rules may not apply to them in the same way.

OAI-AdsBot validates the safety of web pages submitted as ads on ChatGPT. It only visits pages submitted as ads. If you do not buy ChatGPT ads it is irrelevant to you, which is presumably why zero of the 72 companies in that count named it.

The diagram above is worth sitting with, because it contains the answer to the question most people are actually asking when they ask about GPTBot. There are two routes from your page into a ChatGPT answer. One goes through a search index and produces a citation with a link. The other goes through model weights and produces a mention with no link at all. GPTBot feeds the second route. OAI-SearchBot feeds the first.

Here is the version of this story that circulates, and it is a good story: teams meant to block GPTBot, wrote a sloppy robots.txt rule, and accidentally blocked OAI-SearchBot too, silently switching off their own ChatGPT citations.

It is memorable, it is alarming, and in the one published sample where anybody counted, it did not happen.

The practitioner who had been making that exact claim went and pulled robots.txt from 109 well known sites to check it. Zero of them blocked OAI-SearchBot without also blocking GPTBot deliberately. He then posted the count and publicly reversed himself, which is a considerably more useful contribution than the original claim was.

What that sample found instead is the reverse failure, and it is both more common and less dramatic. Teams believe they have opted out of AI. They have opted out of training. Their pages remain fully eligible for ChatGPT search citations, which may or may not be what they wanted, and nobody involved has noticed the difference.

Look at the disallow rates from that same 109-site sample and the pattern is unmistakable.

Every crawler at the top of that list is a training crawler. CCBot feeds Common Crawl. ClaudeBot, Bytespider, meta-externalagent and anthropic-ai are training agents. Applebot-Extended and Google-Extended are training opt-outs specifically. The crawler that actually decides ChatGPT search citations sits at the very bottom at 7 percent.

That is not an accident and it is not incompetence. It is what happens when a whole industry makes a decision about training and a separate question about retrieval quietly rides along underneath it.

One caution about that table before anyone quotes it. It is one person's count of 109 sites, posted to a subreddit, with the method stated only as pulling robots.txt. We did not re-derive it. The relative shape is informative; the absolute percentages are a sample, not an industry prevalence rate, and this post does not treat them as one.

The same count surfaced a pattern worth naming: how often a crawler is named tracks how OLD it is rather than how much it matters. CCBot has existed since 2011 and was the most-named crawler in that sample at 13 sites. The agent OpenAI introduced this year was named by nobody.

The same 109-site sample splits by industry, and the split is more informative than the headline rate. The author pulled 120 recognisable domains across news, SaaS, ecommerce, marketing publishing, health, finance and travel, of which 109 returned something parseable.

GPTBot Disallow rate first, then OAI-SearchBot, per vertical: marketing and SEO publishers 33 percent and 0 percent, news 31 percent and 27 percent, health 25 percent and 0 percent, ecommerce 13 percent and 6 percent, travel 11 percent and 0 percent, SaaS 6 percent and 3 percent, finance 0 percent and 0 percent.

Two rows deserve attention. Marketing and SEO publishers have the highest training-block rate in the sample and touch the search crawler zero times. People who write about search for a living have evidently worked out the distinction and are playing it deliberately: do not train on me, do cite me. That is the sophisticated position, and it is the one this post is recommending by a different route.

News is doing something else entirely. 27 percent block the search crawler by name, against 63 percent blocking Google-Extended and 77 percent blocking ClaudeBot. That is not confusion, it is leverage. Blocking the thing somebody wants is how you eventually get paid for it, and at least one publisher blocking OAI-SearchBot by name is in active litigation. If you are a news publisher, the calculus in this post is genuinely different for you, and you should ignore the recommendation.

There is a further pattern in that sample worth stealing whatever vertical you are in. Six of the nineteen sites blocking GPTBot are not doing a flat block at all. They set a site-wide Disallow and then allow specific paths back in: one carries 82 Allow rules under that group covering product and feature pages across every locale it runs, another carries 11. So the real choice is not block or allow. It is deny by default and let the model see your commercial pages while keeping it out of the rest of the library.

And one incoherence, which is the tell that these files are edited once per news cycle rather than reviewed. Of the nineteen sites blocking GPTBot, only fourteen also block Google-Extended. A quarter of them have an anti-training policy that stops at OpenAI and lets Google train on everything.

Often it is not, and this is the finding that quietly invalidates most crawler audits including the two counts above.

A separate practitioner parsed robots.txt from 624 sites to RFC 9309 and paired it with an actual probe of what each origin served. He then took 14 sites that refuse an AI crawler at the origin and probed each one with all eight named crawlers plus three controls: an ordinary browser, plain curl, and an invented crawler user-agent that appears on no blocklist anywhere.

All 14 returned 403 to GPTBot or ClaudeBot while serving a browser 200. Twelve of the fourteen said nothing about AI crawlers in robots.txt at all. The refusal was happening at the server, below the level robots.txt controls, from a security plugin or a host default that nobody had deliberately configured.

And then the finding that should change how you audit your own site:

Five of the fourteen, sitting in five different countries on four different server stacks, refuse ClaudeBot and nothing else, which is the signature of one copied rule circulating rather than five independent decisions. The same probe also recorded two method caveats worth carrying: an HTTP 429 is a rate limit and not a block, and anything behind a bot-verifying CDN is unverifiable rather than blocked.

Two of those fourteen sites had a hand-written robots.txt section explicitly allowing the exact crawler their own server was refusing. Somebody sat down, thought about AI crawlers, made a decision, wrote it into the file, and the decision had no effect whatsoever because a different layer was answering first.

The invented user-agent control is what makes this a real finding rather than a story about generic bot protection. If the sites had simply been refusing everything unfamiliar, the invented agent would have been refused too. It was not. The blocks were specific to named AI crawlers, configured somewhere other than the file everyone reads.

There is a smaller version of the same failure worth naming. A robots.txt plugin can write rules you never chose, and the file is generated rather than authored, so reading your own repository tells you nothing. The only reliable check is fetching your live robots.txt over the network and then, separately, checking what your origin serves to each named agent.

On 2026-07-01 Cloudflare announced a three-way classification of AI bot traffic into Search, Agent and Training, and gave the controls to every customer tier including Free. It also announced new defaults to follow.

Those defaults take effect on 2026-09-15. We read the announcement at Cloudflare's own blog on 2026-09-09 rather than relying on the secondary reports, because the detail that matters is easy to garble and several summaries already have.

For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on pages that display ads, while Search remains allowed by default. Cloudflare's stated reasoning is that an ad signals a site owner meant a person to land there, so on those pages it keeps away bots that prevent human attention, while Search is the behaviour that funnels visitors back.

Three things about the scope, stated precisely, because this is where the summaries go wrong. It applies to new domains onboarding to Cloudflare, not retroactively to every site already on the platform. It applies to pages that display ads, not to every page. And Search remains allowed, so the crawler this post is about is on the permitted side of the new default.

The genuinely interesting thing about the Cloudflare taxonomy is not the deadline. It is that a CDN and a model vendor, working independently and for different commercial reasons, arrived at the same three-way distinction.

Cloudflare defines Agent as automated behaviour acting in real time on a person's behalf, and names ChatGPT-User as an example. That is exactly the job OpenAI documents for ChatGPT-User. Cloudflare defines Training as a crawler taking content to permanently absorb it into a model, which is exactly OpenAI's description of GPTBot. Cloudflare defines Search as behaviour that funnels visitors back, which is OAI-SearchBot.

When two parties with no reason to coordinate carve a space into the same three pieces, that is reasonable evidence the pieces are real. It is the strongest argument available that the GPTBot versus OAI-SearchBot distinction is infrastructure rather than a practitioner theory, and it is a better argument than any of the counted samples in this post.

It also means the decision is moving. Once a CDN classifies and gates this traffic, your robots.txt is one input among several, and reading it tells you even less about what your site actually serves than it did before.

These two get conflated more often than GPTBot and OAI-SearchBot do, and the difference matters because they fail in different ways.

OAI-SearchBot crawls on a schedule to build an index. It is not reacting to any particular person. It visits, it reads, and what it finds becomes part of the candidate set ChatGPT search can draw on later. If you block it, you are absent from that candidate set, and no amount of somebody asking about you will put you back in it.

ChatGPT-User fetches on demand. Somebody asks ChatGPT a question that requires looking at a specific page right now, and ChatGPT goes and gets it. There is a human waiting on the other end of that request. If you block it, the specific case that breaks is a person pointing ChatGPT at your URL and getting told it cannot read the page.

OpenAI adds a caveat to ChatGPT-User that has real consequences and is easy to miss. Because these actions are initiated by a user rather than by an automatic crawl, robots.txt rules may not apply to them in the same way. That is OpenAI describing its own behaviour, and it means a robots.txt Disallow is not a guarantee for this agent the way it is for the other three.

The practical reading is that these are three separate decisions and you should make them separately:

1. Do you want to appear in ChatGPT search answers? That is OAI-SearchBot.
2. Do you want a person who pastes your URL into ChatGPT to get a useful answer? That is ChatGPT-User.
3. Do you want your content used to train future models? That is GPTBot.

Most teams have an opinion about the third question and have never been asked the first two. That is the actual problem, and it is a much less exciting problem than the one the scare story describes.

OpenAI documents that it does, and publishes IP ranges so you can verify that traffic claiming to be GPTBot actually originates from OpenAI.

Some practitioners dispute it. There is an r/nginx thread, which ranks ninth for gptbot robots txt, accusing OpenAI of not respecting robots.txt and of varying user-agent strings to avoid detection. We are recording that as an unverified user report rather than repeating it as fact, because we have not reproduced it and neither has anybody else publicly with a method attached.

The important thing is that you do not have to resolve this argument to act on it. Whether a given crawler honours your policy file is an empirical question about your own server, and your access logs answer it directly. Filter for the user-agent, compare the requested paths against your Disallow rules, and check the source IPs against the vendor's published ranges. If a crawler is fetching paths you disallowed, you have your answer and it is specific to you.

This is a recurring theme in this whole subject. Almost every disputed claim about AI crawlers is settled faster by looking at your own logs than by reading another blog post about it, and almost nobody does.

The clean version of this post's thesis would be: block GPTBot freely, it costs you nothing, just leave OAI-SearchBot alone. That version is wrong, and the correction came from the comments of the very thread that first made the distinction well.

There are two ways your brand can turn up in a ChatGPT answer. The retrieval path finds you, quotes you and links you, and OAI-SearchBot governs it. The other path is the model simply knowing about you, because your content was absorbed during training, and naming you without a link or a fetch. GPTBot feeds that path.

The second path is worth less per appearance, because there is no link and no referral traffic. It is also close to unmeasurable from where you sit, since you cannot distinguish a model recalling you from a model retrieving you by looking at your own analytics. And it is slow, because it only updates when models are retrained.

But it is not nothing. Blocking GPTBot is a real trade: you keep the measurable retrieval channel and give up a slower, weaker, unmeasurable route into answers that never fetch a page at all. That is a defensible decision. It is not a free one, and anybody telling you it is free is skipping the part they cannot measure.

Note the asymmetry, which is the actually useful conclusion. Blocking OAI-SearchBot costs you something specific, immediate and measurable. Blocking GPTBot costs you something diffuse, delayed and unmeasurable. If you are going to make one of these decisions carelessly, make it the second one.

The short version is that you allow the search and retrieval agents and you make a separate, deliberate decision about the training agents.

For OpenAI specifically, verified against its own documentation on 2026-09-09, that means allowing OAI-SearchBot if you want to appear in ChatGPT search answers, and allowing ChatGPT-User if you want a person who points ChatGPT at your page to get a real answer. GPTBot is your training decision. OAI-AdsBot only matters if you buy ChatGPT ads.

For other vendors, and this is the part where we are deliberately not giving you a copy-paste block, verify each user-agent string against that vendor's own current documentation before you write it into a file. The rosters change. OpenAI added a fourth crawler that zero of 72 surveyed companies had heard of. A list of user agents published in a blog post six months ago is a list of the crawlers that existed six months ago, and a robots.txt full of stale agent names is exactly how the 14-site probe found sites naming no user-agent introduced in 2025 or later.

That two-minute check is worth running right now, before the rest of this post. Open your own robots.txt in a browser, search the page for GPTBot, then search it for OAI-SearchBot. Four outcomes:

• Neither appears. You have made no decision about OpenAI at all. That is fine, and it means you are citable by default, assuming no other layer is refusing.
• Both appear, both allowed. You have made a decision and it is coherent.
• Only GPTBot appears, disallowed. You have opted out of training and remain fully citable. This is the most common shape and it is usually what people meant, though rarely what they think they did.
• OAI-SearchBot disallowed. You have switched off ChatGPT search citations. If that was deliberate, fine. If you cannot remember doing it, this is the finding.

And then the caveat that the first three quarters of this post exists to establish: none of those four outcomes tells you what your server actually does. The file is a statement of intent. The audit below is a measurement.

The two-minute check reads a policy file. This one measures behaviour, and it is the only version that can catch a block sitting below robots.txt.

The design is lifted from the 14-site origin probe, and its two controls are what turn a list of status codes into evidence.

Probe your origin once per named user agent, then add the controls:

for ua in "OAI-SearchBot" "GPTBot" "ChatGPT-User" "PerplexityBot" "ClaudeBot" \
          "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/126 Safari/537.36" \
          "totally-invented-agent-9931"; do
  curl -L -sS -A "$ua" -o /tmp/probe.html \
    -w "%{http_code}  %{size_download} bytes  $ua\n" \
    https://your-site.example/
done

Read the result as three comparisons rather than seven numbers.

Named crawler against browser control. If a named AI crawler gets a 403 while the browser gets a 200, that crawler is being refused specifically. This is the finding the whole audit exists for.

Named crawler against invented-agent control. If the invented agent is also refused, you have generic bot protection rather than an AI-crawler decision, and the fix is a different conversation. If the invented agent sails through while named crawlers are refused, somebody deliberately configured a blocklist, and it is worth finding out who and when.

Byte counts against each other. This is the check almost nobody runs and it catches a completely different failure. If your crawler responses are 200 but dramatically smaller than the browser response, your content is being assembled client-side and the crawler is receiving a shell. The 11-site test found two sites serving more than twice as many words after rendering as they had in raw HTML. A crawler that gets a 200 and an empty page is refused just as effectively as one that gets a 403, and your robots.txt looks perfect throughout.

Compare the result against your robots.txt afterwards, not before. If they disagree, the server wins, and your policy file is documentation of an intention rather than a description of your site.

A post arguing that crawler configuration decides your citations owes you the sample that found nothing wrong.

A practitioner tested 11 recent sites with four AI crawler user agents. All 11 returned HTTP 200 to OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. None served a block page. His conclusion was that content delivery was the more useful check, because two of the remaining ten sites had more than twice as many words after rendering as in raw HTML.

The same run recorded three other checks across those sites: an llms.txt file present on 6 of 11, a valid sitemap on 11 of 11, and structured data on 10 of 11. He also disclosed that he operates the scanning tool involved and stated plainly that this is a small opt-in sample and not a prevalence estimate. That caveat is the reason the thread is worth citing at all, and it is a better standard than most published crawler statistics meet.

Take the honest reading. On most sites, no AI crawler is blocked, robots.txt is not the problem, and the reason you are not being cited has nothing to do with any of this. It is far more likely to be that your content renders client-side, or that it does not answer the question anybody is asking, or that there is nothing on the page worth quoting.

Crawler access is a floor, not a lever. Check it once, fix it if it is broken, and then go and work on the thing that actually decides citations, which is whether your page contains a specific, quotable, sourced answer to a question a person is asking.

No, and this is the point where most posts on this subject quietly change the claim they are making.

Everything above establishes a negative: disallowing OAI-SearchBot prevents you from appearing in ChatGPT search answers. That is a claim about a blocker. It says nothing about what happens when you remove it.

The positive version, that allowing the crawler causes citations, is a completely different claim and there is no public controlled evidence for it. Access is necessary and nowhere close to sufficient. Every site that gets cited allows the crawler, and the overwhelming majority of sites that allow the crawler are never cited.

We are in a position to be specific about why we will not give you a number here, because we have measured the thing that stops us.

We ran a step-zero measurement on our own query set before running anything else: 12 money queries, 5 identical repeats each, one model, one day, nothing changed between repeats. All 60 calls succeeded.

Seven of the twelve queries changed their answer between identical asks.

That is the noise floor of the instrument everybody in this market reports numbers from. If more than half your queries return a different answer when nothing at all has changed, then a single before-and-after reading cannot distinguish an intervention from ordinary variance. Most published AI-visibility results are exactly that: one reading before, one reading after, and a percentage in between.

We then ran a permutation test on the same data. Twenty thousand random splits of the query set with no intervention applied at all, which produces the distribution of differences you get from pure noise. That null distribution centres on zero, mean plus or minus 0.0016, standard deviation 0.215. Any real effect has to be distinguishable from that spread before it means anything.

The number that governs this post is what falls out of it. The minimum detectable lift on that 12 by 5 design is 51.1 percentage points. Our own instrument is roughly four times underpowered to detect any effect smaller than that, and we published that figure rather than burying it.

So when somebody tells you that unblocking OAI-SearchBot produced a 30 percent citation lift, the question is not whether they are honest. The question is what their design could have detected. A 30 percent result from an instrument with a 51 point detection floor is not a finding, it is noise with a decimal point.

We hold no causal-lift figure for crawler configuration. We are not going to produce one until we have a design that could survive being wrong, and this post would be dishonest if it implied otherwise while selling a service on the back of it.

This post cites four other people's counts. All four are useful. None is an industry statistic, and the difference matters enough to make explicit.

Five questions to ask of any figure in this space, including ours:

What is the n, and is it stated? A percentage without a denominator is not a measurement. "17 percent of sites block GPTBot" means something quite different at n of 109 than at n of 100,000, and the 109-site figure in this post is exactly that: 109 sites, one person, one afternoon.

Who counted, and do they sell something? Two of the accounts in this post disclose operating scanning tools. That does not make their numbers wrong. It does mean the sample was assembled by somebody with a commercial interest in the finding, and it should be attributed by name rather than laundered into "research shows".

Did anybody probe the server, or only read robots.txt? This is the single biggest quality divide in the available evidence. The 109-site and 72-company counts read policy files. The 14-site probe measured behaviour and found the two disagreeing. A robots.txt-only count cannot see a server-level block, so it systematically undercounts refusals.

Were there controls? The 14-site probe included a browser control and an invented user-agent that appears on no blocklist. Without those two you cannot tell an AI-crawler decision from generic bot defence, and most published crawler audits have neither.

Is it presented as a sample or as a rate? Every number in this post is a sample. We have said so at each one. When you see the same figures repeated elsewhere in three months as "17 percent of the web blocks GPTBot", that will be a sample that got promoted to a rate somewhere along the way, and the promotion will have happened without any new measurement.

Take a developer-tools company with docs, a blog and a marketing site, spread across two hosts and a CDN. This is the shape where the failure in this post is most likely, because the layers multiply.

Step one, fetch the live robots.txt over the network for each hostname, not from the repository. Docs subdomains routinely serve a different file from the apex, generated by a docs platform nobody on the team configured. This alone finds a surprising number of surprises.

Step two, run the probe loop against each hostname separately. A marketing site on one platform and docs on another are two different origins with two different sets of rules, and citation-worthy content is usually in the docs.

Step three, compare byte counts, not just status codes. Documentation platforms are the worst offenders for client-side rendering. A docs page that returns 200 with 4KB to a crawler and 90KB to a browser is invisible in exactly the way a 403 is invisible, and it will never show up in a robots.txt audit.

Step four, check the CDN rules separately from the origin. After 2026-09-15 this stops being optional for anyone onboarding a new domain to Cloudflare, and a bot-management rule set can refuse a crawler without the origin ever seeing the request.

Step five, write down what you decided and why. The 14-site probe found two sites whose hand-written robots.txt allowed a crawler their server refused. Somebody made a decision, recorded it in the wrong place, and never checked that it took effect. A one-line comment saying which layer owns this decision would have caught it.

We read the top ten United States results for oai-searchbot on 2026-09-08, plus the top ten for three sibling queries, and it is worth describing what is there because it explains why this post exists.

Ten distinct domains rank for the head term and no domain appears twice, which means there is no incumbent. The composition breaks down cleanly.

Four are bot directories. DataDome, Cloudflare Radar, CrawlerCheck and one practitioner-run user-agent index. These are templated pages, one per crawler, generated at scale. They will tell you the user-agent string, the IP ranges, and sometimes a blocking rule. They are accurate and they are useful for exactly one thing, which is looking up a string. None of them says a word about what blocking the crawler does to your visibility, because that is not what a directory is for.

One is first-party. OpenAI's own crawler documentation holds position one, and deservedly. It is complete, it is current, and it is the source every load-bearing claim in this post rests on. It is also deliberately silent on the marketing consequence, because OpenAI is documenting a product, not advising you on your funnel.

One is a news item about a crawler update. One is an agency glossary entry. One is a vendor comparison page, which is the closest thing on the SERP to a competitor for this post. One is vendor documentation from an adjacent tool.

And one is a Reddit thread, at position ten, which carries the actual thesis.

That is the gap in a sentence. The definitional half of this question is owned outright by first-party documentation and there is no point competing with it. The consequential half, what the choice does to you, is currently answered best by a self-post with four upvotes.

The sibling queries sharpen it. On block gptbot, a Reddit thread ranks first, above Vercel, above Neil Patel and above OpenAI's own docs. On that sub-intent Google is preferring a discussion to a document, which is a reasonable signal that people arriving there want a judgement call rather than a specification.

This is not a criticism of the directories, but it is worth being explicit about their structural limit, because it is the same limit that makes most crawler content useless.

A directory entry is generated per user agent. Its unit is the crawler. It can tell you everything about GPTBot in isolation and it can tell you everything about OAI-SearchBot in isolation, and it structurally cannot tell you the one thing you need, which is what happens when you treat them as interchangeable.

The decision you are actually making is never about one crawler. It is about a set of them, made once, usually in a hurry, often by copying a block from a blog post. The failure mode lives in the relationship between the entries, not inside any one of them.

That is also why "should you block AI crawlers" is a badly formed question, and every guide answering it is answering something you did not ask. There is no single answer because there is no single crawler. There is a training decision, a retrieval decision and a live-fetch decision, and they have different costs, different reversibility and different measurability.

If your buyer is a developer evaluating tools, three things about this get sharper.

Your documentation is the asset, and it is the most likely thing to be misconfigured. Developers ask assistants comparative and procedural questions, and the answers get built out of reference documentation and quickstarts rather than marketing pages. Documentation platforms are also the most common source of client-side rendering and of a separately generated robots.txt that nobody on your team wrote. That combination is exactly the failure this post describes, sitting directly on top of your most citable content.

Check the docs subdomain as its own origin. We keep repeating this because it is the single highest-yield check for this buyer. Your apex domain being clean tells you nothing about docs. on a different platform behind different CDN rules.

Access still is not the hard part. Being reachable and being worth quoting are separate problems, and for developer tools the second one usually comes down to whether your docs contain a specific, copyable, correct answer to the question somebody actually asked. We looked at the adjacent version of this question, whether community activity moves what an assistant recommends to developers, in does Reddit marketing move AI API recommendations, and the honest answer there has the same shape as the honest answer here.

Five patterns worth checking for specifically, all of which we have either seen described with a method or which follow directly from documented behaviour.

A wildcard user-agent block with a narrow allow list. User-agent: * followed by Disallow: / and then a handful of explicitly allowed agents is a deny-by-default posture. Every crawler introduced after you wrote it is refused, which is how a site ends up refusing a crawler that did not exist when the file was authored. OAI-SearchBot and OAI-AdsBot are both recent enough for this to bite.

A generated file you never wrote. Plugins, docs platforms and site builders generate robots.txt. The file in your repository may not be the file being served, and the only way to know is to fetch the live URL over the network.

Group inheritance misunderstandings. Rules under User-agent: * and rules under a named agent do not combine the way most people expect, and search engines differ in how they merge groups. If a named crawler has its own group, it typically stops reading the wildcard group entirely, so an allow you thought applied globally may not apply to it.

Stale agent names. The 14-site probe found that not one of the sites it examined named a user agent introduced in 2025 or later. A robots.txt is a snapshot of the crawler landscape on the day somebody last edited it.

A decision recorded in the wrong layer. The strongest single finding in this post: two of fourteen sites explicitly allowed, in a hand-written robots.txt section, the exact crawler their own server refused. The decision was real, deliberate and recorded. It just had no effect.

It is the floor, and this post has been at pains not to oversell it. It is worth saying where the rest of the work actually is, since that is the question you have after fixing this.

The thing that decides whether you get cited, once you are reachable, is whether your page contains a specific, attributable, quotable answer. We wrote the general version of that argument in what answer engine optimization actually is.

Two of the most popular technical levers in this market have been tested and do not survive contact with a control. We looked at whether an llms.txt file moves anything in does llms.txt move AI citations, and at whether structured data does in isolating schema markup as a variable. Crawler access belongs in the same family of questions, and it is the one member of that family with a documented, first-party, mechanical answer, which is unusual and is why this post can be more definite than those two.

The measurement problem underneath all of it is the one worth internalising. If you take one thing from this post other than the crawler distinction, make it the number from why a single AI visibility score is noise: seven of twelve identical queries changed answer between identical asks. The procedure for measuring this properly, rather than reading a dashboard, is in how to track AI visibility.

If you want somebody to run the origin probe and establish the noise band on your own queries, that is the measurement half of what we do at answer engine optimization.

In the first hour. Fetch your live robots.txt over the network for every hostname you own, including docs and any subdomain serving content. Search each for GPTBot and OAI-SearchBot. Write down which of the four outcomes above you are in. This is ten minutes and it tells you whether you have a decision recorded at all.

In the first week. Run the probe loop against each origin with the browser control and the invented-agent control. Compare status codes and byte counts. If your CDN has AI bot management, read its rules as a separate artifact from robots.txt. Reconcile all three, and where they disagree, write down which layer you intend to own the decision. This is the step that finds the failures the hour-long version cannot see.

Once a quarter. Re-run the probe, because crawler rosters change, CDN defaults change, and plugins update themselves. Cloudflare is changing defaults on 2026-09-15 and that is a public, dated example of the class of change that will keep happening. Check your vendor documentation for new user agents at the same time. OpenAI has four crawlers today and had fewer not long ago.

That is the whole operational content of this subject. It is genuinely small, and the reason it deserved this many words is not that the fix is complicated but that almost everything written about it is answering a different question than the one you have.

Two claims here are dated, load-bearing, and were flagged as unverified in our own research notes before drafting. Both were checked at primary source on 2026-09-09 rather than taken from the thread that surfaced them. We are showing the work because the checking is the part most posts skip.

Claim one, the crawler independence. The research notes carried the distinction from a practitioner thread. A practitioner thread is a hypothesis, not a fact about a vendor's product. We fetched OpenAI's crawler documentation directly, confirmed all four user-agent strings were present in the served document, and ran a control term to prove the extractor was reading real page text rather than an empty shell, which matters on a page that renders client-side. The independence sentence and the "will not be shown in ChatGPT search answers" sentence are both quoted verbatim from that page. The distinction is not a community theory. It is vendor-documented behaviour.

Claim two, the Cloudflare date. The thread that surfaced this described a 15 September change blocking "Agents" traffic by default on ad-bearing pages, and stated the default would apply to new sites and free-tier accounts. We fetched Cloudflare's own announcement and the substance held, but the paraphrase was wrong in a way worth correcting. Cloudflare's text says the new defaults apply "for all new domains onboarding to Cloudflare", and separately notes that the controls are available to every tier including Free. Those are two different statements and the thread had merged them into one. The date, the three-way classification, the ad-page scope and the fact that Search remains allowed are all confirmed at source.

That second one is a small correction and it is exactly the kind that propagates. A slightly wrong paraphrase gets quoted, then quoted again, and three months later "Cloudflare blocked AI crawlers on free accounts" is something everybody knows and nobody checked.

The byte counts in that table are there so the check is reproducible rather than asserted. The OpenAI crawler page returned 318,282 bytes, the Cloudflare announcement 535,209, and the Google crawler documentation 88,967. On the OpenAI page the control term returned 103 occurrences, and on the Google page the control returned 33, which is what makes the crawler-name counts beside them meaningful rather than an artefact of a failed read.

The redirect detail in that table is worth one sentence, because it is the kind of thing that silently produces a wrong answer. Our first fetch of the OpenAI page wrote two URLs into one output file, so the extractor ended up reading a 77-byte redirect stub rather than the 318,282-byte document. Every search term returned zero. A zero from a failed read and a zero from a genuine absence look identical, which is why every fetch in that table carries a control term that has to appear.

We are not the only people who found the policy-versus-reality gap. A practitioner running client audits kept seeing sites that block AI crawlers at the CDN or firewall level while their robots.txt welcomes them, and built a free tool to test the two against each other across the OpenAI, Anthropic and Perplexity agents. That is the same finding as the 14-site origin probe, reached independently, by somebody looking at a different population.

That tool post carries 36 favourites, 10 reposts, 45 bookmarks and 3,725 views, which is small, and it is cited here for what the author observed rather than for how many people agreed with him.

Separate but relevant, and it cuts against the alarmist reading of all this: one practitioner instrumenting roughly 6,000 AI bot requests across 4 domains reports no discernible change in request volume from the crawlers he intercepted, and concludes that AI crawlers can read content perfectly well. He also observed that OAI-SearchBot, once served, can either call the API itself or hand the session over to GPTBot, which is a behavioural link between the two agents that no vendor document describes. We have not reproduced either observation and record them as his.

For the mechanics themselves, rather than the consequence, the clearest vendor-neutral explainer we found is an 8 minute Ahrefs walkthrough of robots.txt, GPTBot and llms.txt, at 6,791 views. It is good on what the agents are and it does not answer which one decides citation, which is the gap this post fills.

What we could not confirm, stated as absence rather than as zero. Our research pass did not measure AI Overview presence for any of these queries, nor the People Also Ask panel, because the search tooling used returns organic results only and no browser pass was run. The queue entry for this post carries zero AI Overview trials. So we make no claim about whether this page can earn an AI Overview citation, and if you see that claim made about a page like this one, ask who looked at the panel.

We also did not re-derive any of the four practitioner counts. We read what each author said about their own method and reported it with the n attached. Where an author disclosed a commercial interest, we said so. Where two of them disagreed, which happened between the 72-company count and the 109-site count, we reported the disagreement rather than picking the more convenient number.

Everything above about crawler prevalence belongs to somebody else. One number is ours, and it is the reason this post has no lift figure in it.

Twelve queries, five identical repeats each, one model, one day, nothing changed between runs. Sixty of sixty calls returned. Seven of twelve queries produced a different answer on identical asks.

We publish that alongside the permutation test that gives it context: twenty thousand random splits of the same query set with no intervention at all, producing a null distribution centred on zero with a standard deviation of 0.215. And we publish the consequence, which is that the minimum lift this design could detect is 51.1 percentage points.

It is worth being clear about what that figure is and is not. It is not a claim that AI citations are random. It is a claim about a specific instrument on a specific query set on a specific day, and a different design with more repeats would have a lower floor. What it establishes is narrow and it is enough: on this kind of measurement, a single before-and-after reading is not evidence of anything, because the same question asked twice with nothing changed already disagrees with itself more than half the time.

That is why this post will tell you with confidence that blocking OAI-SearchBot stops ChatGPT search citing you, which is a documented mechanical fact, and will refuse to tell you what unblocking it is worth, which is an empirical question nobody has answered properly including us.

The distinction between those two kinds of claim is most of what separates useful writing in this field from the rest of it.

"This is all going to be obsolete when the crawler names change." Partly, and the post is built to survive it. The user-agent strings will change, and OpenAI has already added one that almost nobody has heard of. The structure will not: there will continue to be a crawler that builds a retrieval index, a crawler that collects training data, and an agent that fetches live on a person's behalf. Cloudflare has now built that same three-way split into its product. Learn the structure, look up the current strings.

"Nobody actually blocks OAI-SearchBot, so this does not matter." Largely true, and this post says so twice. Seven percent of one 109-site sample disallowed it, and zero did so accidentally. The reason to read your own configuration is not that the disaster is common. It is that the check costs two minutes, the failure is silent, and you currently do not know which case you are in.

"You are telling me to allow a crawler and then telling me it will not do anything." Yes. Those are both true and they are not in tension. Removing a blocker is necessary and not sufficient. The honest framing is that crawler access is a hygiene item you fix once and then stop thinking about, not a growth lever, and a post that sold it as a growth lever would be the thing this post is arguing against.

"Your own numbers are from twelve queries." They are, and the post says so, and that is precisely why we report a detection floor rather than a lift. A small sample honestly described is more useful than a large one whose method is unstated. The 12 by 5 design is enough to establish that the noise exists and roughly how large it is, and it is explicitly not enough to measure an effect. Both halves of that are published.

"The Reddit threads you cite are tiny." They are. Of the five embedded here, the largest has 7 upvotes and two have zero. Our own topic-radar gate requires 50 upvotes and 10 comments for a thread to count as corroboration, and zero of the 25 threads we pulled cleared it. We are citing them as evidence that specific people counted specific things, attributed by name, date and n, and the load-bearing claims in this post rest on first-party vendor documentation instead. That is the right way round.

Structurally yes, and this is the part that makes the effort worth it, because once you have the shape you can apply it everywhere instead of memorising strings.

Every major assistant that can cite a live page needs the same three capabilities. It needs to build a retrievable index of the web, which is a scheduled crawl. It needs to fetch a specific page on demand when a person asks about it, which is a live request with a human waiting. And whoever trains the underlying model needs a way to collect training data, which is a third crawl with entirely different purposes and entirely different consequences for you.

Because those three jobs are genuinely different, vendors that take webmaster controls seriously end up exposing them separately. OpenAI does it with four named agents. Cloudflare, coming at it from the infrastructure side rather than the model side, arrived at the same three-way split and productised it as Search, Agent and Training, which is the convergence this post keeps pointing at.

Google is the clearest confirmation that this is not an OpenAI quirk. Its crawler documentation states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal, and Google-Extended is the token that controls whether your content is used for Gemini and for grounding. So Google has built the same separation OpenAI has: one control for training and model grounding, and no effect on the search surface that sends you traffic. We read that sentence at the live document on 2026-09-09.

Note the mechanical detail there, because it is a trap of its own. Google-Extended has no separate HTTP user-agent string at all. Crawling is done with existing Google user agents and the robots.txt token exists purely as a control. So you cannot probe for Google-Extended the way you can probe for GPTBot, and a server-log check will not show it to you. That is a case where reading robots.txt genuinely is the only available answer, which is the exact opposite of the OpenAI situation this post spends most of its length on.

So the transferable procedure is this, and it does not depend on any string staying current:

For each assistant you care about, find its vendor documentation and identify which agent does which of the three jobs. Not a blog post listing user agents, and not a bot directory. The vendor's own page, because it is the only source that is both authoritative and updated when the roster changes.

Allow the search agent unless you have a specific reason not to. That is the one governing whether you can be cited, and blocking it has an immediate, specific, measurable cost.

Decide about the live-fetch agent based on whether you want a person who points the assistant at your page to get a useful answer. For most sites that is an easy yes.

Treat the training agent as a genuinely separate policy question, with real considerations on both sides that have nothing to do with retrieval, and note that the cost of blocking it is deferred and unmeasurable rather than zero.

Then verify all of it against your own origin, because everything above describes intent and none of it describes what your server does.

What you should not do is copy a robots.txt block out of any article, including this one, which is why this post does not contain one. A pasted block is a snapshot of somebody else's decision about a roster that has already changed, and it is how sites end up with hand-written rules that name no agent introduced in the last two years.

A published controlled test on crawler access. Take a set of pages currently refusing OAI-SearchBot at the server, unblock half, hold half back, and measure citation rates on a pre-registered query set with enough repeats to clear the noise band. Nobody has published this. It is entirely doable and it would settle the sufficiency question this post refuses to answer.

A counted sample that finds accidental OAI-SearchBot blocks. The 109-site count found zero. A larger sample, or one drawn from a different population such as sites using a particular CMS or security plugin, could easily find them. If it does, the scare story becomes true for that population and we will say so.

Evidence that GPTBot access measurably affects unlinked brand mentions. The deferred-cost argument in this post is reasoning, not measurement. If somebody demonstrates that training access moves how often a model names a brand without citing it, the trade-off calculation changes and blocking GPTBot gets more expensive than we have described.

OpenAI changing the independence guarantee. The entire post rests on one documented property: each robots.txt setting is independent. That is a vendor statement about a vendor product and it can change. It is worth re-reading the source page rather than trusting this one after enough time has passed.

Blocking GPTBot does not stop ChatGPT citing you. OAI-SearchBot does that job, OpenAI says so in its own documentation, and the two settings are independent by design.

The accident everyone warns about, blocking the citation crawler by mistake, did not occur once in the only sample where somebody counted. The common failure runs the other way: teams who believe they opted out of AI and only opted out of training.

And the thing that will actually catch a problem on your site is not reading your robots.txt, because in twelve of fourteen probed cases the refusal was not in that file at all. Probe your own origin with each named agent, include a browser control and an invented one, compare byte counts as well as status codes, and check your CDN separately.

Then fix whatever that finds, once, and go back to the work that actually earns a citation. Access gets you into the room. It does not get you quoted.