Measuring ChatGPT Citations: Why an API Proxy Recovers Only a Quarter of Them

Paste the question a buyer would ask into ChatGPT and watch which sources it names. Then open your AI visibility tool and compare. For most tools the two will not match, because the tool is not reading the answer you just saw. It is reading a stand-in, usually a Google result set or an API endpoint, and a 2026 study has now put a number on how often the stand-in agrees with the real thing.

That study, Understanding ChatGPT Citations, posted on 2026-10-11, tried to recover the pages production ChatGPT cites using two external signals: Google Search results and the OpenAI Responses API. Combined, the two recovered 28.3 percent of the exact cited URLs for GPT-5.3 Instant and 26.2 percent for GPT-5.5 Instant. At the hostname level the match rose to about half, 50.3 percent and 48.7 percent. The authors are direct about what that means: external proxies recover the same hostnames more reliably than the same exact URLs.

This post is for teams buying or building an AI visibility tool for developer tools and APIs. It covers what the study measured, why a Google or API proxy misses most of what ChatGPT cites, a second problem that affects even a perfect reading, and what it takes to measure ChatGPT citations in a way you can defend to a skeptical buyer.

The study measured recovery, which is a plain question with an uncomfortable answer: if you only had Google results and the OpenAI Responses API, how many of ChatGPT's real citations could you reconstruct? For the exact URL the answer was 28.3 percent and 26.2 percent across two model generations, and for the hostname it was 50.3 percent and 48.7 percent. A supervised model that combined the two signals did better, clearing an AUC of 0.74 at every level, but that is a trained classifier, not the raw proxy a dashboard reads.

Those numbers describe the best case for a proxy, not the worst. The researchers used both Google Search and the Responses API together, where most tools lean on one. They ran it against production ChatGPT web search, the same surface a buyer sees. And they still recovered fewer than three in ten exact URLs. The paper itself, read 2026-10-19, frames this as a structural gap rather than a tuning problem: the proxy and the production answer are different surfaces that happen to overlap.

The reason the hostname figure is higher than the URL figure matters for how you read any tool. A proxy can often tell you the right site was involved while naming the wrong page on it. If your tool reports hostnames, it is working near its most reliable level. If it reports exact URLs or specific pages, it is working where the agreement is weakest.

The proxy misses because ChatGPT does not cite from the surface the proxy reads. It cites from its own retrieval and its own index, selecting pages by semantic similarity to the query before it opens any of them. Ahrefs reached the same conclusion across 1.4 million prompts: ChatGPT favours its general search index, treats the title, snippet and URL as the first gate, and the average page it cites is about 500 days old. A live Google result set is a different population than a 500-day-old index, so the two will disagree by construction.

There is a second, blunter reason, and it is larger than the first. A citation only exists when ChatGPT ran a web search. Ahrefs analysed 66.6 million free, logged-out responses from August and September 2026 and found that only 14.4 percent of them searched the web at all. The other 85.6 percent answered from the model's own memory and named no sources. A tool that scores you on citations is scoring the minority of answers that produced one, and cannot see the majority where your brand was present or absent in the prose itself.

Put the two together and the proxy faces compounding losses. Of the answers that do search, it recovers the exact cited URL about a quarter of the time. Of all answers, most never cite anything. A single score built on that pipeline is a thin reading of a narrow slice, presented as a measure of the whole.

An AI visibility tool that reads citations through a proxy is counting agreement between two surfaces, and the study says that agreement runs about 28.3 percent at the URL level and about half at the hostname level. It is not counting what ChatGPT cited. It is counting what a Google result set or the Responses API would have surfaced for the same query, which matches the real citation roughly a quarter of the time at the page level. That is not a defect in any one product. It is the ceiling the surface imposes on every product that reads it.

This is why two tools can disagree about the same brand on the same day. Each is sampling a different proxy, or the same proxy at a different moment, and both are a quarter-accurate stand-in for the answer a buyer actually saw. The honest version of the number is a confidence interval wide enough to swallow most week-over-week movement, which is a hard thing to put on a dashboard that wants to show a clean line going up. A practitioner thread from 2026-10-10 made the point from the other direction, arguing that a visibility report is a diagnostic tool, not proof, and that correlation between being cited and having traffic does not establish that one caused the other.

None of this means ChatGPT citations are unmeasurable. It means the measurement has to read the right surface and has to carry its own error bars. We keep a longer treatment of the tool landscape in our note on AI-citation measurement, and what counts, and a direct comparison of buying a tool against measured proof in an AI visibility tool versus an answer-engine agency.

Even a perfect reading of the production surface runs into a second wall, which is that the surface itself is not stable. Ask ChatGPT the same money query twice and you can get two different answers, with two different sets of citations. We measured this in our own step-zero run, written up in why a single AI visibility score is noise: 12 money queries by 5 identical repeats on one model on one day, 60 of 60 calls succeeded, and 7 of the 12 queries flipped outcome across repeats that were identical by construction. That is a property of the system, not a vendor's bug.

If 7 of 12 queries change answer with no intervention at all, a single reading is one draw from a distribution, not a score. A tool that samples each query once and plots the result as a point is drawing a trend line through noise. The movement it shows between two Tuesdays can be entirely the run-to-run variance, with nothing underneath it. This is the same failure as the proxy gap, arriving from a different direction: the first adds error because the surface is wrong, the second adds error because the surface is unstable.

We also checked that the noise was symmetric rather than hiding a signal. Across 20,000 random splits of the same query set with no intervention applied, reported in that same AI visibility score analysis, the difference between the two halves centred on zero, mean plus or minus 0.0016 with a standard deviation of 0.215. Under that null the two arms are genuinely the same even while each one jumps around, which is exactly the condition that makes a before-and-after against a control recoverable and a single level meaningless.

Defensible measurement requires four things a single score does not have: the production surface, repeats, a control, and a stated sample size. Read the answer ChatGPT returns to a user rather than a proxy that recovers the exact URL a quarter of the time. Repeat each query several times, because 7 of 12 flipped on us and yours will too. Hold out a comparable set of queries you leave unworked, so a model update or a market-wide shift moves both arms and subtracts out. And say how many calls sit behind the number, because that is what decides whether you can see a real effect or only a very large one.

The sample size is not a detail. Our 12 by 5 pilot has a minimum detectable lift of 51.1 percentage points, which means it can only distinguish a move of fifty-one points or more from noise. A real engagement moves citation share by something closer to five to fifteen points, so a design that coarse would report almost every genuine win as nothing. Getting the floor down to 9.9 points takes 40 queries by 40 samples, roughly 3,200 calls per timepoint, and you pay that cost twice because you need a before and an after. We walk through that arithmetic in why a single AI visibility score is noise and the sampling design in our note on prompt tracking and sample size.

The shape that survives all of this is a difference, not a level. You are not reporting that a brand scored 72. You are reporting that the worked arm sat some measured number of points above the held-out arm, across a stated number of calls, with an interval around it. That is a claim a skeptical buyer can check, and it is the one thing a proxy-fed single score can never become.

Before you trust a tool's trend line, ask six questions, and the first one settles most of the argument: where does it read citations from? If the answer is an API endpoint or a Google result set, you now know the agreement with real ChatGPT citations is about 28.3 percent at the URL level. Ask how many repeats sit behind each data point, because one sample per query cannot see the volatility that flipped 7 of our 12 queries. Ask whether there is a control group, because without one a market-wide move reads as your lift.

Then ask the three that expose how the number is framed. Ask whether the tool reports a confidence interval or a single figure, because a point with no interval is hiding its own error. Ask whether it distinguishes a cited answer from an answer that never searched, since only 14.4 percent of free responses search at all. And ask whether it can show you the exact pages cited or only a hostname, because the study puts URL-level recovery far below hostname-level. A tool that answers all six well is rare, and the ones that cannot answer them are selling a dashboard, not a measurement.

None of this requires you to distrust every number. It requires you to read each number at the size the method supports. A hostname-level, interval-bearing, control-backed difference is worth acting on. A single URL-level score from an unrepeated proxy run is worth about what its recovery rate suggests.

Measuring citations well is the instrument, not the work. The work is making your pages the ones an assistant reaches for when a buyer asks a question that does not contain your name, and then proving a change moved the answers. The measurement is what lets you tell a real win from run-to-run noise, which is why we treat it as the first deliverable rather than a report at the end.

What we sell is that measured chain for developer tools and APIs. Our answer engine optimization work reads citations from the production surface, repeats the queries, and measures a difference against a held-out set left unworked, and we write up how we measure the lift so the method is auditable before any number is. If you want the short version of why we refuse to hand you a single score, it is on this page.

Every recovery percentage in this post is the figure reported in Understanding ChatGPT Citations, arXiv 2610.15382, read on 2026-10-19. We did not reproduce its runs. The 14.4 percent search rate is from the Ahrefs query fan-out study of 66.6 million responses, and the citation-selection findings are from the Ahrefs 1.4 million prompt study, both read the same day.

The volatility and null figures are our own, published on 2026-07-27 with their full method. They are a 12 by 5 step-zero run and a 20,000-split permutation test, one model, one day. We state plainly what they are not: there is no completed causal-lift result here, the design is about four times too small to detect a realistic effect, and nothing in this post claims an intervention raised anyone's citations. The point is narrower and firmer. A proxy agrees with real ChatGPT citations about a quarter of the time, a single run is one draw, and a number without a surface, a repeat and a control is not yet a measurement.