Prompt Tracking, How Many Prompts and How Many Repeats Before a Change Is Real

Every page that ranks for prompt tracking will teach you how to choose prompts. Pick your money questions, mix branded and unbranded, cover the buying stages, group them by topic. It is decent advice and most of it is correct. Then the article ends, and you are left holding a list of questions and no idea how long it should be.

Nobody tells you how many. Nobody tells you how many times to ask each one. And nobody tells you how large a week-over-week change has to be before it means anything, which is the only reason the first two questions matter at all.

We ran the cheapest version of that check on our own work and published the numbers. Twelve money questions, five identical repeats each, one model, one day, nothing changed between runs. Sixty calls went out and sixty came back. Seven of the twelve questions changed outcome between asks that were identical by construction. Not between Monday and Friday. Between two asks a few seconds apart with the same words.

Fifty eight point three percent of our set disagreed with itself. That single result is the reason this post exists, because if more than half your prompts flip under repetition, then the number on your dashboard is a coin landing on a decimal point and the size of your prompt set is not a style preference. It is the thing that decides whether the number can answer any question at all.

This post is the arithmetic. How many prompts, how many repeats, how the two trade against each other inside a fixed budget, and how to tell whether the design you can afford could have detected the result you are hoping for. It is not a review of tools and it names no vendor as better than another. It is the sizing step that comes before you buy anything.

The number you see most often is eight to twelve. Pick eight to twelve real buyer questions, run them across a few assistants on a schedule, watch the trend. It appears in practitioner threads, in vendor onboarding, in the getting-started section of most tools in this category. It is repeated because it is reasonable on its face: twelve questions is enough to feel like coverage and few enough that a person can read every answer.

The trouble is that eight to twelve is a claim about variance, and nobody making it has stated whose variance. A prompt set is large enough when the noise it produces is smaller than the change you want to see. That is one sentence and it has two unknowns in it, and neither of them appears anywhere in the advice.

Here is the falsification, using our own published figures. A design of twelve prompts asked five times each has a minimum detectable lift of 51.1 percentage points. That is the smallest change the design could reliably tell apart from nothing. A programme that takes a brand from being cited in a fifth of answers to being cited in a third has moved by about 13 points, which is a good quarter by any standard, and this design cannot see it. It will report something that looks like noise, and if you read that as a negative result you will cancel work that was in fact working.

Now go the other way. Eight prompts asked once a week with a single repeat each is eight calls per reading. That design has no repeat structure at all, so no prompt carries a spread, so there is no denominator under any number in it. It is not an underpowered experiment. It is not an experiment.

The picture below is one query asked five identical times, with the outcome moving between runs. It is the entire argument for repeat counts in a single frame, and it is drawn from the shape of our own step-zero run rather than from a hypothetical.

Where the recommendation comes from, and why it is not stupid

It is worth being fair about this, because the eight to twelve number is not lazy. It comes from a real constraint. Somebody has to read the answers. If a human is going to look at every response and judge whether the brand was described accurately, twelve questions times a few assistants is already a morning's work, and a hundred is not happening.

That is a genuine limit and the recommendation is a sensible response to it. The failure is not the number. The failure is that the number was chosen for one purpose, human review, and then reused for a completely different purpose, statistical comparison, without anyone checking whether it survives the change. It does not. Reading twelve answers carefully and detecting a ten point movement are different jobs with different sample sizes, and a set sized for the first cannot do the second.

You can see the same slippage in the published material. Semrush documents how to add and organise prompts in its tracking product. Moz explains what prompt tracking is and why a brand should care. Search Engine Ranking has a whole article on how to choose prompts to track. All three are useful and none of them states a prompt count tied to a detectable effect, because the question they are answering is selection and the question here is size.

Before any arithmetic, the ambiguity has to go. When somebody asks how many prompts they need, they are asking one of three questions, and the three are sized by completely different things.

The first is coverage. Which of the questions my buyers ask am I absent from? This one is cheap. Absence does not need a tight interval, because a zero across five repeats is a zero. You can answer it with a small set and one pass, and the output is a content queue, which is a real and useful deliverable.

The second is reliability. Is this one prompt's number worth reading at all? This is a repeat-count question and nothing else. Adding prompts does not make any individual prompt's number better. Only asking that prompt more times does.

The third is causality. Did the work I did move anything? This is the expensive one, and it is the only one of the three that supports a spending decision. It needs prompts, repeats, a group of prompts you deliberately did not work, a fixed window, and a distribution to read the difference against. Almost every argument about prompt counts is really an argument about somebody wanting the third answer at the first answer's price.

The confusion is not accidental and it is not anybody's fault in particular. All three produce a percentage, all three go on the same slide, and all three get called visibility. But the first is a description, the second is a property of the instrument, and only the third is a result. We wrote the full instrument out in the procedure and proof runbook for tracking AI visibility, and this post is the sizing chapter of it.

Do not take a prompt count from a post, including this one. Measure the number that every sizing decision divides by, which is how much your own prompts disagree with themselves when nothing has been done to them.

The probe is small. Take ten to fifteen prompts you already care about. Ask each of them five times, identically, on one model, on one day, with no changes between runs. Record the outcome each time. Count how many prompts produced a different outcome across their own repeats.

That count over the total is your instability rate. Ours was seven over twelve, which is 58.3 percent. It cost sixty calls and about twenty minutes of wall clock, and it is the single most useful number produced in the first month of any measurement programme.

Three things follow immediately from that number, and none of them is available to a team that skipped the probe.

The first is a floor on repeats. If most of your prompts flip, one ask per prompt is not a measurement, it is a sample of size one from a distribution you have not characterised. Five repeats is the point at which a proportion becomes computable at all, and it is a floor rather than a target.

The second is a sanity check on everything downstream. A high instability rate says the noise in your set is large, which means the change you need to see has to be larger still, which means your design has to be bigger. A low rate is genuinely good news and sizes the whole programme down. Either way, you now know which case you are in.

The third is a lie detector for vendor claims. If a tool reports a weekly visibility score to one decimal place off a small prompt set with a single pass, the decimal place is decoration. Your own instability rate tells you exactly how much of that number is real.

One caution about the probe itself, because it has a failure mode. Run it on one model on one day. If you spread it across two days or two model versions, you have measured instability plus drift plus whatever changed in between, and you will not be able to separate them. The probe answers one narrow question and it answers it only if nothing else is allowed to move.

Here is the part that surprises people who have been reading these dashboards for a year.

A prompt asked five times can produce exactly six numbers. Zero, zero point two, zero point four, zero point six, zero point eight, or one. That is the whole range of values available to it. There is no zero point three five. There is no zero point four seven. The instrument physically cannot express one, because a proportion over five trials moves in steps of one fifth.

So when a dashboard shows you a prompt at 47 percent, either it is aggregating across something it has not told you about, or it is rounding a number that came from a much larger denominator, or it is displaying a precision the underlying data does not have.

Now the spread. The standard error of a proportion is the square root of p times one minus p, divided by the number of trials. At the worst case, where the true rate is one half, five repeats gives 0.224. Multiply by 1.96 for a conventional interval and you get plus or minus 0.438.

Read that again. A single prompt measured five times carries an interval nearly as wide as the entire scale it is measured on. The honest reading of a prompt at 0.6 with five repeats is that its true rate is somewhere between about 0.16 and slightly above 1.0, which is to say somewhere.

Twenty repeats brings the standard error to 0.112 and the interval to plus or minus 0.219, which is a genuinely different instrument. Forty brings it to 0.079 and plus or minus 0.155. The curve flattens, which is the important structural fact: going from five repeats to ten buys you more than going from twenty to forty. Repeats stop being the cheap lever quite early.

The granularity point, which nobody mentions

Granularity and precision are different problems and repeats fix both, but they fix them at different rates and they fail differently.

Precision failing means your number has a wide interval and you know it, provided somebody computed the interval. Granularity failing means your number is quantised so coarsely that small movements are invisible even in principle. A prompt whose true citation rate goes from 0.30 to 0.38, which is a real improvement, cannot show that at five repeats. It will read 0.2 or 0.4 both before and after, and which one you get is a coin flip.

This is why aggregating across prompts helps. The mean of twelve coarse proportions is much less coarse than any one of them. But it only helps in one direction: it fixes the arm-level number and leaves every individual row on your dashboard exactly as unreadable as it was. If your reporting shows per-prompt rows, and almost all of it does, those rows are the coarsest thing in the report and they are the part stakeholders point at.

This is the part that has to be understood before any budget conversation, and it is the part the whole category gets backwards.

Repeats and prompts do not do the same job. They act on different layers of the same calculation, and no amount of one substitutes for the other.

Repeats act inside a prompt. Asking one question forty times instead of five gives you a much better estimate of that question's citation rate. It does nothing else. You still have one question.

Prompts act across the arm. The number a spending decision rests on is the difference between the mean of the prompts you worked and the mean of the prompts you held back. That difference gets tighter when there are more prompts in each arm, and it barely notices how many times each individual prompt was asked once you are past the point where a proportion is computable at all.

Say the sentence a different way. If you asked twelve prompts one thousand times each, you would have twelve exquisitely precise numbers. Split into two arms, that is six numbers on each side. Six numbers do not make a comparison, no matter how precisely each one was measured. The uncertainty in the difference is driven by how much the prompts differ from each other, and repeating them does not change how much they differ from each other.

Conversely, if you asked four hundred prompts once each, you would have four hundred extremely noisy numbers, and their means would still be reasonably stable because averaging four hundred noisy things works. The problem there is different: with a single repeat you cannot compute a proportion at all, and you cannot tell a prompt that is genuinely borderline from a prompt that is genuinely stable, which matters for everything you do next.

So the answer is not one or the other. It is that repeats have a floor and prompts have a slope. Get repeats above the floor where a proportion means something, which our own instability rate puts at five and where we would rather see ten to twenty, and then spend everything else on prompts.

There is one honest exception worth naming. If your set is genuinely tiny and cannot be grown, because you sell into a narrow category where there are only fifteen real buyer questions, then the arm-level comparison is not available to you at any budget and you should stop trying to build one. Spend the calls on repeats instead, get very tight per-prompt numbers, and report per-prompt movement with intervals attached. That is a smaller claim and it is defensible, which is more than most reporting manages.

We published a ladder of minimum detectable lifts computed from the variance in our own step-zero run. The four points are 51.1 percentage points at twelve prompts by five repeats, 21.4 at twenty by twenty, 11.4 at thirty by thirty, and 9.9 at forty by forty.

Those are the numbers, and they are the reason we do not publish a causal lift figure of our own. Our pilot was the first design on that list, which cannot see anything smaller than about half the scale.

But it is worth doing something with those four points rather than just quoting them, because the shape they make answers the question this post is about.

The obvious guess is that detection scales with the square root of the total number of calls, which is how most sampling arithmetic behaves. Twelve by five is sixty calls, twenty by twenty is four hundred, thirty by thirty is nine hundred, forty by forty is sixteen hundred. If the guess were right, you could take the 51.1 anchor and divide it by the square root of the call ratio to predict the rest.

Do it and you get 19.8 for the four hundred call design against a published 21.4, and 13.2 for the nine hundred call design against a published 11.4. The sixteen hundred call design comes out at 9.9, which matches exactly.

Two points on the line, two points off it by about 1.6 and 1.8 percentage points in opposite directions. That is the finding, and it is a small one that matters a lot: the detection floor does not depend only on how many calls you buy. It depends on how you split them between prompts and repeats. If total calls were the whole story, all four points would sit on one curve, and they do not.

Which is the same thing the previous section argued from first principles, arriving now from the other direction. Prompts and repeats are not interchangeable currency. A design of eighty prompts by twenty repeats and a design of forty prompts by forty repeats cost exactly the same sixteen hundred calls and they are not equally good at the same job.

If you want one rule out of this section, it is this. When you cannot afford the design you want, take the cut out of repeats before you take it out of prompts, as long as repeats stay above the floor where a proportion is computable. Cutting prompts costs you the comparison. Cutting repeats costs you precision on rows nobody is going to make a decision from anyway.

We should be precise about what we are and are not saying, because a small prompt set is not worthless and pretending otherwise would be its own kind of dishonesty.

A set of eight to twelve prompts, read once, buys three real things.

It buys an absence map. You learn which of your buyers' questions produce answers that never mention you. That is genuinely valuable and it does not need statistics, because a zero across every repeat is a zero and does not require an interval.

It buys a content queue. Every prompt sitting at zero is a page you have not written or a page that exists and is not being retrieved. That queue is the most actionable output of any measurement programme, and a twelve prompt set produces it perfectly well.

It buys a weekly number for a stakeholder, and this is the one that causes the damage. The number exists, it updates, it goes on a slide. Nothing about the number says how much of its movement is the instrument, and by the time somebody asks, six months of that chart is sitting in a shared drive being read as a trend.

What it does not buy is a comparison. Split twelve prompts into a worked arm and a held back arm and you have six and six. Six proportions per side, each of them quantised into six possible values at five repeats, differenced against each other, and read against a noise distribution whose standard deviation we measured at 0.215. There is no arrangement of that data that supports a claim about causation.

The practical version of this: keep the twelve prompt set if it is doing the first two jobs. Just stop presenting its weekly delta as a result, and stop letting a flat quarter from it cancel a programme. Those are two different failures and they cost real money in opposite directions.

Everything above collapses into one line of arithmetic. Prompts times repeats times engines times readings equals calls, and calls is what your provider bills you for.

Every dimension you add spends the same pool. Adding a second assistant does not cost a bit more, it doubles the whole design. Adding a weekly reading instead of a monthly one multiplies by roughly four. Raising repeats from five to twenty multiplies by four. The four numbers are not independent knobs, they are factors in one product, and the product is fixed by whatever you are willing to spend.

The picture above makes the equality visible before it makes the difference visible, which is the part that is easy to lose. Two allocations that cost identically are not equally useful, and the whole sizing question is which shape of the same spend answers your question.

Take sixteen hundred calls per reading and lay out three ways to spend them.

Wide: three hundred and twenty prompts, five repeats each. Enormous coverage. Every prompt carries a standard error of 0.224 and six possible values. Excellent absence map, no comparison.

Balanced: eighty prompts, twenty repeats each. A real working set with an honest per-prompt spread of 0.112, and forty prompts per arm after the split, which is a comparison you can defend.

Deep: forty prompts, forty repeats each. The tightest per-prompt numbers available at this budget, twenty prompts per arm, and total blindness to everything outside forty questions.

We would spend in the middle, and the middle is not free. Choosing it means genuinely not knowing what happens on the questions you left out, which should be stated in the report rather than quietly absorbed. A team whose buyers ask a genuinely wide variety of things needs a bigger budget, not a cleverer allocation of a small one.

The one allocation we would argue against in all cases is spreading a small budget across four assistants. They retrieve differently, they disagree with each other, and a number blended across them has a spread that belongs to none of them. If the budget forces a choice, sample one assistant properly rather than four of them thinly, because a well sampled reading of one supports a claim and four thin ones support nothing. That is a scoping decision and it belongs in the plan, alongside everything else in measuring answer engine optimization lift.

Abstract arithmetic is easy to agree with and hard to use, so here is the whole thing on one page with real numbers in it.

Start with the probe. Twelve prompts, five identical repeats each, one model, one day. Seven flipped. Instability rate 58.3 percent. That took sixty calls.

Now name the effect. This is the step teams skip and it is the only step that cannot be computed. What is the smallest change that would alter what you do next quarter? Not the change you hope for. The change that would make you keep funding this, or stop. For most programmes the honest answer is around ten percentage points of citation share, because that is roughly the size of a good quarter of work and it is large enough to survive a sceptical read.

Write that number down, dated, before you look at any comparison. A team that picks its acceptable effect after seeing the result will pick whatever the result was, every time, and nobody involved will feel like they cheated.

Now read the floor off the ladder. Fifty one point one at twelve by five, 21.4 at twenty by twenty, 11.4 at thirty by thirty, 9.9 at forty by forty. Your named effect is ten points. Only the last design clears it, and it clears it by a tenth of a point, which is uncomfortably tight.

Convert to calls. Forty prompts by forty repeats on one assistant is sixteen hundred calls per reading. A minimum defensible window is a baseline reading and a closing reading, which is thirty two hundred calls. If you want a mid-window integrity check that you have committed in advance not to make a decision from, call it forty eight hundred.

Check the split. Forty prompts divided into two arms is twenty worked and twenty held back. Twenty is the smallest arm we would defend, and it is smaller than we would like.

Then look at the result of the whole exercise honestly, because there is a good chance it says something you did not want to hear. If your budget tops out at four hundred calls per reading, the arithmetic has told you that a ten point effect is not detectable by anything you can afford. That is not a failure of the exercise. That is the exercise working. The alternative is spending the same four hundred calls, getting a null, and reporting it as evidence the work did not do anything, which is a much more expensive mistake than knowing in advance that you could not have seen it.

There is a version of this conversation that goes better, and it goes better because the arithmetic happened first. You walk in and say: to prove a ten point move we need this design and this many calls, here is the cost. To prove a twenty five point move we need this smaller one. Below that we can tell you where you are absent and we cannot tell you whether we caused anything. Pick. Every one of those sentences is defensible and none of them requires anybody to trust you.

Reading interval feels like a scheduling decision. It is a statistical one, and it is the one that quietly destroys the most programmes.

Extra readings do not add detection power. Power comes from the size of the design and the length of the window, not from how often you look at it. What extra readings add is chances to be wrong.

Set a threshold of two standard deviations, which against our own measured null of 0.215 puts the bar at about 0.43. At a single look, that threshold gives you roughly a five percent chance of crossing it when nothing happened. That is what the threshold was chosen to give you.

Look twice and it is about 9.8 percent. Look four times and about 18.5 percent. Read it every week for a quarter, which is twelve looks and exactly what almost every team does, and the chance of at least one false crossing is about 46 percent.

Those figures assume each look is independent, which real weekly readings are not, so the true numbers are lower than the ones above. We are stating the assumption because it is generous to the practice we are criticising and the conclusion survives anyway. The direction is not in doubt even if the exact figure is.

Here is what makes this expensive rather than merely academic. Nobody peeks neutrally. A team reads the dashboard every Monday, and the week the number happens to jump they tell the client, and the week it happens to sag somebody asks whether the strategy is working. Both of those are decisions made from a crossing that the design cannot distinguish from noise, and the fact that they are informal decisions rather than a formal stopping rule makes them worse, not better, because nothing about them is written down.

The fix costs nothing. Write the decision date on the plan before the baseline reading. Look at the data in between as much as you like for integrity checks, that failures are not piling up on one arm, that the model version string has not changed, that no prompt has started returning refusals. Do not compute the difference between arms until the window closes. The difference is the one number that is very hard to give up once you have seen it.

There is a second cadence question underneath the first one, which is how long the window itself should be. That is not a statistical question, it is a question about how long your work takes to show up in a retrieved answer, and it varies enormously by what you shipped. A new comparison page and a schema change land on different timelines. Pick the window from the mechanism, then pick the design to fit the window, and never the other way round.

A sample size is only meaningful against a threshold, and the threshold is the part most teams import from somewhere else instead of computing.

We computed ours. Take the same prompt set, apply no intervention at all, throw away which arm each prompt was really in, reassign the labels at random at the prompt level, and recompute the difference between arms. Then do it twenty thousand times. What comes out is the distribution of differences that your own instrument produces when nothing whatsoever has been done.

Ours centred on zero, with a mean of plus or minus 0.0016 and a standard deviation of 0.215.

The zero mean is the kill test. If that distribution had not centred on zero, the instrument would be manufacturing a difference out of nothing and every result it ever produced would be void. It centred on zero, so the instrument is not lying to us, which is the least a measurement has to do before anyone argues about its size.

The standard deviation is the part that matters for sizing. Zero point two one five is the width of the noise. A conventional two standard deviation bar puts a real result at about 0.43, and that is an enormous number: it means a difference between arms has to be forty three points of citation share before this particular set clears its own noise.

Which brings the whole post back to one line. Your prompt set has to be large enough that the noise it produces is smaller than the change you care about, and both halves of that sentence are measurable. The noise is measurable with a permutation test on data you already have. The change you care about has to be named by a human before the fact.

Note what the permutation test needs, because it constrains the design. It shuffles at the prompt level, keeping each prompt's repeats together, which means it needs prompts to shuffle. With six prompts per arm there are very few distinct ways to reassign them, and the null distribution comes out lumpy and unusable. This is the same argument as before wearing different clothes: prompts are what the comparison is made of.

There is a failure here that deserves its own section because it is common, it is usually not malicious, and it produces exactly the movement a client is paying to see.

You start with forty prompts. Six weeks in, somebody notices that the brand does well on a whole topic area that is thinly covered in the set, so they add eight prompts covering it. Perfectly reasonable instinct. The dashboard number goes up the following week and everybody is pleased.

Nothing happened. The denominator changed. Eight prompts were added from a region of question space where the brand already wins, so the mean rose for a reason that has nothing to do with any work.

The reason this is a bias rather than merely noise is that the added prompts are never a random sample. Nobody wakes up and adds eight prompts a brand is invisible on. The prompts people remember to add are the ones somebody just saw a good answer for, which means the selection is correlated with the outcome, which means the direction of the error is always the same direction as the reported improvement.

Practitioners describe doing exactly this and reporting the result as a visibility improvement. Read as a description of the work it is honest. Read as a measurement it is a denominator change wearing the clothes of a result, and no amount of downstream statistics recovers from it, because the two arms are no longer being compared on the same set of questions.

The fix is one rule and it is free. Freeze the set at day zero. If new prompts are worth adding, and they usually are, they start a new cohort with its own baseline and their own window, reported separately. That gives you both things: the comparison stays clean and the new questions get measured. What you cannot do is quietly widen the set and keep the old chart running.

The same logic covers prompts leaving. If an assistant starts refusing a prompt, or a prompt begins returning an error, do not silently drop it. Void it, record the void, and check that the voids are not landing more heavily on one arm than the other. A set that loses four prompts from the worked arm and none from the held back arm is not a smaller set, it is a broken comparison.

We are not the only ones who have measured this, and it is worth pointing at the independent readings, because a first-party number that nothing corroborates is a weaker claim than one that lines up with other people's.

A practitioner in r/aeo posted a run of the same buying question thirty times in a row. Their reported findings: a different top brand four times out of ten, the full list of recommended brands overlapping only about half the time between one ask and the next, and the top pick still switching one time in four even after turning the randomness settings down as far as they could go. Their own conclusion was that in any contested category it is basically a coin toss.

Their own summary of it, in their words:

The only exception was markets with a clear dominant player. In any contested category, it was basically a coin toss.

That sentence is a sample-size statement wearing a different vocabulary. A coin toss is a distribution with a standard deviation, and the whole of this post is about how many tosses you need before a difference between two coins means anything.

That is a different set, a different category and a different operator, and the shape matches ours. We measured 7 of 12 prompts flipping outcome across five identical repeats. They measured a top-brand change in 4 of 10 asks. Neither of us is measuring exactly the same quantity, so the numbers should not be expected to be equal, but both say the same structural thing: a single ask is a draw, not a reading.

The most useful thing in that thread is not the post, it is the reply setting a methodology floor at twenty to thirty runs per prompt. That is a practitioner arriving at a repeat count from experience rather than from a formula, and it sits comfortably above the five we would call a floor and comfortably below the forty our own power ladder wants for detection.

The same question has been asked at much higher volume. Rand Fishkin put it directly: "If you give ChatGPT the same request for product recommendations 100X, will you ever get the same list twice?" and then, in the next line, "And what does that answer mean for folks who try to track their brand presence in AI tools?" That second sentence is the whole of this post compressed into eighteen words, and it went out to an audience of hundreds of thousands.

Now look at the other end of the range. There are tutorials teaching how to assemble prompt sets of a thousand and upward for tracking a brand across assistants.

Put the two recommendations side by side. Eight to twelve at one end, a thousand and more at the other. Three orders of magnitude apart, both offered as the sensible default, and neither one derived from a stated detection threshold. That gap is not a disagreement about statistics. It is the absence of any statistics at all, which is what makes it possible for both numbers to sound reasonable at the same time.

Neither pole is wrong for every purpose. A thousand prompts read once is a superb absence map and a terrible experiment, since at one repeat there is no proportion under anything. Twelve prompts read forty times each is a decent set of per-prompt estimates and not a comparison. The number that resolves the argument is not a prompt count, it is a detection threshold, and once you have one the prompt count falls out of it.

One more correction before the cost arithmetic, and it usually moves a design by a factor of two.

Not every prompt in your set can contribute to a change. Sort them into three groups after the first reading.

Saturated prompts are hit on every repeat. They sit at 1.0. They cannot move up, so they can never contribute to a gain, and every call spent on them is a call spent confirming something you already know. They are not useless, since they can detect a loss, but they contribute nothing to the measurement most programmes are trying to make.

Dead prompts are hit on no repeat. They sit at 0.0. In a single window these mostly stay at zero, because the work that moves a prompt off zero is usually publishing something that does not exist yet, and that runs on a slower clock than the window does. They are your content queue, which is valuable, and they are close to inert as measurement.

Unstable prompts are the ones somewhere in the middle, hit on some repeats and not others. Almost all the signal is here, and so is almost all the noise.

So if a twelve prompt set has three saturated and three dead, the working set is six, and the design is much closer to six by five than to twelve by five. The power arithmetic should be run on the movable prompts, not on the total, and almost nobody does this because the total is the number on the invoice.

Two consequences follow. First, size up rather than down, because your effective set will be smaller than your nominal one and you will not know by how much until the first reading. Second, look at the distribution at baseline and act on it. A set that comes back mostly saturated is telling you that your prompts are too easy, and a set that comes back mostly dead is telling you they are too hard or too specific. Both are fixable at set-construction time and neither is fixable afterwards. The question of which questions belong in the set at all is a separate piece of work, and for our own buyers it looks like the questions developer tool buyers ask AI.

None of this is expensive in the way people fear, and being concrete about it removes an objection that is usually standing in for a different objection.

Eight prompts at one repeat is eight calls per reading, 416 calls a year at weekly cadence. That is the design most teams are actually running, and it costs approximately nothing, which is the real reason it is popular.

Twelve prompts by five repeats is sixty calls per reading and 3,120 a year weekly.

Forty by forty is 1,600 calls per reading. Weekly that is 83,200 calls a year on one assistant. Monthly it is 19,200.

Two engines doubles all of it before anything else does, which is the argument for choosing one and sampling it properly rather than spreading thin. And notice what the table makes obvious: the honest design costs about twenty seven times the calls of the cheap one. That ratio is the actual reason the category settled where it did. It is not that nobody knows the arithmetic. It is that the arithmetic asks for a budget nobody had to approve as long as the cheap version produced a number that looked the same on a slide.

We are deliberately not converting calls into money here. Per-call pricing depends on your provider, your model and your contract, and it moves. What we will say is that the design most teams need is a low four-figure annual API bill on one assistant at monthly cadence, which is smaller than the tool subscription that is currently producing an unusable number, and that comparison is the one worth putting in front of whoever signs off the budget.

Everything above assumed one assistant. Most teams want four, so it is worth being explicit about what the second one does to the arithmetic, because it is not what people expect.

An assistant is not a dimension you can average over. Two assistants retrieve differently, cite differently, and disagree with each other on the same question. That is not a defect to be smoothed out, it is the most interesting thing in the data. A number blended across them has a spread that belongs to none of them, and when it moves you cannot say which surface moved, which is the only version of the answer that tells you what to do next.

So the correct structure is one complete experiment per assistant. Its own baseline, its own arms, its own null distribution, its own reading. Reported side by side, never merged.

That means the design multiplies. Forty prompts by forty repeats on one assistant is 1,600 calls per reading. On two it is 3,200, and every downstream figure doubles with it. The prompt set is shared, which is the one economy available, but nothing else is.

Which forces a decision most teams have never made explicitly. At a fixed budget you can have a well sampled reading of one assistant, or a badly sampled reading of four. There is no third option and the arithmetic will not be talked out of it. Four thin readings support no claim at all, and a claim about one surface is worth more than four numbers about nothing.

We would pick the assistant your buyers actually use, sample it properly, and say plainly in the report that the others were not measured. That last sentence is the part teams resist, and it is the part that makes the rest of the report trustworthy. A stated blind spot is a limitation. An unstated one is an error waiting to be found by somebody else.

There is one exception. If the purpose is the absence map rather than the comparison, spread as wide as you like. Coverage across four assistants at one repeat each is a perfectly good way to find out which questions produce answers you are missing from, on which surfaces. It is the third question from earlier, the causal one, that cannot be bought thin.

A note on model versions while we are here, because it interacts with sizing. If an assistant upgrades mid-window, the two halves of your window came from two instruments and the comparison across them is void regardless of how many prompts you had. No sample size defends against this. Record the model version string on every call, check it at the end of the window, and treat a change as a reason to re-baseline rather than as noise to absorb. The same applies when a provider changes its retrieval behaviour without changing a version string, which is harder to catch and is one reason we prefer a shorter window over a longer one.

The buyer questions themselves also differ by surface and by category, which is a set-construction problem rather than a sizing one, but it lands on the same design. What CLI and MCP buyers ask AI is not what an infrastructure buyer asks, and a set built for one and read on the other will look thin for reasons no repeat count can fix.

This will be a common outcome, so it deserves a real answer rather than a shrug. There are three honest moves and one dishonest one.

The dishonest move, for completeness, is to run the small design, get a result that does not clear the bar, and report it as though the design could have seen something. That is expensive in both directions. A false negative cancels work that was working. A false positive renews work that was not.

There is a fourth option worth naming that is not on the list because it is not a sizing decision: change what you are measuring. Citation share across a prompt set is one observable among several, and it is a noisy one. If the question is whether a specific page is being retrieved and used, that is a narrower observation with much less variance and a correspondingly smaller sample requirement. Narrower questions are cheaper to answer, which is a general property of measurement and not a trick.

Every failure below is caught by a decision that takes minutes and has to happen before the first call rather than after the first reading. That timing is the whole point. None of these is expensive to prevent and all of them are unfixable afterwards.

Three of them are worth a sentence each because they are the ones we see most.

Taking a prompt count from a post is first on the list because it is the one this whole article exists to stop. Every number in this post came out of our own instability rate, and yours will be different. If your prompts are more stable than ours, you need a smaller design than we do, and taking our number would waste money in the other direction.

Splitting a small set into two arms is second because it is the failure that hides inside a good decision. Somebody reads about held-out controls, correctly decides they need one, and applies it to twelve prompts. Now there are six per arm. The instinct was right and the set was too small for it, and the resulting comparison is worse than no comparison because it looks like rigour.

Reporting an underpowered null as a negative result is third because it costs the most. A design that cannot detect a ten point move returning no detectable move is not evidence about the work. It is evidence about the design. The output should be a resized design and a request for a bigger budget, not a recommendation to stop.

Sizing settles one question and it is important not to let it stand in for the others.

It does not tell you whether the prompts are the right prompts. A perfectly sized experiment on forty questions your buyers never ask is a precise measurement of nothing. Prompt selection is a separate discipline and the pages ranking for this term are mostly about it, which is why Conductor's explainer on prompt tracking and its neighbours are worth reading alongside this one rather than instead of it.

It does not tell you whether the assistant is behaving the same way across the window. A silent model upgrade mid-window changes the instrument, and no sample size protects against that. The only defence is recording the model version string on every call and checking it at the end. That belongs in the record schema, which we wrote out in the procedure and proof runbook.

It does not tell you what a citation is worth. Every number in this post is about detecting a change in citation share. Whether a change in citation share is worth what it cost is a question about your funnel, and this instrument does not observe a single step of that path.

And it does not tell you that a single score is meaningful, which is the argument we made separately in why a single AI visibility score is noise. Sizing makes a comparison believable. It does not rescue a number that was never a comparison in the first place.

One further limit that applies to this post specifically. The term itself is not settled. Six of the ten pages currently ranking for it sell a tracking product, and one of them, Datadog's documentation on prompt tracking, is about prompt versioning inside an observability tool, which is a different activity that happens to share a name. If you arrived here looking for that, none of the arithmetic above applies to you.

Everything above in the form you would actually keep on a page. Every figure here appears earlier with its derivation attached; this is the lookup, not the argument.

The step-zero probe. 12 prompts, 5 identical repeats, 1 model, 1 day, 60 calls. Ours returned 60 of 60 and flipped 7, an instability rate of 58.3 percent. Run it before choosing anything.

The repeat floor. At 5 repeats a prompt's hit rate has a standard error of 0.224 and exactly 6 possible values. At 10 it is 0.158 and 11 values. At 20 it is 0.112 and 21 values. At 40 it is 0.079 and 41 values. Below 5 there is no proportion at all.

The detection floors, computed from our own variance. 51.1 points at 12 by 5. 21.4 at 20 by 20. 11.4 at 30 by 30. 9.9 at 40 by 40. Pick the smallest design whose floor sits under the effect you named.

The call arithmetic, one assistant, one reading. 60 calls at 12 by 5. 400 at 20 by 20. 900 at 30 by 30. 1,600 at 40 by 40. Multiply by 52 for weekly and by 12 for monthly: 3,120 and 720 for the small design, 20,800 and 4,800 for the middle one, 83,200 and 19,200 for the large one. Two assistants doubles every figure, so 40 by 40 weekly across two is 166,400 calls a year.

The exchange rate. Going from 60 calls to 1,600 is 26.7 times the spend for 5.2 times the resolution, which is the square relationship and the reason there is no cheap way to buy detection.

The bar. Our own null across 20,000 label shuffles centred on zero with a standard deviation of 0.215, so a conventional 2 standard deviation threshold sits at about 0.43.

The peeking cost, at that threshold. About 5 percent false crossings at 1 look, 9.8 at 2, 18.5 at 4, and about 46 percent across 12 weekly looks in a quarter. Read once, at the end.

Take your own instability rate.

Everything else in this post is downstream of it. Twelve prompts, five identical repeats, one model, one day, count the flips. It costs sixty calls and it replaces every recommended prompt count you will ever read, including the ones here, with a number measured on your own questions.

Then name the smallest change you would act on, before you look at any comparison. Read the floor off a power ladder computed from your own variance. If the smallest design that clears your named effect is affordable, buy it. If it is not, say so out loud and pick one of the two honest fallbacks, a narrower claim or a larger effect size, rather than running a design that cannot answer the question and reporting its output as though it could.

The reason none of the ten pages ranking for this term states a number is not that the number is secret. It is that stating one requires measuring your own variance first, and a recommendation that begins with "run this probe on your own set" is a worse blog post and a better answer. We would rather publish the arithmetic and the places our own design falls short of it than a prompt count we cannot defend, which is also the position we take across answer engine optimization generally and everywhere else in our measurement writing. This same arithmetic is the whole answer to how you prove an AEO engagement caused a citation lift rather than coincided with one.