Measurement

Why a level reading is not a measurement

Most writing about AI visibility reports a level. A score, a share of voice, a rank, read once and pasted into a slide. The trouble starts when you read it twice.

Twelve money queries were run five times each, on one model, on the same day, with nothing changed between runs. Seven of the twelve came back with a different outcome. A single reading of where a company stands is not a weak measurement. It is a coin toss with a decimal point on it.

What survives the noise

A difference between two arms survives where a level does not. Across 20,000 random splits with no work applied at all, the null difference centred on zero, which is what makes a treated group readable against a held out control group when one number on its own is not.

Sample size then decides whether that difference means anything. A twelve by five design has a minimum detectable lift of 51.1 percentage points, which is another way of saying it can detect nothing worth paying for. Forty queries by forty samples brings that to 9.9.

The posts collected here run along both edges of that. Some take apart the tools people buy to answer this question, what they charge, and the point at which a tool on its own stops being able to say whether a rising number was you or the model changing its mind. The rest set out method: what a query has to survive before it counts, why the split between the two arms is written down before any baseline is taken, and which assumptions in that design are still unproven rather than quietly buried.