Reading an AI visibility figure honestly

Key takeaways

• Every rate is a count over a base. Ask for both before you believe the rate.
• The unit that matters is the prompt family, not the answer. Forty answers to four questions is closer to four observations than forty.
• An interval says how wide the truth could be. On a handful of prompt families it is wide enough that a rate is a direction, not a measurement.

Sooner or later someone will put a number in front of you. "Your brand is recommended in 0.75 of AI answers." It might come from a vendor, an agency, a tool or your own team. It will be stated with confidence, and it may well be computed correctly. The question is what it can support.

AI answers are a difficult thing to measure. They vary from one run to the next, they differ between assistants and over time, and the questions behind them are chosen by people. None of that makes measurement pointless. It does mean a bare number is never enough. This note sets out what to ask of any AI visibility figure, and how we answer those questions in our own work.

Throughout, we write rates as decimals between zero and one. Read 0.75 as three in every four.

A rate is a count in disguise

Every rate is a count over a base. "0.75" might mean 3 of 4 or 300 of 400, and those are not the same claim. The first is an observation. The second begins to look like a measurement.

So the first question to ask of any figure is: out of what? Then ask what the base actually contains.

• Which answers? Answers to which questions, from which assistant, collected over which period?
• Which base? A citation rate counted over all answers looks different from one counted over answers that cite any source at all. Both are defensible. They are not interchangeable.
• Which answers were left out? Answers that could not be read or classified should be reported as missing, not silently dropped or counted as zero.

A figure that cannot tell you its counts and its base is not ready to be acted on. In our own reporting every figure carries its counts beside it, so the question never has to be asked.

The unit that matters is the prompt family

Suppose a report says your brand was named in 30 of 40 answers. That sounds like a reasonable sample. Now suppose those 40 answers came from four questions, each asked ten times.

Answers to the same question, or to close rewordings of it, are not independent. If an assistant names you when asked about regulated reporting, it is likely to name you again when asked the same thing tomorrow. Ten answers to one question tell you a lot about that question and very little about your category. The honest sample size for this figure is much closer to four than to forty.

That is why we group wordings of the same intent into prompt families (see Building a buyer-question set) and treat the family, not the answer, as the unit of independence. When we compute an interval for a figure built from repeated answers, we resample whole prompt families rather than individual answers, so the interval reflects how many distinct questions the figure really rests on.

Repetition is still worth doing. Asking each question more than once shows you how stable your role is, and it smooths out single odd answers. It does not, however, turn four questions into forty.

Every figure needs an interval

An interval is a range that says, given the evidence, how high or low the true rate could plausibly be. It is the difference between "0.75" and "somewhere between 0.30 and 0.95".

For a simple rate where each prompt family contributes one answer, a standard choice is the Wilson score interval, introduced by the statistician E. B. Wilson in 1927 and widely recommended for small samples and for rates near zero or one. The table below shows Wilson intervals at the conventional ninety-five per cent confidence level, for the same observed rate on samples of different sizes.

Brand named in	Rate	Interval
3 of 4 families	0.75	0.30 to 0.95
15 of 20 families	0.75	0.53 to 0.89
30 of 40 families	0.75	0.60 to 0.86
75 of 100 families	0.75	0.66 to 0.82

The rate is identical in every row. What it can support is not. On four families, the evidence is consistent with your brand being named in fewer than a third of answers or in nearly all of them. That is not a finding. It is a hint.

The same logic applies at the other end. If your brand is absent from all 30 families you asked about, the Wilson interval still runs from zero to about 0.11. Absent everywhere you looked is strong evidence that you are rarely named, but it does not prove you never are.

Small samples should say so

Even with an interval, a figure resting on very few prompt families is easy to misread. Someone will quote the rate and drop the range.

So we flag them. When a figure rests on fewer prompt families than our method sets as the threshold for reading a rate, it is marked as a small sample before anyone acts on it, and we describe it as a direction rather than a rate. The threshold counts families, not answers, for the reason above: thirty answers from three families is not a large sample.

A direction is still useful. If your brand is named in 1 of 4 families and a rival in 4 of 4, that is worth investigating. It is not yet worth a board slide that says "we are named in a quarter of answers". Read small samples as a reason to look closer: open the answers, read the reasons and decide whether the pattern is worth measuring properly with more questions.

Comparing two figures

Most decisions rest on a comparison: this month against last, your brand against a rival, before a piece of work against after it. Comparisons need more care than single figures.

Look at the intervals, not only the rates. If last month's interval was 0.40 to 0.80 and this month's is 0.45 to 0.85, the rate has moved but the evidence has not shown that anything changed. Heavily overlapping intervals are a strong reason to hold off on conclusions.

Check that the questions are the same. A comparison between two different question sets is not a comparison of your position. It is a comparison of two question sets.

Check that nothing else changed underneath. We record four kinds of change that break a series: a change to how a figure is defined, a change to the rules that read the answers, a change to the project's settings (such as which questions or brands are tracked), and a change to how answers are collected. When any of these happens, a figure before it and a figure after it are not directly comparable, and we say so beside the figure rather than drawing a line straight through it.

Report the flat results too. If a piece of work was meant to move a figure and the figure did not move, that is a result. Leaving it out of the report turns measurement into marketing.

One more distinction belongs here. Some numbers in this field are measured: counted from answers that were actually observed. Others are modelled: estimated from assumptions, such as the size of an opportunity in a part of your category. Modelled numbers can be useful for deciding where to look. They should always be labelled as estimates, and they should never be compared with measured figures as if they were the same kind of thing.

Five questions to ask of any figure

Keep these beside you the next time a visibility number lands in your inbox, including one of ours.

1. What are the counts? Out of how many answers, and what was the base?
2. How many prompt families does it rest on? If it is a handful, read it as a direction.
3. What is the interval? If there is none, ask for one. If it is wide, believe it.
4. Is the comparison like for like? Same questions, same definitions, same collection method, no series break.
5. Is it measured or modelled? And is that labelled?

A figure that survives all five is worth acting on. A figure that fails one is not necessarily wrong. It is simply not yet strong enough to carry a decision, and knowing that is worth as much as the number itself.

Sources

• Wilson score interval, introduced by E. B. Wilson (1927), and its behaviour for small samples and rates near zero or one: Binomial proportion confidence interval, Wikipedia