Why can the same AI question produce different names?
Repeating a question does not fix the generated shortlist. On 4 August 2026, Claude's mean vendor-set Jaccard agreement across repeated answers was 0.442 over 74 pairs, while Google AI Overviews was 0.499 over 62 pairs. The study demonstrates variation within one collection window; it does not isolate an internal cause for each change of name.
What we measured
Measured. The Volume III repeat protocol used a 25-question stability subsample with three scheduled runs per question. These repeat observations were collected on 4 August 2026 between 05:28 and 16:29 UTC.
Jaccard agreement divides the number of vendor names shared by two answer sets by the number appearing in either set. Claude's mean was 0.442 across 74 usable pairs, with a published bootstrap 95 percent interval of 0.389 to 0.494. AI Overviews was 0.499 across 62 pairs, with an interval of 0.441 to 0.556.
Those set-agreement figures are not a brand's naming probability, a naming-rate standard deviation or a prediction for another collection.
Our answer-study method, v1.1.
What this means for you
Reasoned. A single answer tells you what happened on that run. When a repeated result differs, retain both answers and check whether the question, engine interface, date window and scoring rule were actually held constant.
Do not describe every difference as an improvement or decline. If the wording changed, it is another question condition. If the same wording was used, the repeat can reveal variation without establishing that a website edit or supplier action caused it.
- If you sell software: ask for repeated observations before treating one appearance or omission as a stable position.
- If you prepare a report: keep the individual runs and the variation summary together, rather than choosing the most favourable answer.
- If you buy measurement: ask what was repeated and how agreement was calculated; a number near one half does not mean every vendor has a 50 percent appearance probability.
The study gives a reason to investigate a difference before acting on it. It does not supply a universally sufficient repeat count or an optimal budget. Those decisions require the event, precision and conditions you intend to measure.
Also asked as
Why can the same AI question produce different names?
Related
Next step
Treat Free 10-question check (one engine) as a dated bounded observation, retaining the question and engine beside the result. No email is needed; runs are limited per network and per day, and by the site's daily allowance.
When repeated evidence identifies one supported page change, Fix one answer, $490. Keep that deliverable separate from an unsupported claim that ordinary answer variation was a gain.
Prepared by the Broadcastwell team.
Page owner: Priya Nair, Management Consultant. Corrections: hello@broadcastwell.com