How do you compare answers before and after a page edit?
Broadcastwell measures and fixes whether AI search engines name B2B software companies.
Run the free checkMatch baseline and follow-up observations to the published change. Keep exclusions and no improvement visible; a paired comparison does not prove cause.
Measurement / BW-RESOURCE-06
Version: bw-content-2026-09-24-v1 | Claim ID: BW-RESOURCE-06 | Prepared: 2026-09-24
Decision: Compare observations without claiming causality or hiding no improvement.
Original Broadcastwell proposed operating method. Synthetic examples illustrate the method; they are not observed client outcomes.
Record two events separately
A page being published and an answer changing later are different events. Preserve the baseline receipts before work begins. Then keep the actual changed file, approved wording, route, timestamp and accessible publication evidence. A draft is not published work. A correction request being sent is not a third-party listing being updated.
Choose comparable records
Match question wording, method, interface, provider and locale where possible. Preserve both dates. If a surface or method changes, separate that observation or clearly limit the comparison. Do not assume a model label alone proves identical conditions; record the actual method details and known differences.
Read a fictional no-improvement example
In the included recheck fixture, DEMO-Q-002 has a reviewed absence at baseline and at follow-up on the same fictional local surface. The summary is “no observed mention improvement in the matched fixture.” A separate question has a failed follow-up and is excluded from mention comparison, with the failure retained. No sale, ranking movement or causal effect is represented.
Avoid the attractive shortcuts
Do not turn missing follow-up data into zero, remove an unfavorable run or count correlated repeats as independent buyers. Report what completed and what was actually reviewed. If the evidence is too small or changed to support a useful comparison, say that directly and decide whether another bounded investigation is justified.
Use appropriate primary guidance
For Google Search AI features, satisfying technical eligibility does not guarantee crawling, indexing or serving. That is a reason to keep implementation acceptance separate from appearance in an answer. It is not proof that a particular change worked or failed. Google Search guidance, reviewed September 24, 2026.
Expect movement before you change anything
Added 9 November 2026. Answers move between runs even when nothing on your site changes, so a re-check needs a baseline of repeats, not one answer. In the Absence Index release 2026-09, captured 22 to 23 September 2026, 1,259 of 14,192 checked vendor, question and engine cells were mixed: the vendor was named in one or two of three valid runs, not all three or none. That is 8.9 percent, with a Wilson 95 percent interval of 8.4 to 9.4 percent. Repeat disagreement table.
| Engine | Mixed cells | Share | Wilson 95% interval |
|---|---|---|---|
| ChatGPT | 159 of 2,860 | 5.6% | 4.8% to 6.5% |
| Claude | 253 of 2,860 | 8.8% | 7.9% to 9.9% |
| Perplexity | 200 of 2,860 | 7.0% | 6.1% to 8.0% |
| Google AI Overviews | 302 of 2,752 | 11.0% | 9.9% to 12.2% |
| Google AI Mode | 345 of 2,860 | 12.1% | 10.9% to 13.3% |
Seen from one run, the effect is larger. Our Study 3, Run-to-run noise, published 26 September 2026, found that 22.2 percent of single-run vendor namings were not repeated by both other runs of the same question on the same engine, on a base of 8,055 namings. A re-check that compares one baseline answer with one follow-up answer can therefore show a change that three runs would not.
A dated check before you compare
Added 9 November 2026. Write these five lines down before the follow-up runs, and keep them with the baseline receipts:
- The exact questions, engines and country, copied from the baseline.
- The number of runs per question and engine on each side; our method schedules three.
- What counts as valid, failed and excluded, so a failed follow-up stays visible instead of becoming a miss.
- The published change, with its route and timestamp, and any engine release between the two captures from our model release tracker.
- The result you will accept as no improvement, stated before you see the follow-up.
Lesson 5 of our AI visibility course works through the same comparison on fictional data, and session D of our client training path turns this list into a team decision. Visit records belong in a separate column; our AI referral analytics template keeps them apart from answer counts.
Keep revenue in its own ledger
A resource completion, qualified inquiry, held conversation, invoice and collected payment are distinct commercial signals. Exclude tests and preserve unknown attribution. A later payment may matter to the business without proving this page caused it. Use the recheck worksheet to recommend continuing, revising or stopping the hypothesis; include no-improvement evidence in that decision.
Use the worksheet
Editable worksheet | Blank JSON | Worked example
Compare one matched pair, retain exclusions and decide whether the evidence supports continuing the work.
Provenance
This original method extends the supplied September 24 execution-pack draft and blueprint; no competitor writing or outcome is reused. Definitions and synthetic evidence: source ledger. External technical statements, where present, have nearby primary-source links. Source freshness is limited to the stated review date.