Course lesson 5: did a page change help AI answers?
Broadcastwell measures and fixes whether AI search engines name B2B software companies.
Run the free checkThe last lesson of our AI visibility course: you can only tell whether a page change helped by asking the same frozen questions on the same engines under the same method, and by checking untouched control questions at the same time. A change counts only when its interval clears zero, and a fall is recorded exactly like a rise.
Updated 11 November 2026.
Reading time: about 5 minutes. Lesson 5 of 5 in the course. Previous lesson.
Four conditions for a fair comparison
- Same questions, word for word. Our method freezes the wording before the first capture. Changing one question opens a new baseline and ends the old comparison.
- Same engines, recorded apart. A rise on one engine says nothing about the others; Google AI Overviews and Google AI Mode are separate products.
- Same method version. If the collection or scoring rules change, the before and after are no longer the same instrument.
- Same kind of runs. Compare scheduled runs with scheduled runs; extra runs given to disagreeing pairs stay out of the headline.
If any one of the four breaks, report the two readings as separate dated observations rather than as a change.
Why you need control questions
Engines change on their own. A provider can release a new version between your two captures, and every company's numbers can move without anyone editing a page; our model release tracker dates those releases. To separate your change from that drift, our method's addendum v1.2 draws target questions and untouched control questions from the same bank and locks the assignment before the follow-up.
Controlled lift is the change on the target questions minus the change on the controls, each with its own 95 percent interval. There are exactly three verdicts:
- Lift shown: the lower bound of the lift is above zero and the target questions rose.
- Gain on target questions, not separable from drift: the targets rose, but the interval includes zero.
- No gain observed: the target questions did not rise.
There is no fourth verdict and no score. Repeated runs are correlated, so the printed interval is a floor on the real uncertainty, not a ceiling.
A fall, recorded the same way
In the fictional Kalvenor sample (Sample data, fictional company), Kalvenor was named on ChatGPT in 15 of 29 answers in the baseline run of 12 July 2026, and in 0 of 10 in the monthly run of 12 August 2026. The intervals, 34.4 to 68.6 percent and 0.0 to 27.8 percent, do not overlap, so the sample records a fall on ChatGPT. Both bases are small, and the record says so. A report that only shows rises is a sales document, not a measurement.
Exercise
A fictional company edits one page. Before the edit, its target questions name it in 12 of 50 answers and its control questions in 10 of 50. After the edit, the targets name it in 20 of 50 and the controls in 17 of 50.
- Write each count with its base, before and after.
- Work out the change on each group in percentage points, then the lift: target change minus control change.
- Choose one of the three verdicts.
Check your work: the targets rose 16 points, from 24.0 to 40.0 percent, but the interval for that change alone runs from about minus 2 to plus 33 points. The controls rose 14 points. The lift is 2 points, and the verdict is "Gain on target questions, not separable from drift". Most of the rise happened on questions nobody touched.
Our guide on comparing answers before and after a page edit has a worksheet for your own recheck, and our Manual chapter on how long the work takes to show covers timing.
Where to go after the course
- Look up any term from the five lessons in our AI search glossary.
- Practice on our self-paced workshops, each built from published material.
- To see your own buyer question measured, a Live question is $19 once, credited in full against Answer Repair within 30 days.
Sources
- Broadcastwell, measurement method v1.1 and controlled lift addendum v1.2, read 9 November 2026.
- Broadcastwell, fictional Kalvenor sample, values as shown on our completed-case format page.
- Exercise figures are fictional; Wilson and Newcombe intervals computed by us.
Back to the start: the course overview.