What can an AI assistant learn from a software website?
An assistant can recover a software vendor's public description, but that does not amount to testing the product. On 6 October 2026, the Agent Gap Describe task found pages for all 70 selected vendors and recorded 69 correct task results. One result was incorrect, so finding the right website did not make every description reliable.
What we measured
Measured. The Agent Gap study method used the ChatGPT Work browser interface to describe each of 70 selected B2B software vendors. In this task, 70 pages were found, 69 results were scored correct and 67 reached the study's endpoint. These are three separate recorded fields.
The study also contained pricing-page and demo-path tasks. Together they made 210 initial sessions; 30 preselected repeat sessions were kept separate. The 69 of 70 description result belongs to the Describe task, rather than to all 210 sessions.
The observations record the pages encountered through that interface on 6 October 2026. They do not test actual product functionality or every assistant's ability.
Our answer-study method, v1.1.
What this means for you
Reasoned. A public page can tell an assistant what the vendor says it sells, whom it serves and where a visitor can go next. Read the returned description against the page instead of treating its fluency as verification.
Use a source passage for each important product statement. If a page only describes an integration at a broad level, the assistant should not turn that into a promise of a specific native feature. If a fact is missing, a clear unknown is more useful than a plausible completion.
- If you publish the website: make the product and intended user explicit. Keep supported statements close to the evidence a reader can inspect.
- If you evaluate software: use the assistant's account as a route to the original page, then verify the details relevant to your work.
- If you report the task: retain the page, date and returned description, and distinguish found, correct and reached.
The 67 reached outcomes also need their own reading. A task can yield a correct description without satisfying every step of its destination condition. That gap is useful precisely because the study did not compress discovery and completion into one success label.
Also asked as
What can an AI assistant learn from a software website?
Related
- Agent Gap study
- Page finding versus understanding
- Connector versus browsing
- Read a source-backed answer
Next step
Use Free 10-question check (one engine) to inspect an answer about a company while keeping the cited product facts available for comparison. No email is needed; runs are limited per network and per day, and by the site's daily allowance.
If a public description leaves an important product fact unclear, Fix one answer, $490. Bring the source passage that supports the wording you need.
Prepared by the Broadcastwell team.
Page owner: Vikram Patel, AI Architect. Corrections: hello@broadcastwell.com