How do AI search crawlers differ from AI training crawlers?
Search crawlers support finding web information, while training crawlers collect material for model development; some providers also distinguish a user-requested page visit. Choose access for each purpose separately. In answers captured on 22 September 2026, Housecall Pro's own domain was cited in 43 of 150 answers. That output count does not reveal which training crawler, if any, fetched its pages.
What we measured
Measured. The September Index method covered ten field service questions, three scheduled answers on ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode. Housecall Pro's own-domain citation rate was 28.7 percent, with a descriptive Wilson 95 percent interval of 22.0 to 36.4 percent. The event required its name and a supporting citation to its domain in the same answer. No crawler-request or training-use event was collected.
Reported. Anthropic's bot guide, read 28 October 2026, distinguishes ClaudeBot for potential training collection, Claude-SearchBot for search and Claude-User for user-directed access. These are provider-specific purposes, rather than three names for an identical request.
Google's AI feature guidance, read the same date, assigns Search crawl access to Googlebot. It points separately to Google-Extended for training and grounding in some other Google systems. Google Search's AI features therefore should not be treated as a training-only access decision.
Every figure comes from the published method, v1.1.
What this means for you
Reasoned. Write the policy you intend before selecting a bot token. A publisher might allow public research to be found in search while making a separate choice about model development. Record those choices in distinct rows so the delivered rules can be checked against them.
Keep a user-directed fetch separate too. It describes access initiated for a person's request, not a general promise that a page will appear in future search answers. Use the provider's current documentation to identify the purpose and supported control rather than guessing from a familiar name.
Then verify the right event. A served robots file demonstrates the instruction you published. A request log can show a recorded request under its identification rules. An answer citation shows a linked source at the time of that answer. None supplies all the other evidence.
When reporting to a buyer, leave the output count beside the output label. The 43 Housecall Pro observations should not become a training-access score or a claim that a particular permission caused citation.
- For a site owner: approve search, training and user-directed access as separate choices.
- For a developer: retain the documented purpose beside each configured token.
Also asked as
What kinds of AI crawlers should a site owner distinguish?
Related
- Housecall Pro citation record
- Broader bot-access guide
- Robots rules and access
- Google Search control boundary
Next step
Use the Free 10-question check (one engine). No email is needed; runs are limited per network and per day, and by the site's daily allowance. Keep its source citation separate from your access-policy record.
For one factual error on an approved public source, Fix one answer, $490 addresses the page claim rather than a training-policy preference.
Claiming or correcting a vendor record is free and never depends on a purchase.
Prepared by the Broadcastwell team.
Page owner: Rohan Iyer, AI Engineer. Corrections: hello@broadcastwell.com