What do AI crawlers do, and how can site owners control access?
On our 29 October 2026 read, OpenAI and Anthropic document different web robots for search, training and user-directed retrieval. Choose the rule for the job you mean to permit or block. Our Agent Gap study reached the endpoint in 67 of 70 initial description tasks on 6 October 2026; those website tasks did not identify a crawler or test the effect of a robots rule.
What we measured
Reported. OpenAI's bot documentation separates OAI-SearchBot for search from GPTBot for training. It describes ChatGPT-User as acting on user requests; that agent is separate from automatic search crawling. Anthropic's crawler article distinguishes ClaudeBot, Claude-SearchBot and Claude-User, with their own purposes and robots.txt instructions. Both sources were read on 29 October 2026.
Measured. The Agent Gap protocol records the 67 reached endpoints within its initial Describe task for 70 software vendors. The task concerned the vendor's public website. It did not classify requests as OAI-SearchBot, GPTBot or any Anthropic robot.
The endpoint count cannot establish whether a particular crawler was allowed or blocked.
What this means for you
Reasoned. Name the robot in the decision. A rule about GPTBot concerns OpenAI's training crawler; the same provider documents OAI-SearchBot for search. A site owner who intends to restrict training while allowing search needs to inspect those distinct rules, rather than treating the company name as one access setting.
Anthropic makes another distinction useful for a site owner: Claude-User serves user-directed retrieval, while Claude-SearchBot supports search. Its guide places controls in robots.txt for the site or subdomain concerned. If your product documentation lives on a separate subdomain from your homepage, check the instructions at that address too.
For OpenAI search eligibility, the search help article also asks publishers to allow traffic from the published searchbot IP addresses at the host. A robots permission and a host restriction are separate checks. Do not change one and report that both were verified.
Keep the task receipt separate. In Agent Gap, reaching a description-task endpoint is the reported event, so its 67 successes cannot be relabelled successful crawler visits. A missing company name in a later AI answer would likewise require its own answer record; it would not identify which robot requested a page.
For an agency managing a documentation subdomain, write down the exact intended purpose beside the bot rule and the address inspected. That makes the Claude-User or OAI-SearchBot choice clear to the person maintaining that specific host.
Also asked as
What do AI crawlers do, and how can site owners control access?
Related
Next step
Use Free 10-question check (one engine) to inspect a defined set of buyer questions. No email is needed; runs are limited per network and per day, and by the site's daily allowance.
For one documented gap, Fix one answer, $490. Keep the question and source receipt with the requested change.
Our answer-study method, v1.1.
Prepared by the Broadcastwell team.
Page owner: Rohan Iyer, AI Engineer. Corrections: hello@broadcastwell.com