I was reviewing Scavo's AI crawler checks this week and found we were asking one question where the web now asks four.
"Can AI bots access this site?" sounds tidy. It is also not very useful.
A business can reasonably say yes to search discovery, yes to a page being fetched when a user asks for it, no to model training, and yes to an advertising landing-page review. That is not a contradiction. It is a policy.
The crawler names have multiplied, the jobs behind them have split, and a copied robots.txt block can now remove useful discovery while leaving the thing you meant to block untouched. So here is the practical version I would use before changing anything.
If you first need the difference between robots.txt, sitemap.xml, and llms.txt, start with the wider Scavo guide. This article deals with the next decision: what different AI agents are actually there to do.
The four decisions
1. Search and discovery
These crawlers help an AI search or answer product find, index, and link to public pages.
Current examples include:
OAI-SearchBotfor ChatGPT search discovery.Claude-SearchBotfor Claude search results.PerplexityBotfor Perplexity search results.Applebotfor Apple search features, including Spotlight, Siri, and Safari.
If you want customers to discover your public pages through those products, an accidental block here is a visibility problem.
It is not a guarantee of inclusion, a citation, or traffic. It simply avoids closing the door before anyone has knocked.
2. User-requested retrieval
These agents fetch a page because someone has asked an AI product to look at it.
Examples include ChatGPT-User, Claude-User, and Perplexity-User.
This is different from a crawler building a search index in the background. It can be the difference between an assistant being able to inspect your pricing page now and having to rely on an older source or a search snippet.
The providers do not all handle this identically. Perplexity currently says its user-requested fetcher generally ignores robots.txt, for example. Treat each provider's documentation as the source of truth rather than assuming one rule covers every product.
3. Model training and wider data use
These controls deal with whether public content may contribute to model development or related data use.
Examples include:
GPTBotClaudeBotCCBotGoogle-ExtendedApplebot-Extended
The last two are easy to misunderstand. Google-Extended and Applebot-Extended are policy tokens, not separate bots you should expect to see making their own HTTP requests. Google says its existing user agents do the crawling. Apple says Applebot-Extended does not crawl pages at all.
That means a fake browser request with one of those names proves very little. Their value is in the policy declared in robots.txt.
4. Advertising landing-page review
OpenAI now documents OAI-AdsBot separately. Its job is to validate landing pages submitted for ads on ChatGPT and help determine when those pages are relevant.
This will not matter to every site today. It does matter if a marketing team starts testing ChatGPT ads and discovers that the landing page is blocked by a rule, a CDN challenge, a WAF, authentication, geo restrictions, or a rate limit.
That is not an SEO issue. It is an ad delivery issue with a crawler-shaped error message.
A useful policy for many business websites
For many service businesses, SaaS products, ecommerce sites, or agencies, a reasonable starting position might be:
- Allow public search and answer discovery.
- Allow user-requested retrieval of public pages.
- Block model-training use if that matches the company's content policy.
- Allow ad review if the business uses, or may soon test, ChatGPT ads.
- Keep account, checkout, billing, admin, and private application paths out of every public crawler route.
Here is a compact example that allows normal public crawling while blocking several training and data-use controls:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/
Disallow: /billing/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xmlDo not paste that blindly. Replace the paths, check what your CMS already emits, confirm the business policy, and test the result. A publisher, membership site, public-sector service, or rights-sensitive archive may need a different answer.
Also remember that robots.txt is guidance for compliant crawlers, not a security boundary. If a URL contains private data, protect it with authentication and application permissions.
The specific-group trap
This was the most useful detail from the Scavo check review.
Many teams add a specific crawler group like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/It looks as though OAI-SearchBot can access the public site but still inherits the private-path exclusions below. Under standard matching rules, that is not a safe assumption.
Google documents that a crawler uses the most specific matching user-agent group, and that a specific group is not combined with the wildcard group. Other compliant crawlers commonly follow the same Robots Exclusion Protocol model.
So once you create a specific group, repeat every path rule that crawler still needs:
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /admin/
User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/Or, if the wildcard policy already gives the crawler the access you want, do not add a specific group just to make the file look more deliberate.
The shortest correct file usually beats the longest impressive-looking one.
Robots.txt is only the first gate
A crawler can be allowed in robots.txt and still fail at the front door.
Check the whole request path:
- DNS and TLS: Can the hostname resolve and complete a secure connection?
- Redirects: Does the crawler reach the intended canonical page without a loop or odd regional detour?
- CDN and WAF: Is bot protection returning
403,429, or a JavaScript challenge? - Application response: Does the final URL return useful HTML with a successful status?
- Page directives: Do
noindex,nosnippet, orX-Robots-Tagheaders change the outcome? - Rendered content: Is the useful answer actually in the response, or hidden behind fragile client-side behaviour?
OpenAI's ad guidance is unusually helpful here because it names the boring failure points: robots rules, WAF and CDN controls, bot mitigation, authentication, CAPTCHAs, geo rules, and rate limiting. The same checklist is useful well beyond ads.
How to verify the result without fooling yourself
Use three levels of evidence.
Level 1: Parse the file
Run the exact live robots.txt through the free Scavo AI robots.txt tester. Check each relevant agent separately and inspect the rule that actually won.
Do this for the real hostname. example.com, www.example.com, docs.example.com, and a staging host have separate files and separate policies.
Level 2: Probe the live route
Request an important public page using the documented crawler user agent and compare it with a normal browser response.
This can reveal obvious blocks, redirects, empty responses, and bot challenges. It cannot prove that a request came from the official provider because user-agent strings are easy to copy.
Level 3: Verify genuine traffic
When the provider publishes IP ranges or verification guidance, use it. Review CDN/WAF events and server logs, then match user agent and network identity together.
Do not permanently allowlist an address because it appeared once in a log. Provider ranges change, and a user-agent string on its own is not identity.
A five-minute policy worksheet
Before engineering edits the file, answer these questions:
- Which public pages do we want found in AI search and answer products?
- Should an assistant be able to fetch those pages when a user asks?
- What is our position on model training and wider data use?
- Are any ad landing pages likely to need automated review?
- Which paths must stay out of every public crawler route?
- Who owns the decision: marketing, product, legal, engineering, or a named combination?
- When will we review it again?
Write the answers down. A robots.txt file without an owner tends to become a small museum of old decisions.
What to do next in Scavo
- Test your live file with the AI robots.txt validator.
- Run a fresh website scan and open the AI Visibility checks.
- Review search discovery, user retrieval, model data use, and ads review as separate outcomes.
- Check the bot-access parity evidence for CDN, WAF, redirect, or response differences.
- Re-scan after every crawler-policy or edge-security change.
- Give the policy an owner and a review date so new agents do not quietly make it stale.
Sources
- OpenAI: Publishers and Developers FAQ
- OpenAI: Advertiser Guidance for Allowing Web Crawlers
- Anthropic: How its three web robots are used
- Apple: About Applebot and Applebot-Extended
- Google: Google-Extended crawler control
- Google: How robots.txt user-agent groups and rules are selected
- Perplexity: PerplexityBot and Perplexity-User
