Should You Let AI Bots Read Your Website?

I've spent part of this week tidying up the AI crawler checks inside Scavo.

What should have been a quick sense check turned into me staring at a robots.txt file full of names like GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, and Google-Extended.

They all mean something. They are also not very helpful if you run a business and just want to know whether blocking one of them will make your website disappear from ChatGPT.

robots.txt is a public file that asks well-behaved crawlers which parts of a website they can visit. It can guide them, but it does not secure anything.

So, the useful answer first:

For most public business websites, I would keep AI search and user-requested access open, then make a separate decision about model training.

That is really the whole point of this post.

Why there isn't one "AI bot" switch

Some AI bots help tools like ChatGPT, Claude, or Perplexity find public pages and include them in search results or answers.

Some visit a page because a person has specifically asked an assistant to read it.

Others are used for model training or wider data collection.

In practical terms, search access is about whether an AI assistant can find your services, products, opening hours, or advice today. Training is about whether a company can use your pages to help improve its models more generally.

OpenAI also has a separate bot for checking advertising landing pages.

Those are different uses. Blocking all of them because you do not want your writing used for model training can also block the tools you hoped might recommend your business to a potential customer.

It does not mean every website should allow everything. It means checking what a rule will block before adding it.

What I would do for most business websites

If the website exists to explain a service, sell a product, take bookings, or bring in enquiries, my starting point would be:

• Let AI search tools reach the useful public pages.
• Let assistants open those pages when somebody asks about the business.
• Decide separately whether the content can be used for model training.
• Keep account, checkout, admin, and genuinely private areas protected properly.
• Avoid adding a long list of bot rules unless somebody knows why each rule is there.

That keeps the website available as a source without quietly agreeing to every possible use of its content.

Publishers, membership sites, and businesses where the content itself is the product may come to a different decision. That is fine. For a small service or brochure website, the decision can usually stay fairly simple.

What the common names actually mean

For whoever ends up editing the file, here is a rough working guide:

• OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Applebot are mainly about finding and surfacing public pages.
• ChatGPT-User and Claude-User may fetch a page because a user has asked the assistant to look at it.
• GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended are more closely tied to training or wider data use.
• OAI-AdsBot checks landing pages submitted for ChatGPT ads. If you are not running those ads, it is unlikely to be today's problem.

There are details and exceptions behind that list. Google-Extended and Applebot-Extended, for example, control how already-crawled content is used rather than acting like separate crawlers you would expect to see in server logs.

But the list above is enough to stop "allow AI" and "block AI" being treated as one decision.

A starting example

The following example keeps normal public crawling open but says no to several training and data-use agents:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/
Disallow: /billing/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Please do not paste that straight into a live website without adapting it.

The private paths need to match the real site. Your CMS may already add rules. Your business may be perfectly happy to allow training, or may need to block more than this. The point is the shape of the decision: public discovery stays open while training is handled separately.

And robots.txt is not security. A private page needs a login and proper permissions. Asking well-behaved bots not to visit it is not the same thing as protecting it.

One annoying robots.txt detail

This was the bit that made me go back and improve the Scavo check itself.

Imagine a site has these rules:

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /account/
Disallow: /admin/

It looks like OAI-SearchBot can read the public site but will still avoid the account and admin paths underneath.

That is not how the usual matching rules work. Once a crawler has its own specific group, it may use that group instead of combining it with the * rules.

So either repeat the exclusions inside the specific group:

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /admin/

Or, if the general * rules already give it the access you want, do not add a special group at all.

It is a small bit of syntax, but it can completely change the result.

The file is only one part of it

I have seen websites where robots.txt says a bot is welcome, while Cloudflare, a security plugin, or the web server blocks it anyway.

So a proper check needs to look beyond the file:

• Does the page return a normal successful response?
• Does the crawler get sent through a strange redirect?
• Is a firewall returning 403 or a bot challenge?
• Does the useful page content appear in the response?
• Are page-level rules such as noindex or nosnippet changing the outcome?

Changing a browser's user-agent name can reveal obvious differences, but it cannot prove a request really came from OpenAI, Anthropic, Apple, or anybody else. For that, you need server or CDN logs and the provider's current verification guidance.

What Scavo now shows

I have changed the Scavo checks so the result is less likely to collapse everything into one vague "AI access" answer.

The useful questions are now:

• Can AI search tools reach the public website?
• Can an assistant retrieve a page when a user asks?
• Is training or wider data use allowed, blocked, or simply not stated?
• Does the live server treat those crawlers differently from a normal visitor?
• Which exact robots.txt rule produced the result?

That last one matters. A warning is much easier to act on when you can see the line that caused it.

What I would check today

1. Open your real robots.txt file at https://yourdomain.com/robots.txt.
2. Do not delete or replace anything until you know why the existing rules are there.
3. Run it through the free Scavo AI robots.txt tester.
4. Decide two things: do you want the public site found in AI answers, and are you happy with model-training use?
5. If the result needs changing, update the smallest possible part of the file.
6. Run a fresh Scavo scan afterwards to check the live response as well as the written rule.

If you want the wider detail on sitemaps and llms.txt too, I have covered that separately in the Scavo robots, sitemap, and llms.txt guide.

Most businesses do not need to settle every AI crawler question in one sitting.

They do need to avoid blocking useful discovery by accident, stop treating robots.txt as a security system, and make sure somebody knows why the rules are there.

That is a good enough place to start.

Sources

• OpenAI: Publishers and Developers FAQ
• OpenAI: Advertiser Guidance for Allowing Web Crawlers
• Anthropic: How its three web robots are used
• Apple: About Applebot and Applebot-Extended
• Google: Google-Extended crawler control
• Google: How robots.txt user-agent groups and rules are selected
• Perplexity: PerplexityBot and Perplexity-User

What to do next in Scavo

1. Run a fresh scan on your main domain.
2. Open the matching help guide in /help, assign an owner, and ship the smallest safe fix.
3. Re-scan after deployment and confirm the trend is moving in the right direction.