AI Bot Policy Not Set in robots.txt

Start here

Before you fix it: what this check means

Scavo reads robots.txt to see which named AI crawlers are allowed or blocked. Search access and model-training preferences are separate choices; the result does not prove a provider visited your site. Scavo reads `robots.txt` status/content and evaluates root-path policy for key AI crawlers.

Why this matters in practice: clear content and access preferences help you describe what AI tools may find and use.

Use this as evidence, not a final verdict. Requests from real AI providers may receive different responses from the scanner, depending on location and bot-protection settings. Check the live site first. Open the reported page or file and compare its contents with your intended access preferences Make one controlled change. Normalize robots groups and remove contradictory root rules. Re-scan the same URL and compare the finding; check the feature yourself as well.

Background sources

TL;DR: Your robots.txt doesn't specify rules for AI crawlers like GPTBot or ClaudeBot, leaving your AI visibility to chance.

GPTBot is blocked by 5.89% of all websites, with 35.7% of the top 1,000 sites blocking it (Ahrefs, 2024). Nearly 38% of indexed sites now have AI-specific restrictions, up from 8% in 2023 (EngageCoders). If you don't set explicit policy, you can't control whether your content appears in AI products or training data. A deliberate policy — whether allowing or blocking — is better than leaving it undefined.

What Scavo checks (quick version)

Scavo reads robots.txt status/content and evaluates root-path policy for key AI crawlers:

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
anthropic-ai
PerplexityBot
Google-Extended
CCBot

Scavo classifies each agent policy as explicit/wildcard and state (allowed, blocked, mixed, unspecified).

How Scavo scores this check

Result behavior:

Warning: robots.txt missing/empty (404 or blank)
Fail: wildcard root block with no explicit per-bot exceptions
Info: robots policy unavailable in scan
Info: default/wildcard policy is clean but no explicit bot rules
Warning: one or more agents blocked or mixed/conflicting
Pass: explicit, consistent per-bot policy with no critical conflicts

In your scan report, this appears under What failed / What needs attention / What is working for ai_crawler_policy, followed by Recommended next steps and Technical evidence (for developers) when needed.

Scan key: ai_crawler_policy
Category: AI_VISIBILITY

Why fixing this matters

Policy clarity matters more than hype. Teams need to know whether they are intentionally visible, intentionally restricted, or unintentionally drifting due to inherited wildcard rules.

Without explicit policy, legal, content, and engineering can make conflicting assumptions about AI usage rights and discoverability.

Common reasons this check flags

User-agent: * with broad Disallow: / and no AI exceptions.
Duplicate groups produce mixed allow/disallow outcomes.
Robots exists but does not mention any modern AI agents explicitly.
Production robots differs from documented policy in legal/content docs.

If you are not technical

Decide business stance for each major crawler class (allow, block, conditional).
Ensure legal/comms language matches technical robots policy.
Ask engineering for one plain-language matrix by bot.
Re-run Scavo and check blocked/mixed counts.

Scavo flagged AI Crawler Policy (ai_crawler_policy). Please clean up robots.txt so each target AI bot has clear root-path intent (allow/block), remove mixed directives, and document policy decisions for legal/content stakeholders.

If you are technical

Normalize robots groups and remove contradictory root rules.
Add explicit directives for bots you intentionally manage.
Avoid relying on ambiguous inherited wildcard behavior for critical decisions.
Keep one source-of-truth robots file under version control.
Reconcile legal policy text with actual robots directives.

How to verify

Fetch live robots.txt from production.
Parse each target bot and confirm single clear root policy.
Confirm wildcard behavior does not accidentally override intent.
Re-run Scavo and verify policy score + blocked/mixed counts improve.

What this scan cannot confirm

It does not enforce contractual/legal licensing terms by itself.
It does not guarantee third-party compliance beyond published robots norms.
It does not test all non-standard/private crawler identities.

Owner checklist

[ ] Assign owner for AI crawler stance and robots implementation.
[ ] Keep a reviewed bot policy matrix (intent + technical directive).
[ ] Version control robots updates with approval trail.
[ ] Audit robots policy after CDN/security migrations.

FAQ

Is blocking all AI crawlers always wrong?

No. It can be intentional. The issue is unintentional or undocumented blocking.

Why does missing robots.txt return warning instead of fail?

Because policy is undefined rather than explicitly contradictory, but it still increases control risk.

Why include Google-Extended separately?

It allows policy distinction for AI-related usage controls beyond standard crawl/index behavior.

Should we list every bot on earth?

No. Prioritize major bots relevant to your business and review regularly.

Sources

Need a bot-policy matrix draft (intent + robots syntax) your team can approve quickly? Send support your preferred allow/block stance per crawler.

AI Bot Policy Not Set in robots.txt