Web Bot Auth Directory Missing or Invalid

TL;DR: Most sites do not need Web Bot Auth yet. If your service runs outbound bots or agents that need verifiable identity, publish a valid well-known key directory and keep it usable.

When "Web Bot Auth Directory Missing or Invalid" is failing, you usually do not need a full rebuild. It improves how answer engines understand and attribute your content. Pick one clear owner, ship one focused fix, and run a quick verification loop.

This is not a universal website requirement. Web Bot Auth is mainly relevant when your own bots or agents make requests to other sites and you want those requests to be verifiable without relying only on IP ranges or user-agent strings.

Scavo keeps this check intentionally simple. It looks for a public key directory at the standard well-known path and verifies whether the response is usable JSON with recognizable JWK keys. If you are not participating in this ecosystem yet, an info result is fine.

What Scavo checks (plain English)

Scavo requests /.well-known/http-message-signatures-directory and checks whether the response is valid JSON containing at least one usable key entry.

A pass means the directory is present and contains usable keys. A warning means the endpoint returned 200, but the JSON or keys were broken. An info result means the directory was not detected.

• Scan key: ai_web_bot_auth
• Category: AI_VISIBILITY

How Scavo scores this check

• Warning: the well-known directory returned 200, but the body was not valid JSON or did not contain usable keys.
• Pass: the directory was present and contained at least one usable published key.
• Info: no usable Web Bot Auth directory was found.

Why fixing this matters

As more sites try to distinguish legitimate agent traffic from spoofed automation, signed requests become more trustworthy than plain user-agent strings. If your bot participates in that model, the public key directory is the starting point for verification.

A broken directory wastes time for every receiver trying to validate your requests. It is effectively an identity system that advertises itself but cannot complete the first lookup.

Common reasons this check flags

• The well-known path is missing or served at the wrong URL.
• The response returns JSON-like text, but not a valid JWK Set structure.
• Keys are missing required fields such as kty or a usable identifier.
• The endpoint exists in staging only, or a CDN rule blocks public access in production.

If you are not technical

1. Only prioritize this if your product sends automated traffic to other sites and wants to be recognized as a trustworthy bot or agent.
2. If that is not your use case, accept the info result and focus elsewhere. Do not implement cryptographic bot identity just to make a dashboard line go green.
3. If it is your use case, ask engineering who owns key rotation, directory hosting, and request signing so the system has clear operational ownership.

Scavo flagged Web Bot Auth (ai_web_bot_auth). Please review the production well-known key directory at /.well-known/http-message-signatures-directory, return valid JSON with usable public keys, and confirm our request-signing setup matches the keys we publish before re-running the scan.

If you are technical

1. Serve the directory at the exact well-known path over HTTPS.
2. Return a valid JSON Web Key Set and include only public key material. Never expose private key data.
3. Use stable key identifiers and a documented rotation process so receivers can validate during rollover.
4. If you are targeting Cloudflare verification specifically, follow the published requirements for Ed25519 keys, Signature, Signature-Input, and Signature-Agent headers.
5. Validate both the directory response and at least one signed outbound request as part of rollout testing.

How to verify

• Fetch the well-known directory directly from production and validate the JSON.
• Confirm the published keys match the keys your signing implementation actually uses.
• If you integrate with a verifier such as Cloudflare, run one real signed request and confirm it validates.
• Re-run Scavo and confirm the result moves to pass or remains intentional info.

What this scan cannot confirm

• Scavo does not execute a full cryptographic verification of your outbound request signatures. It checks the public discovery surface.
• Scavo cannot determine whether your business case truly needs Web Bot Auth. It only reports whether the standard well-known directory is live and usable.

Owner checklist

• [ ] Name one owner for this check and note where it is controlled (app, CDN, server, or CMS).
• [ ] Add a release gate for this signal so regressions are caught before production.
• [ ] After deploys that touch this area, run a follow-up scan and confirm the result is still healthy.
• [ ] Re-check AI crawler and citation signals after robots, schema, or author metadata changes.

FAQ

What does Scavo actually validate for Web Bot Auth Directory Missing or Invalid?

Scavo checks live production responses using the same logic shown in your dashboard and weekly report.

Will AI visibility changes show immediately after we ship fixes?

Usually not instantly. Crawlers and answer engines refresh on different schedules, so confirm technical signals first, then monitor citations and mentions over time.

What is the fastest way to confirm the fix worked?

Run one on-demand scan after deployment, open this check in the report, and confirm it moved to pass or expected info. Then verify at source (headers, HTML, or network traces) so the fix is reproducible.

How do we keep this from regressing?

Keep one owner, keep config in version control, and watch at least one weekly report cycle. If this regresses, compare the release diff and edge configuration first.

Sources

• Cloudflare Docs: Web Bot Auth
• RFC 9421: HTTP Message Signatures
• RFC 7517: JSON Web Key (JWK)

Need stack-specific help? Send support your stack + check key and we will map the fix.