OAuth Discovery Metadata Missing or Broken
If agents or third-party clients need OAuth to access your service, they should be able to discover the correct authorization metadata without guessing endpoints by hand.
Open guideAI visibility help for crawler policy, content structure, and citation readiness.
Controlling how AI systems crawl, interpret, and cite your content. Reducing silent visibility drift across answer engines.
Agent Protocol Discovery Documents Missing or Broken
If your service exposes machine-usable tools, APIs, or agent endpoints, discovery documents can tell clients what exists before they start guessing. Broken discovery docs create more friction than having none at all.
Open guideWeb Bot Auth Directory Missing or Invalid
Web Bot Auth lets bots prove who they are with signed HTTP requests and published keys. It is optional for most sites, but if you use it, the key directory must be valid and public.
Open guideAbout ai visibility
15 guides
15 active checks
4 sources
AI systems parse pages differently to search engines. They need explicit crawler policy, extractable content sections, and strong attribution signals to cite your work reliably.
Many teams assume that if a page is indexable for search, it is automatically ready for AI retrieval and citation. In practice, answer engines and AI crawlers often depend on clearer bot policy, more extractable page structure, stronger attribution signals, and content that still makes sense when lifted out of the full page chrome.
This category is about making that machine-readable path more intentional. It covers explicit crawler policy, llms.txt, parity across major AI user agents, how dependent the page is on JavaScript, whether sections are chunkable and answer-shaped, and whether canonical and attribution signals are strong enough to support citation.
Why it matters
AI visibility can fail quietly. You may not get a dramatic error message when bots are blocked, content is too JS-heavy, or attribution signals are weak.
Clear policy matters because different crawlers do not all inherit the same access assumptions. Being explicit reduces accidental gaps.
Common pitfalls
Treating wildcard robots rules as enough when individual AI user agents need explicit clarity or are being treated differently upstream.
Publishing long, visually polished pages whose core meaning disappears once the surrounding interface and JavaScript are stripped away.
What's covered
AI crawler policy and robots rules so major user agents receive deliberate, documented instructions.
llms.txt and related machine-readable discovery files where you want to publish a cleaner content map.
Where to start
Decide policy first: which crawlers you want to allow, block, or handle consistently, then make that policy explicit in robots and related files.
Keep the main answer-bearing content in server-rendered HTML with a clear heading hierarchy and obvious section boundaries.
