AI robots.txt tester
Free, no sign-upTest crawler access, change bot rules and download your robots.txt.
Choose who gets access
Search and training are separate choices.
Load, paste or start a file to see its settings.
Other bots, comments and sitemaps stay as they are. This changes your draft, not your website.
These are permissions in your file, not proof a bot visited or followed them. Robots.txt does not protect private content.
Your robots.txt
One complete file, ready to hand to your developer.
Changes are not published automatically. Replace /robots.txt on the same website after reviewing your draft.
File checks
Check a specific page
Useful when your file allows some pages and blocks others.
OAI-SearchBot
Awaiting file
No file tested.
How this test works and its limits
We fetch only this host's /robots.txt, or read the text you paste. Each bot uses its own groups, then the general (*) rules if none match. Applebot can use Googlebot's rules. Paths are case-sensitive; the longest matching rule wins, with Allow winning a tie. Rules for a different host do not apply.
Training permissions do not guarantee an opt-out. Search permissions do not guarantee indexing or citations. This tool does not test firewalls, authentication, page-level robots tags or actual crawler visits. Content-Signal preferences are preserved, but not evaluated as access rules.
Loading a website sends its URL to Scavo. Draft edits stay in your browser. A shared draft includes your file contents. Only share information you are happy to pass on.
A little more detail
Allow search without allowing everything
Search crawlers and model-training crawlers can have separate rules. A deliberate block is a policy decision, not a website error.
Read the robots.txt guideIs robots.txt a privacy or security control?
No. It is a request to cooperative crawlers, not access protection. Private content needs authentication or another real access control. A blocked URL can still be discovered, and non-cooperative bots can ignore these rules.
Does an allowed result mean the bot can reach my site?
It means the tested rules permit the selected path. Firewalls, authentication, rate limits, HTTP errors and page-level indexing directives may still prevent access or appearance in search.
Can I publish the generated file straight away?
Review it against your existing rules first. Keep restrictions for private areas and any sitemap declarations you need. The tester checks policy matching; it cannot know every part of your site's intended policy.
Keep an eye on your website
Let Scavo do the regular checks.
Get uptime alerts and weekly health checks, with findings you can send straight to your web team.