Robots.txt, Sitemap.xml, and llms.txt: What Each File Actually Controls
A no-nonsense guide to what robots.txt, sitemap.xml, and llms.txt do in production, and what they definitely do not do.
Read articleSEO guidance from real scan results.
Indexing, canonical signals, metadata, and crawl discovery.
Indexability Signals Conflicting — Canonical vs Noindex vs Hreflang
Learn how Scavo checks for contradictions between meta robots, X-Robots-Tag, canonical tags, and hreflang so one URL does not send search engines mixed instructions.
Open guideMeta Robots or X-Robots-Tag Blocking Indexing by Accident
Learn how Scavo checks both the robots meta tag and X-Robots-Tag headers so hidden noindex directives do not quietly keep important pages out of search.
Open guideCanonical Tag Missing — Duplicate Content Splitting SEO Authority
When multiple URLs serve the same content (with and without trailing slashes, query parameters, HTTP vs HTTPS), search engines either index all versions — wasting crawl budget and diluting rankings — or pick the wrong one as canonical. A single rel=canonical tag consolidates all link equity to the version you choose and prevents indexing bloat.
Open guideH1 Heading Missing or Duplicated
78% of advanced screen reader users navigate pages by headings (WebAIM Survey #10, 2024), so a missing or duplicated H1 directly impacts accessibility. For SEO, the H1 tells Google what the page is about. Without it, crawlers have to guess from surrounding content, which weakens your keyword relevance signals and dilutes page focus.
Open guideHeading Hierarchy Broken — H1 to H6 Structure Issues
88.8% of screen reader users find heading levels very or somewhat useful for page navigation (WebAIM Survey #10, 2024). When headings skip levels or are out of order, screen reader users can't build a mental map of your content structure. Search engines also use heading hierarchy to understand content relationships and assign keyword relevance.
Open guideHreflang Tags Missing or Incorrect — International SEO Issues
Sites with correct hreflang implementation see a 15-20% increase in international traffic and reduced bounce rates from international visitors (JEMSU, 2024). But 75% of hreflang implementations contain errors — 31% have conflicting directives and 16% are missing self-referencing tags (JEMSU). Bad hreflang is worse than no hreflang, so accuracy matters.
Open guideInternal Linking Gaps — Orphaned Pages and Weak Authority Flow
URLs with 40-44 internal links receive 4x more Google traffic than URLs with 0-4 internal links (Internal Linking SEO study, 2024). A SearchPilot experiment found that expanding internal links yielded a 5% uplift in organic traffic to destination pages, with another project seeing a 9,500 weekly increase in organic visits within three weeks. Orphaned pages with no internal links stay invisible to both crawlers and users.
Open guideMeta Description Missing or Poorly Optimised
Pages with optimised meta descriptions see up to 5.8% higher click-through rates, and using calls-to-action in descriptions can boost clicks by 20% (Webflow SEO research). Without a meta description, Google generates its own snippet from random page content — which rarely sells your page as well as you would. Keep descriptions between 120-155 characters for best results.
Open guideStructured Data (Schema Markup) Missing — No Rich Results
Rich results account for 58% of clicks in search results, with FAQ schema achieving an average 87% CTR and product schema boosting clicks by 25% (Lantern Digital/Schema Markup Statistics, 2026). One case study showed a 20% CTR increase within 30 days of adding structured data, with overall traffic improving by 25% (SearchPilot). Without markup, you're invisible to rich result formats.
Open guidePage Title Missing, Weak, or Being Rewritten by Google
Google rewrites 76% of title tags, retaining only 35% of the original title content on average (Zyppy/Search Engine Land, Q1 2025). Titles between 51-60 characters have the lowest rewrite rate at 39-42%, and matching your H1 to your title drops the rewrite rate to 20.6% (Portent). If Google is rewriting your titles, you've lost control of your search appearance.
Open guideTwitter/X Card Meta Tags Missing — Plain Links in Social
Rich social cards with images and descriptions get significantly more engagement than plain text links. Without Twitter card meta tags, every time someone shares your link, it appears as a bare URL with no preview — reducing clicks, shares, and the viral potential of your content. This applies to X/Twitter and many other platforms that read card markup.
Open guideXML Sitemap Missing, Invalid, or Incomplete
Sitemaps help search engines use crawl budget more effectively by presenting a clear page hierarchy — especially critical for larger sites (Google Search Central). Without one, Google relies on link discovery alone, which means new pages, updated content, and deep pages may take weeks to get indexed. Note: Google ignores priority and changefreq values, so focus on accuracy and completeness.
Open guide