Meta Robots or X-Robots-Tag Blocking Indexing by Accident

TL;DR: This check looks at the two places a page can tell search engines what to do: the HTML robots meta tag and the HTTP X-Robots-Tag header. If either one effectively says noindex, the page can be kept out of search results even if the rest of your SEO looks fine.

This matters because teams often look only at page source and miss a header being added by the server, CDN, or edge platform. Google supports both surfaces, and when rules conflict it applies the more restrictive one. That means a single hidden noindex in a response header can quietly override what the template appears to say.

What Scavo checks (plain English)

Scavo reads both of these surfaces on the same URL:

• <meta name="robots" content="..."> in the page head
• any X-Robots-Tag headers returned with the response

Scavo then combines the observed directives and checks whether the page is effectively being told to stay out of the index.

How Scavo scores this check:

• Pass: no robots restrictions were found, or robots directives were found but they do not include noindex
• Warning: the effective directives include noindex
• Pass: robots directives exist but do not currently block indexing

Important nuance:

• Scavo records both noindex and nofollow in the details
• this check currently warns on effective noindex
• it does not fail or warn on nofollow alone

How Scavo scores this check

Scavo assigns one result state for this check on the tested page:

• Pass: baseline signals for this check were found.
• Warning: partial coverage or risk signals were found and should be reviewed.
• Fail: required signals were missing or risky behavior was confirmed.
• Info: Scavo could not gather enough reliable evidence on this run to score pass/fail confidently.

In your scan report, this appears under What failed / What needs attention / What is working for meta_robots, followed by Recommended next steps and Technical evidence (for developers) when needed.

• Scan key: meta_robots
• Category: SEO

Why fixing this matters

A page can have perfect titles, content, links, and structured data, but if it is effectively marked noindex, it still will not be eligible to appear in normal search results. That makes this one of the highest-leverage SEO checks in the whole system.

The practical danger is hidden drift. A developer may remove noindex from the HTML template and assume the problem is solved, while a CDN rule or framework middleware is still adding X-Robots-Tag: noindex in the response.

Common reasons this check warns

• A staging or preview noindex rule leaked into production.
• A plugin, middleware, or CDN rule is adding X-Robots-Tag: noindex.
• A page-level CMS override was set for testing and never removed.
• Launch checklists verified the HTML but not the response headers.

If you are not technical

1. Ask whether this page is supposed to be indexable.
2. Ask your developer or SEO owner for proof from both the HTML source and the live response headers.
3. If the page should rank, ask them to remove the accidental noindex from whichever layer is setting it.
4. Re-run Scavo on the same URL and confirm the warning clears.

Scavo flagged Meta Robots (meta_robots) because the effective robots directives for this URL include noindex. Please inspect both the HTML robots meta tag and any X-Robots-Tag response headers, remove accidental exclusions where appropriate, and share before/after evidence from the same URL.

If you are technical

1. Inspect the live HTML source for <meta name="robots">.
2. Inspect the live response headers for X-Robots-Tag.
3. Decide the intended outcome for this URL: indexable or intentionally excluded.
4. Remove accidental noindex from the layer that truly owns it: template, middleware, CDN, or server config.
5. If the page is intentionally excluded, document that decision so it is not “fixed” later by mistake.
6. Re-check representative templates, not just the single URL, if the rule is shared.

How to verify

• View live source and inspect the robots meta tag.
• Run curl -I https://your-url and inspect any X-Robots-Tag headers.
• Confirm the effective directives no longer include noindex on pages meant to rank.
• Use Search Console URL Inspection for extra confirmation after deployment.
• Re-run Scavo and confirm meta_robots returns to pass where expected.

What this scan cannot confirm

• It does not guarantee ranking or indexing speed; it checks directive state, not Google’s recrawl timing.
• It does not decide whether noindex is strategically correct for your business.
• It records nofollow, but it does not currently fail a page on nofollow alone.

Owner checklist

• [ ] Assign one owner for robots directives across templates and edge layers.
• [ ] Document which page groups are intentionally excluded from search.
• [ ] Add release checks for both HTML source and response headers.
• [ ] Revalidate after SEO plugin, CDN, or middleware changes.

FAQ

Why can this warn even if the page source looks fine?

Because X-Robots-Tag can be added in the response headers, and that is not always obvious from the visible HTML alone.

Does Google really support both meta robots and X-Robots-Tag?

Yes. Google documents both, and when there is a conflict it applies the more restrictive rule.

Is missing a robots tag a problem?

Not by itself. If no robots restrictions are present, default indexing behaviour is generally allowed.

What about nofollow?

Scavo records it in the details, but this check is mainly focused on whether the URL is effectively blocked from indexing via noindex.

Sources

• Google Search Central: Robots meta tag and X-Robots-Tag
• Google Search Central: Consolidate duplicate URLs
• MDN: <meta> element reference

Need help tracing where noindex is really being set? Send support one affected URL and whether you suspect template, middleware, or CDN ownership.