Robots Meta Tag vs X-Robots-Tag: Differences and When to Use Each

Search engines accept page-level instructions — "do not index this", "do not show a snippet", "do not follow links" — in two places: a meta tag inside the HTML, or an HTTP header sent by the server. They support the same directives but suit different situations.

The robots meta tag

Placed in the HTML head, for example <meta name="robots" content="noindex, follow">.

• Easy to set per page in most CMSs and SEO plugins.
• Visible in the page source.
• Only works for HTML pages.
• You can target one crawler: name="googlebot".

The X-Robots-Tag header

Sent in the HTTP response, for example X-Robots-Tag: noindex.

• Works for any file type: PDFs, images, videos, spreadsheets.
• Can be applied to whole folders or file types with server rules.
• Invisible in the page source — easy to forget, and easy to apply by mistake.

Common directives

• noindex — do not show this in search results.
• nofollow — do not follow links on this page.
• nosnippet — no text snippet or video preview.
• max-snippet:[number] — limit snippet length.
• max-image-preview:large — allow large image previews (useful for Discover).
• noimageindex — do not index images on the page.
• unavailable_after:[date] — stop showing after a date.

When to use each

• Normal pages: meta tag.
• PDFs and other files: X-Robots-Tag.
• Whole sections or staging sites: header rules at the server — but password protection is safer for staging.

Important rules

• Crawlers must be able to fetch a page to see either instruction — do not also block it in robots.txt. See noindex vs disallow.
• If both exist with different values, search engines generally apply the most restrictive.
• Check after migrations and server changes — headers often get copied by accident. Read excluded by noindex tag fix.

Check both quickly

The bulk indexability checker shows the meta robots value and the X-Robots-Tag header for many URLs at once, so hidden header problems stand out.

A worked example

A US university department had hundreds of outdated PDF brochures appearing in search. Editing each PDF was impossible, so the web team added an X-Robots-Tag noindex header for PDFs in the old brochures folder only. The outdated files dropped out of results over the following weeks, while current PDFs elsewhere stayed indexed.

Where mistakes usually hide

• Server rules copied from a staging site that add noindex to everything.
• CDN or security settings adding headers to whole folders.
• Plugins that noindex categories, tags or media pages by default.
• PDF folders with blanket rules that also catch important documents.

After any hosting, CDN or plugin change, check a sample of key URLs for both the tag and the header.

Related guides

• Google Search Console vs Bing Webmaster Tools: Do You Need Both?
• Alternate Page With Proper Canonical Tag: What It Means
• JavaScript SEO Basics for Beginners

More free tools: browse all our free SEO tools, no sign-up needed.

Frequently asked questions