Choose the crawler
Use the generic robots name to address crawlers generally, or Googlebot for a Google-specific directive.
Create robots meta and X-Robots-Tag directives for a generic crawler or Googlebot.
Your result will appear here
Create robots meta and X-Robots-Tag directives for a generic crawler or Googlebot.
How it works
Use the generic robots name to address crawlers generally, or Googlebot for a Google-specific directive.
Choose index or noindex, follow or nofollow, and optional nosnippet and max-image-preview settings.
Copy the meta element into the page head or send X-Robots-Tag as an HTTP response header. You must make this change on your site yourself.
FAQ
1. Choose the crawler: Use the generic robots name to address crawlers generally, or Googlebot for a Google-specific directive. 2. Set indexing and preview directives: Choose index or noindex, follow or nofollow, and optional nosnippet and max-image-preview settings. 3. Add the generated directive: Copy the meta element into the page head or send X-Robots-Tag as an HTTP response header. You must make this change on your site yourself.
It asks compliant crawlers not to index an accessible page. A crawler must be able to fetch the page to read its meta tag or response header. It does not prevent access or secure content.
Yes. A robots meta element goes in the page head. X-Robots-Tag is sent as an HTTP response header and can also cover non-HTML files.
No. Crawlers may not follow directives, and indexing changes can take time. Keep the page crawlable while a crawler needs to read noindex.
Keep going
Create a ready-to-use HTML link snippet.
Turn a list of URLs into a valid XML sitemap.
Validate sitemap XML, count URLs, and spot malformed or unsafe entries.
Inspect links from pasted HTML and classify them.
Estimate campaign cost from clicks and CPC.
See a best-effort domain match in DuckDuckGo results; open Google and Bing searches for manual review. No rank history is tracked.