How to use Robots.txt Generator
- Add your sitemap URLPaste the full address of your XML sitemap into Sitemap URL, or leave this optional field empty.
- Choose the default ruleUnder Default rule, pick Allow all to let every crawler in, or Disallow all to ask all crawlers to stay out.
- Generate and upload the filePress Run Robots.txt Generator, copy the result and save it as robots.txt in the root folder of your site.
Common uses
- Launching a new site that should be open to search engines
- Asking crawlers to keep out of a staging or test site
- Adding a Sitemap line so crawlers can find your sitemap
Frequently asked questions
How do I use Robots.txt Generator?
1. Add your sitemap URL: Paste the full address of your XML sitemap into Sitemap URL, or leave this optional field empty. 2. Choose the default rule: Under Default rule, pick Allow all to let every crawler in, or Disallow all to ask all crawlers to stay out. 3. Generate and upload the file: Press Run Robots.txt Generator, copy the result and save it as robots.txt in the root folder of your site.
What does the generated file contain?
One group for all crawlers (User-agent: *) with either an empty Disallow line, which allows everything, or Disallow: /, which blocks everything. If you entered a sitemap URL, a Sitemap line is added at the end. The tool does not check that the sitemap exists.
Can I block specific folders or individual bots?
Not with this generator. It only writes an allow-all or disallow-all rule that applies to every crawler. To block particular paths or user agents, edit the file by hand after copying it.
Does Disallow all remove my pages from Google?
No. robots.txt asks well-behaved crawlers not to fetch pages, but it is not access control and does not guarantee removal from search results. Blocked URLs can still be listed if other sites link to them. To keep a page out of results, let it be crawled with a noindex tag or protect it with a password.
Is my input stored?
The form is sent to the GigaTools server, which builds the text and returns it straight away. The tool does not save your sitemap URL or the generated file, and the response is marked private and not cacheable.