Paste the actual rules
Copy your robots.txt content into the input. The tool does not fetch a live file.
Robots.txt Tester
Test pasted Allow and Disallow rules against crawler tokens and URL paths, with winning rule lines and no live website requests.
Free
No sign-up
Runs in your browser
Result
Your result will appear here
Allowed means no selected rule blocks this path. It does not mean Google has crawled or indexed it. Check robots meta tags separately.
What is Robots.txt Tester?
Robots.txt Tester is a free online seo tool from GigaTools. Test pasted Allow and Disallow rules against crawler tokens and URL paths, with winning rule lines and no live website requests. It runs entirely in your web browser, so nothing is uploaded and it works on any device without installing software.
Common uses
- Check whether a draft directory is blocked
- Compare a specific crawler group with wildcard rules
- Test Allow exceptions and PDF filename patterns
How it works
How to use Robots.txt Tester
Choose the crawler and paths
Enter a crawler product token such as Googlebot and up to 50 paths or full URLs. Test rules for the site the file belongs to.
Review the winning rule
Run the test and inspect the selected groups, allowed status and winning line. Fix your real file separately.
FAQ
Good to know
How do I use Robots.txt Tester?
1. Paste the actual rules: Copy your robots.txt content into the input. The tool does not fetch a live file. 2. Choose the crawler and paths: Enter a crawler product token such as Googlebot and up to 50 paths or full URLs. Test rules for the site the file belongs to. 3. Review the winning rule: Run the test and inspect the selected groups, allowed status and winning line. Fix your real file separately.
Does allowed mean indexed?
No. This checks pasted Allow and Disallow rules only. It does not inspect robots meta tags, X-Robots-Tag headers, canonical tags, HTTP responses or Google indexing.
Which matching rules are supported?
The tester selects the longest matching crawler token prefix and merges equally specific groups. It supports case-sensitive paths, query strings, * wildcards, terminal $ anchors, percent-encoded paths and Allow winning an equal-length conflict. It is a local approximation, not a Google certification or live crawler test.
What is ignored?
Empty Allow and Disallow values are ignored. Sitemap lines do not affect crawling rules. Unsupported directives such as Crawl-delay are reported but not evaluated. Invalid path rules are flagged and skipped; review warnings before relying on the result.
What are the limits?
Up to 500 KB of robots text, 500 groups, 500 non-empty rules and 50 paths, each up to 2,048 characters. Complex matching can stop with an error instead of freezing your browser.
Keep going
More seo tools
Backlink Maker
Create a ready-to-use HTML link snippet.
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
XML Sitemap Validator
Validate sitemap XML, count URLs, and spot malformed or unsafe entries.
Link Analyzer
Inspect links from pasted HTML and classify them.
Keyword CPC Calculator
Estimate campaign cost from clicks and CPC.
Keyword Search Review
See a best-effort domain match in DuckDuckGo results; open Google and Bing searches for manual review. No rank history is tracked.