Robots.txt Rule Tester
Explain crawler-group selection and allow/disallow path matching for a pasted robots file.
Loading tool…
How to use Robots.txt Rule Tester
Test an explicit product token and path against local rules. The trace uses longest matching rule specificity, merges equally specific groups, and favors Allow on a tie.
- Enter input, crawler product token, url or path.
- Choose the settings for robots.txt rule tester, then run the tool.
- Review the result and any limitations before copying or downloading it.
Good to know
- 2,000 lines, 500 selected rules, 512 characters per rule, and an operation budget.
- This path matcher does not model HTTP retrieval/error rules, crawl-delay extensions, access control or search-engine indexing.
- Processing is local. Output is limited to 1,000,000 text characters unless a stricter file limit is stated.
Common questions
Does Disallow stop a page being indexed?
Robots rules govern crawling, not authentication or a guaranteed indexing decision.
Is this input sent to a server?
No. The application processes this input inside your browser tab and does not save it to persistent storage.
Keep going with related tools
All seo →Robots.txt Generator
Build one crawler-rule group and optional sitemap lines with validated paths and URLs.
SEOXML Sitemap Auditor
Inspect a pasted sitemap or sitemap index for namespace, URL, date and duplicate-entry issues offline.
SEOSupported Google Robots Directive Resolver
Combine supplied generic/Googlebot robots tags and response directives under a named limited Google profile.