Robots.txt Generator
Build one crawler-rule group and optional sitemap lines with validated paths and URLs.
Loading tool…
How to use Robots.txt Generator
Prepare a robots.txt draft for the root of a website. The generator keeps the crawler token, allow/disallow paths, and sitemap URLs separate to reduce accidental directive injection.
- Enter one crawler token, optional paths, and absolute sitemap URLs.
- Select Run tool to generate a reviewable robots.txt file.
- Review the result, then copy it or download the generated file.
Good to know
- One User-agent group per run. Paths must begin with /; # characters and line injection are rejected. Wildcard handling depends on crawler support.
- Robots rules are public crawl preferences, not access control or a reliable way to remove an indexed URL. Publish at /robots.txt and verify crawler interpretation.
Common questions
Will Disallow hide private files?
No. Protect private resources with authentication and authorization; robots.txt does not restrict access.
Keep going with related tools
All developer →CSP Policy Builder
Build a conservative Content-Security-Policy response header using exact HTTPS source origins.
DeveloperURL Parser
Inspect HTTP(S) URL components, normalized hostnames, and repeated query parameters without visiting a link.
SEORobots.txt Rule Tester
Explain crawler-group selection and allow/disallow path matching for a pasted robots file.
SEOSecurity.txt Workbench
Build or inspect supported security.txt contact and disclosure-policy fields.
SEOXML Sitemap Generator
Build a sitemap from real HTTP(S) URLs and optional truthful last-modified dates.