Robots.txt Generator
Generate a robots.txt file to control which pages search engine crawlers can access.
How to Use
- Select the default rule: allow all crawlers or block all.
- Add specific bot rules (e.g. block Googlebot from /admin/).
- Add your sitemap URL.
- Copy the output and save it as robots.txt in your site root.
When to Use This
- Standing up a new site and needing a file at the root on day one, so the sitemap location is advertised from the first crawl.
- Keeping crawlers out of faceted search and internal result pages that generate endless parameter combinations and absorb crawl capacity.
- Blocking a specific scraper by name after spotting it in server logs, without touching the rules that apply to everyone else.
- Writing a staging-server file that blocks everything, kept separate from the production file so the two cannot be confused.
Frequently Asked Questions
About This Tool
A robots.txt file is a plain text file at the root of a host telling crawlers which paths they may request. It is organized into groups: a User-agent line naming the crawler, then Disallow and Allow lines whose values are matched as path prefixes. The format was standardized as RFC 9309 in 2022 after decades as a shared convention, and well-behaved crawlers fetch it before anything else on the host.
This generator writes a wildcard group first, either Disallow: / for a full block or an empty Disallow line followed by whatever paths you list, then an optional Crawl-delay, then one group per named bot with a single Disallow path each, then a Sitemap line. The result can be copied or downloaded as robots.txt.
Notes & Limits
- Each named-bot row produces exactly one Disallow line, so blocking three paths for Googlebot means three rows and three repeated User-agent headers.
- Allow lines only ever appear in the wildcard group. A named bot gets no Allow exception here, and the Allow box is ignored entirely when you choose Block all.
- Crawl-delay is written if you fill it in, but Google's crawlers ignore the directive while Bing documents support for it, so it is not a portable control.
- Paths are copied as typed with no syntax checking. A rule missing its leading slash is written out looking exactly like one that works.