Robots.txt Generator
Generate a robots.txt file to control which pages search engine crawlers can access.
How to Use
- Select the default rule: allow all crawlers or block all.
- Add specific bot rules (e.g. block Googlebot from /admin/).
- Add your sitemap URL.
- Copy the output and save it as robots.txt in your site root.
Frequently Asked Questions
QDoes robots.txt prevent Google from indexing a page?
No — blocking crawling via robots.txt does not prevent indexing. Google can still index a URL it's never crawled if other sites link to it. Use the noindex meta tag to prevent indexing.
QWhere should robots.txt be placed?
In the root of your domain — e.g. https://example.com/robots.txt. It must be publicly accessible.
QCan I have multiple User-agent blocks?
Yes — you can create separate rules for different crawlers. Use "User-agent: *" as a catch-all for any crawler not specifically listed.
About This Tool
The Robots Exclusion Protocol (robots.txt) has been the standard for crawler communication since 1994. While well-behaved crawlers respect it, malicious bots may ignore it entirely.