Robots.txt Generator

Create a robots.txt file for your website. Free, no sign-up, works in your browser.

Default for all robots Crawl-delay s

Disallow these paths (one per line)But allow (one per line) Sitemap URLs (one per line)

🔒 100% private — processing happens in your browser. Nothing is uploaded.

How to use the robots.txt generator

Choose whether robots may crawl everything, then list folders to keep out (admin, cart, internal search).
Add exceptions, your sitemap URL and, if you want, block AI training crawlers.
Download robots.txt and upload it to the root of your site (example.com/robots.txt).

robots.txt tells search engines which parts of a site not to crawl. It is not a security feature — blocked pages can still appear in results if other sites link to them; use a noindex tag or a password for that. Google ignores Crawl-delay, but Bing and Yandex respect it.