Robots.txt Generator
Build a valid robots.txt file with allow/disallow rules, crawl-delay and sitemap directives, then copy or download it. Free, instant and generated entirely in your browser.
Generated robots.txt
Place the finished file at the root of your site, for example https://www.example.com/robots.txt.
How to use the Robots.txt Generator
- Pick a starting point. Choose a quick preset such as “Allow all” or “Block AI crawlers”, or add your own rule groups from scratch.
- Add your rules. For each group, set the user-agent(s) and list the paths to disallow or allow, plus an optional crawl-delay. Add sitemap URLs at the bottom.
- Copy or download. The robots.txt updates live. Copy it or download the file and upload it to the root folder of your website.
About this tool
A robots.txt file tells search engine crawlers and other bots which parts of your site they may or may not request. This generator builds a syntactically correct file following the Robots Exclusion Protocol: each rule group starts with one or more User-agent lines, followed by Disallow and Allow directives and an optional Crawl-delay, with global Sitemap lines at the end. The live preview means you always see exactly what will be written, so there is no guesswork about spacing or ordering.
Use it to keep private folders out of search results, throttle aggressive crawlers, point search engines to your XML sitemap, or block the growing number of AI training bots. Everything is generated locally in your browser — your paths and domains are never uploaded — and the output is plain text you can paste anywhere. Remember that robots.txt is a public, advisory file: well-behaved crawlers respect it, but it is not a security barrier, so never rely on it alone to hide sensitive pages.
Frequently asked questions
Where do I put the robots.txt file?
It must live at the very root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there — a robots.txt inside a subfolder is ignored. Upload the downloaded file to your web root via FTP, your hosting panel or your site's static files.
Does an empty Disallow line block or allow everything?
An empty Disallow: value allows everything for that user-agent — it is the standard way to say “crawl the whole site.” To block an entire site, use Disallow: / instead.
Will robots.txt keep a page out of Google?
Not reliably. Disallowing a URL stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or protect it with authentication.
Can I really block AI crawlers with this?
The “Block AI crawlers” preset adds disallow rules for known AI bots such as GPTBot, ClaudeBot, CCBot and Google-Extended. Compliant crawlers honor these rules, but robots.txt is voluntary, so it cannot force a badly behaved bot to obey.