Robots.txt Generator
Build a robots.txt file with per-crawler allow and disallow rules, an optional crawl delay, and a sitemap reference.
Rules
robots.txt
Use * as the user-agent to apply a rule to every crawler, or name a specific
bot like Googlebot. Disallow blocks a path from being crawled, Allow
explicitly permits one (useful for carving out an exception inside a disallowed folder).
How it works
The generator collects your rule rows and groups them by user-agent, so three rules you
wrote for Googlebot in three separate rows get merged under a single
User-agent: Googlebot line, with each group separated by a blank line the way
parsers expect. Rules with an empty path are skipped, and every value has newlines stripped
out before it is written, so pasting something weird into a field can't inject an extra
directive onto its own line. Delete all the rules and the output falls back to
User-agent: * with an empty Disallow:, which is the valid way to
say "block nothing". The Sitemap line goes at the end, and the download button just wraps
the text in a Blob and clicks a temporary link, no server involved.
Two things about robots.txt that trip people up. First, Crawl-delay is in this
tool because Bing and Yandex honor it, but Googlebot ignores it completely and has since
Google formally dropped unsupported directives in 2019; for Google you manage crawl rate
through Search Console or your server. Second, and more important: Disallow does not remove
a page from search results. It only stops crawling. A blocked URL can still get indexed
from links pointing at it, showing up with no snippet. If you want a page out of the index,
do the opposite: let it be crawled and put a noindex meta tag on it. Block it
in robots.txt and Google can never see that noindex.
The file is assembled entirely in this page's JavaScript as you type. Your paths, bot names, and sitemap URL are never sent anywhere; you can confirm it in the DevTools Network tab, which stays silent while you edit and download.
Common questions
Does robots.txt actually block pages from Google?
It blocks crawling, not necessarily indexing. A disallowed URL can still appear in search results (without a description) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag instead, which requires the page to be crawlable.
Where does robots.txt need to go?
At the root of your domain, exactly at https://example.com/robots.txt, nowhere else. Crawlers won't look for it in a subfolder.
What does an empty Disallow mean?
Disallow: with nothing after it means "disallow nothing", equivalent to allowing everything for that user-agent. That's the default this tool starts with for Allow: /.