Robots.txt Generator
Build a robots.txt file with per-crawler allow and disallow rules, an optional crawl delay, and a sitemap reference.
Rules
robots.txt
Use * as the user-agent to apply a rule to every crawler, or name a specific
bot like Googlebot. Disallow blocks a path from being crawled, Allow
explicitly permits one (useful for carving out an exception inside a disallowed folder).
Common questions
Does robots.txt actually block pages from Google?
It blocks crawling, not necessarily indexing. A disallowed URL can still appear in search results (without a description) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag instead, which requires the page to be crawlable.
Where does robots.txt need to go?
At the root of your domain, exactly at https://example.com/robots.txt, nowhere else. Crawlers won't look for it in a subfolder.
What does an empty Disallow mean?
Disallow: with nothing after it means "disallow nothing", equivalent to allowing everything for that user-agent. That's the default this tool starts with for Allow: /.