Robots.txt Generator
Build a valid robots.txt with crawl rules for search engine bots.
Why use this tool?
Create a robots.txt file that tells search engine crawlers which parts of your site they may visit, with a link to your sitemap.
How to use it
- Choose the rules and paths to allow or block.
- Add your sitemap URL.
- Copy the result into a robots.txt file at your site's root.
What robots.txt does
robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which paths they shouldn't fetch. Each group starts with a User-agent line naming a crawler (* means all crawlers) followed by Disallow lines with path prefixes. An empty Disallow line allows everything. A Sitemap line points crawlers to your XML sitemap.
The generator lets you add rules per user agent and a sitemap URL, then produces the file text. Save it as robots.txt and upload it to the top level of your domain; crawlers ignore it anywhere else.
Rules people commonly use
Blocking admin and account areas (Disallow: /admin/), internal search result pages (Disallow: /search), and endless filter or session-ID URLs that waste crawl budget. Paths are prefixes, so /admin also blocks /admin-panel and /administrator; end the path with a slash to target just the folder.
You can also give specific crawlers their own rules, for example to block an AI training crawler by its user-agent name while allowing search engines. Each crawler follows the most specific group that matches its name.
What robots.txt can't do
It is a request, not a lock. Well-behaved crawlers like Googlebot respect it, but nothing stops anyone else from fetching blocked paths, and the file itself is public, so never list secret URLs in it. Protect private pages with authentication instead.
Blocking a page from crawling doesn't remove it from search results: Google can still index a blocked URL if other sites link to it, just without its content. To keep a page out of results, allow crawling and add a noindex meta tag. A single wrong line (Disallow: /) blocks your whole site, so test the file in Google Search Console after uploading.
Frequently asked questions
Where do I put robots.txt?
At the root of your domain, so it's reachable at https://yourdomain.com/robots.txt. Subfolders are ignored.
Does Disallow hide a page from Google?
It stops crawling, not indexing. Use a noindex meta tag on a crawlable page to keep it out of search results.
Is robots.txt a security measure?
No. It's public and voluntary. Use passwords or authentication for anything private.