Robots.txt Generator
Build a correct robots.txt file in seconds, with sitemap lines, a WordPress preset and AI crawler blocking.
How it works
- Choose the default rule and add paths to block or allow.
- Add your sitemap URL and any extra options.
- Copy or download robots.txt and upload it to your site root.
About this tool
The robots.txt file tells search engine crawlers which parts of your site they may visit. It sits at the root of your site, for example example.com/robots.txt. A small mistake, such as Disallow: /, can hide a whole site from search, so the tool warns you about risky rules.
Use the WordPress preset to block the admin area and search pages while keeping admin-ajax available. You can also add rules that block known AI training crawlers. Remember that robots.txt is a request, not a security tool, so never rely on it to hide private files.
Questions
Where do I put the robots.txt file?
In the root folder of your domain, so it opens at https://your-site.com/robots.txt.
Does robots.txt remove pages from Google?
Not reliably. It stops crawling, but a blocked page can still be listed. Use a noindex tag to keep a page out of search results.
Should I block CSS and JavaScript?
No. Google needs them to render and understand your pages.
Does Google use Crawl-delay?
No. Google ignores it, though some other crawlers follow it.