Robots.txt generator
Build a clean robots.txt with presets for WordPress and WooCommerce, optional AI-crawler blocking and your sitemap.
- Files never leave your device
- Instant, no waiting queue
- No watermark, no limits
How to use the Robots.txt Generator tool
- 1
Pick a preset and add your sitemap URL.
- 2
Add custom allow/disallow rules.
- 3
Copy or download robots.txt and upload it to your site root.
Create a clean robots.txt file in seconds
This robots.txt generator builds a valid robots.txt with presets for WordPress and WooCommerce, optional blocking of AI crawlers (GPTBot, CCBot and others), crawl rules for specific folders and your sitemap URL. Copy it or download the file and upload it to your site root.
Robots.txt best practices
- Never block CSS or JS files that Google needs to render pages.
- Use robots.txt to control crawling, not indexing – use noindex for that.
- Always include your sitemap URL.
- Test changes in Google Search Console's robots.txt report.
Create your sitemap with the XML sitemap generator.
Frequently asked questions
Is this tool free?
Yes, it is 100% free with no sign-up, no watermark and no daily limit.
Does robots.txt remove pages from Google?
No. It only stops crawling. To keep a page out of search results use a noindex meta tag instead.
Where do I upload robots.txt?
Place it in the root of your domain so it is available at yourdomain.com/robots.txt. In WordPress, SEO plugins like Rank Math can edit it for you.
How do I block AI bots like GPTBot?
Enable the AI crawler option to add Disallow rules for known AI user agents.
Does robots.txt remove pages from Google?
No. Blocked pages can still be indexed if linked elsewhere. Use a noindex tag to remove pages from search.
Is a robots.txt file required?
It is not required, but recommended to point crawlers to your sitemap and keep them out of admin or cart pages.