Robots.txt Generator
Create a valid robots.txt to control how crawlers index your site.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Generate a correct robots.txt file without memorising the syntax. Choose to allow everything, block everything, or disallow specific paths, optionally set a crawl-delay, and add your sitemap URL. Copy the result to your site root at /robots.txt.
How to use the Robots.txt Generator
- Pick a rule preset: allow all, block all, or custom.
- For custom, list the paths to disallow, one per line.
- Optionally add a crawl-delay and your sitemap URL.
- Copy the file and upload it to your site root as /robots.txt.
Frequently asked questions
Where do I put the robots.txt file?
At the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers always look there.
Does Disallow guarantee a page won't be indexed?
No. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to reliably keep a page out of search results.
Should I add my sitemap here?
Yes — adding a Sitemap line helps search engines discover all your URLs, and it works alongside submitting the sitemap in Search Console.