Free robots.txt Generator Tool Online
Generate a valid robots.txt file with custom user-agent rules, allow and disallow directives, and sitemap support.
IN: Rules & paths · OUT: robots.txt file · RUNS: 100% in your browser
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap-index.xml
Frequently asked questions
Where do I place my robots.txt file?+
It must be placed at the root of your domain, e.g. https://example.com/robots.txt, for search engine crawlers to find it automatically.
What does Disallow actually do?+
Disallow tells well-behaved crawlers not to fetch matching paths. It's a request, not a security measure — it does not prevent access, only discourages compliant crawlers from indexing those URLs.
Can I have different rules for different crawlers?+
Yes, use the '+ Add another user-agent block' button to create separate rule sets, for example one block for '*' (all crawlers) and another specifically for 'Googlebot'.
Should I include my sitemap in robots.txt?+
Yes, adding a Sitemap: line is a widely supported convention that helps search engines discover your sitemap.xml quickly, in addition to submitting it via Search Console.
Does Disallow: / block my entire site?+
Yes, a bare 'Disallow: /' under 'User-agent: *' blocks crawling of your entire site, which is useful for staging environments but should be removed before launching production.
Is this generator sending my site structure anywhere?+
No, the file is generated entirely client-side from the fields you fill in — nothing about your site is transmitted.