Robots.txt Generator
Build a valid robots.txt in seconds — allow rules, disallow paths, AI crawler blocks and your sitemap line.
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart Disallow: /*?sessionid= Sitemap: https://example.com/sitemap.xml
robots.txt is the first file a search engine requests from your domain. It tells crawlers which paths they may fetch and where your sitemap lives. This robots.txt generator writes a syntactically valid file for you, so you never ship the one-character mistake that de-indexes an entire site.
The rules are simple but unforgiving. Disallow: / under User-agent: * blocks your whole site — the single most damaging SEO typo there is. Paths are case-sensitive, must start with a slash, and support * as a wildcard and $ to anchor an ending. robots.txt controls crawling, not indexing: to keep a page out of results use a noindex meta tag, because a blocked page can still be listed if other sites link to it.
The AI crawler toggles add explicit blocks for GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot if you would rather your content not be used for model training. Save the output as robots.txt in your site root — it must be at /robots.txt, not in a subfolder.