Every search engine crawler asks one question before it reads your site: what am I allowed to fetch? The answer lives in a small text file called robots.txt at the root of the domain. This generator writes that file for you from a few choices, shows the result as you change them and lets you copy it or download it as robots.txt, ready to upload.
Create the file in three steps
- Choose whether to allow everything with exceptions or to block the whole site.
- List the folders or files that crawlers should stay out of, one path per line, and add exceptions inside them if you need any.
- Paste your sitemap address, then press Copy or Download robots.txt and upload the file to the root of your site.
What the directives mean
- User-agent: * addresses every crawler that has no block of its own.
- Disallow names a path that must not be fetched. An empty Disallow means that nothing is blocked.
- Allow opens a path inside a blocked folder, for example a public help page inside /admin/.
- Sitemap points crawlers to your sitemap.xml so that they find new pages sooner.
- Crawl-delay asks a crawler to wait between requests. Google ignores it, while some other engines respect it.
Keeping AI crawlers out
The option to block AI training crawlers adds a separate block for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and several others, each with Disallow: /. Search engines are not affected, so your pages stay findable. Well-behaved bots follow these rules, though a robots.txt is only a request and not a technical barrier.
Common mistakes to avoid
Blocking a page in robots.txt does not remove it from search results, because a blocked URL can still be listed if other sites link to it. Use a noindex tag for that. Also do not block the CSS and JavaScript files that a page needs, or search engines cannot render it properly. The file must sit at the top level and be named in lowercase.