AI Robots.txt Generator: Block AI Bots Free

AI Bot Robots.txt Generator

Choose which AI crawlers may read your site and get the robots.txt rules to copy.

Start with

Blocked crawlers are blocked from

One path per line, starting with a slash.

Leave this off unless you need it: a crawler that is not mentioned simply follows your rules for all crawlers.

Add the rules to my current robots.txt (optional)

robots.txt is a request, not a lock. The crawlers listed here say they follow it, except where noted: some fetch a page anyway when a person asks for it. To be certain, also block the crawler in your firewall.

Crawler list checked against each operator's own documentation on 15 November 2026.

An AI robots.txt generator writes the rules that block or allow AI crawlers, such as GPTBot, on your website. You pick a preset, adjust individual crawlers, choose whether to block the whole site or only some folders, then copy or download the finished file. It is free and built for site owners, publishers and SEOs.

What Is a Robots.txt File for AI Crawlers?

A robots.txt file is a plain text file at the root of your domain that tells crawlers which URLs they may request. Each group starts with a User-agent line that names a crawler, followed by Allow or Disallow lines. AI companies publish their own crawler names, so you can write rules for them separately from Googlebot.

Not every AI crawler does the same job, and that difference drives your decision. The generator sorts crawlers into three purposes:

  • Training: collects pages to build or improve AI models. GPTBot is an example.
  • AI search: indexes pages so an AI search engine can cite and link to them. OAI-SearchBot is an example.
  • User requests: fetches a page only when a person asks an assistant to open it. ChatGPT-User is an example.

Robots.txt is optional. Without one, well-behaved crawlers assume your whole site is open. You need a file only when you want to restrict something. When a crawler visits, it looks for the group that matches its own name. If none matches, it follows the group written for all crawlers (User-agent: *).

How to Use the AI Bot Robots.txt Generator

You can build a complete set of AI crawler rules in a few minutes. Follow these steps in order:

  1. Under Start with, choose a preset: Block AI training only, Block every AI crawler or Allow every AI crawler. A short note under the buttons explains what the preset does.
  2. Review the crawler list and switch any single crawler to block or allow. Tick Show every crawler in our list, not only the best known ones to see the full list.
  3. Under Blocked crawlers are blocked from, pick the whole site or only these folders or pages. For folders, type one path per line, starting with a slash, such as /blog/ or /downloads/report.pdf.
  4. Add a Sitemap address (optional), for example https://example.com/sitemap.xml.
  5. Tick Write Allow rules for the crawlers I allow only if you want explicit Allow groups. Leave it off otherwise, because a crawler you do not mention follows your rules for all crawlers.
  6. If you already have a file, open Add the rules to my current robots.txt (optional), paste it into Your current robots.txt and keep Remove my old rules for these crawlers checked.
  7. Read Your robots.txt rules, then click Copy or Download robots.txt and upload the file to your domain root.

Worked Example: Keep AI Training Out of Paid Content

Here is how the tool handles a common request. Imagine a publisher whose free articles live in /blog/, while paid material sits in /members/ and a PDF at /downloads/report.pdf. The owner is happy to be quoted in AI search but does not want training crawlers reading the paid content.

You would choose Block AI training only, select only these folders or pages, and enter /members/ and /downloads/report.pdf on separate lines. Then you add the sitemap address. The output has one group for each training crawler, with its User-agent line and two Disallow lines, plus a Sitemap line. AI search and user-request crawlers get no block, so they follow your general rules.

What this means in practice: GPTBot may still read /blog/ but is asked to stay out of /members/ and the report PDF. If you had picked the whole site instead, each blocked crawler would get a single Disallow: / line. If you also pasted a file containing User-agent: * and Disallow: /admin/, that rule for other crawlers would stay, and any older rules for the crawlers you are changing would be replaced.

Which Preset Fits Your Goal?

The right preset depends on whether you want to be cited by AI search engines. Use this table as a starting point:

PresetBlocksEffect
Block AI training onlyTraining crawlersOpts out of training, stays eligible for AI search
Block every AI crawlerTraining, search, user requestsPages cannot be quoted or linked by those AI engines
Allow every AI crawlerNothingClears earlier blocks or writes explicit Allow rules

Most sites that want traffic from AI answers choose the first row. The second suits sites that want no AI use at all, accepting that they will not be cited.

How to Read and Act on Your Results

The text in Your robots.txt rules is the exact content to publish. Each blocked crawler has its own group, and the generated block sits between two comment lines that mark where it begins and ends, so you can find and update it later.

What a good result looks like

A good result blocks only the crawlers and paths you meant to block. Read each group once. If you chose a folder scope, every Disallow line should be a folder or file you really want protected. If a warning appears above the output, read it before publishing.

What a poor result looks like

The most damaging outcome is a whole-site block you did not intend. If you see Disallow: / under a crawler you want citing you, switch that crawler to allow or change the scope to specific paths. Another poor result is a merged file with conflicting groups for the same crawler. Keep Remove my old rules for these crawlers checked to avoid that.

After you publish

Upload the file so it loads at yourdomain.com/robots.txt, then confirm it behaves as planned. The robots.txt rule tester shows whether a specific URL is allowed for a given bot, and the AI crawler access checker shows which AI bots your live file allows or blocks for a page.

Who Should Use This Generator?

Anyone who wants a clear say in how AI companies use their pages can use it. These are the situations where it saves the most time:

  • Publishers and bloggers who want to opt out of model training but stay visible in AI search answers.
  • Membership, course and paywall sites that need to keep specific folders away from every AI crawler.
  • Agencies that set the same AI policy across many client sites and need a consistent, tidy block of rules.
  • SEOs with an existing file who want to add AI rules without retyping or overwriting what is already there.
  • Site owners preparing for AI search who first want to see where they stand with the AI search readiness checker.

If you need rules for Googlebot, Bingbot or crawl delay as well, the general robots.txt generator covers those.

Best Practices, Common Mistakes and Troubleshooting

Most robots.txt problems come from small slips, not bad strategy. These are the ones to watch for:

  • Wrong location. The file must sit at the root of the host. A file inside a folder is ignored, and each subdomain needs its own file.
  • Overwriting your live file. Paste your current file into the merge box instead of replacing it, so rules for Googlebot and other crawlers survive.
  • Paths without a leading slash. Write /blog/, not blog. The tool adds a missing slash, but check the output, because paths are case sensitive on most servers.
  • Blocking more than intended. Disallow: / blocks everything. Use the folder option when only part of your site is sensitive.
  • Forgetting new crawlers. Operators add and rename crawlers over time. Regenerate your rules now and then.
  • Bots still showing up. Compare the exact user-agent name in your server logs with the names in your file. The AI bot log analyzer shows which AI crawlers reach your site and which pages they read.

Limitations You Should Know About

Robots.txt is a request, not a lock. The crawlers in the list say they follow it, except where noted, but some fetch a page anyway when a person asks an assistant to open it. A few crawlers ignore the file entirely. If you must stop a crawler for certain, also block it in your firewall.

Blocking also does not remove content a crawler already collected. And choosing Block every AI crawler means AI search engines cannot quote or link your pages, which may cost you referral visits. The crawler list is checked against each operator's own documentation on the date shown under the tool, but operators can change names at any time.

You now have what you need to set a clear AI crawler policy: pick the preset that matches your goal, scope it to the right folders, merge it with your current file and publish it at your domain root. Use the generator on this page to build your rules, then explore more tools in the AI SEO tools collection to verify the result.

Frequently Asked Questions

How do I block AI bots with robots.txt?
Add a User-agent group for each AI crawler, followed by a Disallow line. The generator writes these groups for you: choose a preset, then click Copy or Download robots.txt. Upload the file to your domain root so it loads at yourdomain.com/robots.txt.
Is robots.txt necessary?
No, robots.txt is optional. Without it, well-behaved crawlers assume they can fetch your whole site. You need one only to restrict access, such as keeping AI training crawlers or private folders like /members/ out.
How does robots.txt work with AI crawlers?
Robots.txt is a text file at your domain root that tells crawlers which paths they may request. Each crawler follows the group that matches its name, and any crawler without its own group follows the rules written for all crawlers (User-agent: *).
Why are AI bots ignoring my robots.txt?
Some assistants fetch a page when a user asks, and a few crawlers never follow robots.txt. Compare the exact user-agent name in your server logs with your rules, then add a firewall block for any crawler you must stop.
What is the difference between blocking AI training and blocking AI search?
Training crawlers collect pages to build models, while AI search crawlers index pages so they can be cited and linked. The Block AI training only preset opts out of training but keeps your pages eligible for AI search results.
Is the AI Bot Robots.txt Generator free, and can I merge my existing file?
Yes, the generator is free and needs no sign-in. Open Add the rules to my current robots.txt, paste your file, and keep Remove my old rules for these crawlers checked so old and new AI rules do not conflict.
Recommended

Explore VPN options

A VPN changes your network exit. It does not guarantee anonymity.

Hide Your IP Encryption Network coverage
Protect My IP Now