Type a tool name, for example "keyword density" or "transcript".
Choose which AI crawlers may read your site and get the robots.txt rules to copy.
Start with
Leave this off unless you need it: a crawler that is not mentioned simply follows your rules for all crawlers.
robots.txt is a request, not a lock. The crawlers listed here say they follow it, except where noted: some fetch a page anyway when a person asks for it. To be certain, also block the crawler in your firewall.
Crawler list checked against each operator's own documentation on 15 November 2026.
An AI robots.txt generator writes the rules that block or allow AI crawlers, such as GPTBot, on your website. You pick a preset, adjust individual crawlers, choose whether to block the whole site or only some folders, then copy or download the finished file. It is free and built for site owners, publishers and SEOs.
A robots.txt file is a plain text file at the root of your domain that tells crawlers which URLs they may request. Each group starts with a User-agent line that names a crawler, followed by Allow or Disallow lines. AI companies publish their own crawler names, so you can write rules for them separately from Googlebot.
Not every AI crawler does the same job, and that difference drives your decision. The generator sorts crawlers into three purposes:
Robots.txt is optional. Without one, well-behaved crawlers assume your whole site is open. You need a file only when you want to restrict something. When a crawler visits, it looks for the group that matches its own name. If none matches, it follows the group written for all crawlers (User-agent: *).
You can build a complete set of AI crawler rules in a few minutes. Follow these steps in order:
Here is how the tool handles a common request. Imagine a publisher whose free articles live in /blog/, while paid material sits in /members/ and a PDF at /downloads/report.pdf. The owner is happy to be quoted in AI search but does not want training crawlers reading the paid content.
You would choose Block AI training only, select only these folders or pages, and enter /members/ and /downloads/report.pdf on separate lines. Then you add the sitemap address. The output has one group for each training crawler, with its User-agent line and two Disallow lines, plus a Sitemap line. AI search and user-request crawlers get no block, so they follow your general rules.
What this means in practice: GPTBot may still read /blog/ but is asked to stay out of /members/ and the report PDF. If you had picked the whole site instead, each blocked crawler would get a single Disallow: / line. If you also pasted a file containing User-agent: * and Disallow: /admin/, that rule for other crawlers would stay, and any older rules for the crawlers you are changing would be replaced.
The right preset depends on whether you want to be cited by AI search engines. Use this table as a starting point:
| Preset | Blocks | Effect |
|---|---|---|
| Block AI training only | Training crawlers | Opts out of training, stays eligible for AI search |
| Block every AI crawler | Training, search, user requests | Pages cannot be quoted or linked by those AI engines |
| Allow every AI crawler | Nothing | Clears earlier blocks or writes explicit Allow rules |
Most sites that want traffic from AI answers choose the first row. The second suits sites that want no AI use at all, accepting that they will not be cited.
The text in Your robots.txt rules is the exact content to publish. Each blocked crawler has its own group, and the generated block sits between two comment lines that mark where it begins and ends, so you can find and update it later.
A good result blocks only the crawlers and paths you meant to block. Read each group once. If you chose a folder scope, every Disallow line should be a folder or file you really want protected. If a warning appears above the output, read it before publishing.
The most damaging outcome is a whole-site block you did not intend. If you see Disallow: / under a crawler you want citing you, switch that crawler to allow or change the scope to specific paths. Another poor result is a merged file with conflicting groups for the same crawler. Keep Remove my old rules for these crawlers checked to avoid that.
Upload the file so it loads at yourdomain.com/robots.txt, then confirm it behaves as planned. The robots.txt rule tester shows whether a specific URL is allowed for a given bot, and the AI crawler access checker shows which AI bots your live file allows or blocks for a page.
Anyone who wants a clear say in how AI companies use their pages can use it. These are the situations where it saves the most time:
If you need rules for Googlebot, Bingbot or crawl delay as well, the general robots.txt generator covers those.
Most robots.txt problems come from small slips, not bad strategy. These are the ones to watch for:
Robots.txt is a request, not a lock. The crawlers in the list say they follow it, except where noted, but some fetch a page anyway when a person asks an assistant to open it. A few crawlers ignore the file entirely. If you must stop a crawler for certain, also block it in your firewall.
Blocking also does not remove content a crawler already collected. And choosing Block every AI crawler means AI search engines cannot quote or link your pages, which may cost you referral visits. The crawler list is checked against each operator's own documentation on the date shown under the tool, but operators can change names at any time.
You now have what you need to set a clear AI crawler policy: pick the preset that matches your goal, scope it to the right folders, merge it with your current file and publish it at your domain root. Use the generator on this page to build your rules, then explore more tools in the AI SEO tools collection to verify the result.
Your report has been submitted. We will look into this issue.
A VPN changes your network exit. It does not guarantee anonymity.
Themes, plugins, code, video, audio, graphics & more from 90,000+ authors.
Web Development
Design & Media
Unlimited Access
Explore Envato Market