Type a tool name, for example "keyword density" or "transcript".
See which AI crawlers your robots.txt allows or blocks for a page.
An AI crawler access checker shows whether AI bots such as GPTBot, ClaudeBot and PerplexityBot are allowed to fetch a specific page, based on your robots.txt. Enter a page address and you get an Allowed, Blocked or Unknown verdict for each crawler, the rule behind it, and any noindex or nosnippet rules on the page itself.
AI crawler access is whether an AI company's bot is permitted to request your pages. The main control is the robots.txt file at the root of your domain, which lists user agent names and the paths each one may or may not fetch.
Each rule has two parts: a user agent line that names the bot, and Allow or Disallow lines that name paths. A line such as Disallow: /blog/ under User-agent: GPTBot blocks that bot from every URL that starts with /blog/. Rules are matched by path, so one folder can be closed while the rest of the site stays open.
Robots.txt is only part of the picture. A page can also carry a robots meta tag or an X-Robots-Tag HTTP header with directives such as noindex or nosnippet. Those do not stop a bot from fetching the page, but they change whether the page can appear in search results and in AI answers built on them. Your server or firewall can also refuse a bot by name, whatever robots.txt says.
You need one full page address and about ten seconds. Follow these steps:
Say your robots.txt contains a group for GPTBot with Disallow: /blog/, and a general group for all other bots with Allow: /. You enter https://example.com/blog/my-post and run the check.
GPTBot shows Blocked, and the Why column quotes the Disallow line that matched. ClaudeBot and PerplexityBot show Allowed because the general group applies to them. If you test /pricing instead, GPTBot flips to Allowed, because the path no longer starts with /blog/.
Now suppose you also ticked the extra test, and PerplexityBot gets a 403 answer from your server. That tells you robots.txt welcomes the bot, but a firewall, security plugin or host rule is turning it away. Under "Rules on the page itself" you might also see noindex from the X-Robots-Tag header, which means the page is kept out of search results regardless of the crawler verdicts. Each finding points to a different fix, which is why the tool reports them separately.
Every crawler gets one of three verdicts, and each calls for a different response.
| Verdict | Meaning | What to do |
|---|---|---|
| Allowed | No rule stops this bot from the page | Nothing, unless you want it blocked |
| Blocked | A matching rule disallows the path | Edit or remove the rule if unintended |
| Unknown | The rules could not be read or applied | Fix robots.txt, then check again |
Crawlers are grouped by what they are used for, and each group shows a count such as 2 / 6 crawlers blocked. This lets you separate bots that collect training data from those that fetch pages for search or user requests. Well-known crawlers and anything not plainly allowed stay visible, while the remaining allowed ones fold under a "Show more allowed crawlers" link.
Some rows add context. A few entries are robots.txt controls rather than crawlers of their own. Some operators say a page a user asks for may be fetched regardless of robots.txt. Others publish no crawler documentation, so treat their verdicts with extra care.
This section lists directives found in the robots meta tag or the X-Robots-Tag header, with the source of each. A noindex keeps the page out of search results and the AI answers built on them. A nosnippet, or a max-snippet of 0, keeps its text out of Google AI Overviews and AI Mode. Tags such as noai and noimageai are informal and not part of any standard, so do not rely on them. If none are found, the tool says so.
If your robots.txt has content signal lines, the tool lists them. They state your preference for search, AI answers or AI training, but they do not block any crawler. The sitemaps listed in robots.txt appear too, which helps you confirm that crawlers can find them.
Anyone who wants to know, with evidence, what AI bots can reach on a given page.
This tool is built around AI crawlers, but the same logic applies to any bot: robots.txt groups are matched by user agent and path. Open Show the robots.txt we read and look for a Googlebot group or the general group to see what applies to Google. The page rules section is also useful for Google, because noindex and nosnippet directly affect search results and AI features in Google Search. Google's Search Central documentation explains how robots.txt rules are interpreted.
Test the pages that matter, not just the home page, and recheck after every change to robots.txt, your CDN or your security settings.
Disallow: / under User-agent: * closes the entire site to every bot that has no group of its own.After you fix access, the AI search readiness checker covers whether the page can also be read and quoted, and the AI SEO tools collection has more checks for the same goal.
The verdicts describe what your robots.txt says, not what every bot will do. Robots.txt is a voluntary standard, so well-behaved crawlers follow it, but nothing enforces it. Some operators say user-requested fetches may ignore it. If you must keep a page private, use server-level controls such as authentication, and verify them with the extra server test.
The tool checks one page per run and reads the file as it is served at that moment. If you change robots.txt, your host or a CDN may serve the older version for a while, so wait a little and run the check again. The list of crawlers reflects bots known to the tool, so a brand-new bot may not appear.
Enter the page you care about, press Check AI crawler access, and you will see in seconds which AI bots can reach it, which rule decides each verdict and what to change if the result is not what you intended.
Your report has been submitted. We will look into this issue.
A VPN changes your network exit. It does not guarantee anonymity.
Themes, plugins, code, video, audio, graphics & more from 90,000+ authors.
Web Development
Design & Media
Unlimited Access
Explore Envato Market