Type a tool name, for example "keyword density" or "transcript".
See which AI crawlers visit your site, which pages they read and how often your server turns them away.
Apache, Nginx, IIS and JSON logs (for example Cloudflare Logpush). Large files are fine: the log is read piece by piece. Compressed files (.gz, .zip) must be unzipped first.
The log is read in your browser. It is never uploaded or saved.
No crawler from our list was found in this log. Either none visited in this period, or the log does not record user agents.
A user agent can be faked. This tool counts the requests that claim to be each crawler: it does not check their IP addresses against the operators' published ranges.
Only crawlers in our list are counted. Names that exist only in robots.txt, such as Google-Extended, never appear in a log.
An AI bot log analyzer reads your server access log and shows which AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, visited your site, which pages they requested and how often your server refused them. It is built for site owners and SEOs who want evidence of AI crawler activity instead of guesses.
AI crawler log analysis means counting the requests in your web server's access log that come from AI companies' bots. Every line in an access log records one request: the visitor's IP address, a timestamp, the page requested, the status code your server returned and the user agent, which is the name the visitor gives itself.
AI crawlers announce themselves through that user agent. A line containing GPTBot/1.3 claims to be OpenAI's training crawler, and a line containing ClaudeBot/1.0 claims to be Anthropic's. Because these names appear only in your logs, a log file is the one place that shows what AI bots actually did, as opposed to what your robots.txt file asks them to do.
Not every AI bot does the same job, so the report groups requests into four purposes:
This split matters because many site owners want to allow AI search and user-triggered fetches while declining training. You cannot decide that sensibly without seeing which group is actually visiting.
The status code is the second key signal. A 200 means the bot received the page. A 301 means it was redirected, a 404 means the page was missing, and a 403 or 429 means your server, firewall or CDN refused or throttled the request. The report counts these refusals per crawler.
You can get a full report in a few minutes with a log file or a handful of pasted lines. Follow these steps:
/page?utm_source=x and /page are counted as one page.Want to see the output before touching your own data? Press "Load sample" to fill the box with example lines, then press "Clear" when you are done.
Suppose you paste lines from one day and the tool recognizes them as an Apache or Nginx access log. Here is what a realistic mix looks like and how to read it.
/, /pricing and /blog/how-to-bake-bread?utm_source=feed, all with status 200. That is 3 training requests, all served./docs/start and /docs/install and gets a 403 both times. Every request from that crawler was refused./products/widget and gets a 429, which means your server is rate limiting it.With the query-string option off, the bread post shows 3 requests in "Most requested pages": one each from GPTBot, OAI-SearchBot and ChatGPT-User. With the option on, the GPTBot visit is listed separately because of the tracking parameter. The takeaway: OpenAI's bots can read your content, while ClaudeBot is blocked on your docs. If you did not intend that, you have found a firewall or CDN rule to fix.
Each section answers a specific question, and each has a clear next step. The table below summarizes the main signals.
| What you see | What it means | What to do |
|---|---|---|
| Many 403 or 429 responses | The bot is blocked or throttled | Check firewall, CDN bot rules and robots.txt |
| Mostly 200 responses | The bot reads your pages | Confirm that this matches your policy |
| Many 404 responses | The bot requests missing URLs | Fix broken links or add redirects |
| Heavy AI training share | Most AI traffic collects data | Decide whether to restrict training bots |
The "Crawlers" table lists requests per crawler, its share of all requests, and when it was first and last seen in UTC. A bot that was seen once, long ago, is less relevant than one that returns daily. A refusal summary beneath the table shows how often your server turned each bot away.
This table shows which URLs AI bots read most. Pages you want quoted in AI answers should appear here. If your key product or guide pages are missing while old archive pages dominate, check your internal links and sitemap. You can run an AI crawler view on a missing page to see what a crawler would read on it.
Daily counts reveal spikes. A bot that sends hundreds of requests in a few hours may strain a small server, which is a good reason to set rate limits. Dates are always in UTC, so a late-evening visit in your timezone may land on the next day. For a long log, the daily table may list only the most recent days.
Anyone who needs real crawler data will get value from it. Typical situations include:
A few habits make the report far more reliable.
Once you know who is visiting, act on it. Use an AI robots.txt generator to allow or block specific bots, then verify the rules with an AI crawler access checker for any page. If you also want to track visits that arrive from AI assistants in analytics, an AI traffic regex generator helps you build filters for ChatGPT, Perplexity and Claude referrals.
The analyzer reports what your log says, and that has limits. A user agent can be faked, so the tool counts requests that claim to be each crawler. It does not check IP addresses against the operators' published ranges, so verify IPs separately before you block a range or accuse a company of misbehaving.
Only crawlers the tool recognizes are counted. Names that exist only in robots.txt, such as Google-Extended, never appear in a log, so you will not see them here. A CDN may also answer requests before they reach your origin server, so an origin log can undercount bot traffic. Export logs from the CDN layer if you use one.
Finally, a log shows what bots requested, not whether they used your content. Treat the report as proof of access, not proof of citation. To judge how ready a page is to be quoted, pair it with an AI search readiness check, and browse the AI SEO tools hub for related checks.
Upload your access log or paste a few lines into the tool on this page and you will see which AI crawlers visit, what they read and where your server says no. Start with the sample if you want to see the report first, then run your own log.
Your report has been submitted. We will look into this issue.
A VPN changes your network exit. It does not guarantee anonymity.
Themes, plugins, code, video, audio, graphics & more from 90,000+ authors.
Web Development
Design & Media
Unlimited Access
Explore Envato Market