AI Bot Log Analyzer: Free AI Crawler Report

AI Bot Log Analyzer

See which AI crawlers visit your site, which pages they read and how often your server turns them away.

Apache, Nginx, IIS and JSON logs (for example Cloudflare Logpush). Large files are fine: the log is read piece by piece. Compressed files (.gz, .zip) must be unzipped first.

The log is read in your browser. It is never uploaded or saved.

Options

No crawler from our list was found in this log. Either none visited in this period, or the log does not record user agents.

Requests by purpose

Crawlers

Most requested pages

Requests per day

A user agent can be faked. This tool counts the requests that claim to be each crawler: it does not check their IP addresses against the operators' published ranges.

Only crawlers in our list are counted. Names that exist only in robots.txt, such as Google-Extended, never appear in a log.

An AI bot log analyzer reads your server access log and shows which AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, visited your site, which pages they requested and how often your server refused them. It is built for site owners and SEOs who want evidence of AI crawler activity instead of guesses.

What Is AI Crawler Log Analysis?

AI crawler log analysis means counting the requests in your web server's access log that come from AI companies' bots. Every line in an access log records one request: the visitor's IP address, a timestamp, the page requested, the status code your server returned and the user agent, which is the name the visitor gives itself.

AI crawlers announce themselves through that user agent. A line containing GPTBot/1.3 claims to be OpenAI's training crawler, and a line containing ClaudeBot/1.0 claims to be Anthropic's. Because these names appear only in your logs, a log file is the one place that shows what AI bots actually did, as opposed to what your robots.txt file asks them to do.

Why Crawlers Are Grouped by Purpose

Not every AI bot does the same job, so the report groups requests into four purposes:

  • AI training: bots that collect pages to train models, for example GPTBot.
  • AI search: bots that build the index behind an AI search product, for example OAI-SearchBot.
  • User requests: bots that fetch a page because a person asked an assistant about it, for example ChatGPT-User.
  • Search engines: classic crawlers such as Googlebot and bingbot, shown only for comparison.

This split matters because many site owners want to allow AI search and user-triggered fetches while declining training. You cannot decide that sensibly without seeing which group is actually visiting.

Status Codes Tell You What Happened

The status code is the second key signal. A 200 means the bot received the page. A 301 means it was redirected, a 404 means the page was missing, and a 403 or 429 means your server, firewall or CDN refused or throttled the request. The report counts these refusals per crawler.

How to Use the AI Bot Log Analyzer

You can get a full report in a few minutes with a log file or a handful of pasted lines. Follow these steps:

  1. Export an access log from your host, web server or CDN. If the file ends in .gz or .zip, unzip it first, because compressed files are not read.
  2. Under "Choose your access log, or drop it on this box", select the file or drag it in. Files ending in .log, .txt, .json, .jsonl or .ndjson are accepted. To test a few lines instead, paste them into "Or paste a few log lines" and press "Analyze pasted lines".
  3. Open "Options" if you need to change the defaults. "Also count search engine crawlers, for comparison" is on by default. "Keep the part after the question mark in page addresses" is off, so /page?utm_source=x and /page are counted as one page.
  4. Wait for the progress message. Large files are read piece by piece, and you can press "Cancel" to stop. Nothing is analyzed after a cancel.
  5. Review the results: the summary at the top, the date range the log covers, the format that was recognized, then "Requests by purpose", "Crawlers", "Most requested pages" and "Requests per day".
  6. Use "Show pages for" to filter the page table to one crawler. Press "Crawlers CSV" or "Pages CSV" to download a table for a report or spreadsheet.

Want to see the output before touching your own data? Press "Load sample" to fill the box with example lines, then press "Clear" when you are done.

Worked Example: Reading a Small Log

Suppose you paste lines from one day and the tool recognizes them as an Apache or Nginx access log. Here is what a realistic mix looks like and how to read it.

  • GPTBot requests /, /pricing and /blog/how-to-bake-bread?utm_source=feed, all with status 200. That is 3 training requests, all served.
  • OAI-SearchBot requests the bread post once, and ChatGPT-User requests it once with a chatgpt.com referrer. That is 1 AI search request and 1 user request.
  • ClaudeBot requests /docs/start and /docs/install and gets a 403 both times. Every request from that crawler was refused.
  • Amazonbot requests /products/widget and gets a 429, which means your server is rate limiting it.

With the query-string option off, the bread post shows 3 requests in "Most requested pages": one each from GPTBot, OAI-SearchBot and ChatGPT-User. With the option on, the GPTBot visit is listed separately because of the tracking parameter. The takeaway: OpenAI's bots can read your content, while ClaudeBot is blocked on your docs. If you did not intend that, you have found a firewall or CDN rule to fix.

How to Read and Act on the Results

Each section answers a specific question, and each has a clear next step. The table below summarizes the main signals.

What you seeWhat it meansWhat to do
Many 403 or 429 responsesThe bot is blocked or throttledCheck firewall, CDN bot rules and robots.txt
Mostly 200 responsesThe bot reads your pagesConfirm that this matches your policy
Many 404 responsesThe bot requests missing URLsFix broken links or add redirects
Heavy AI training shareMost AI traffic collects dataDecide whether to restrict training bots

Crawlers Table

The "Crawlers" table lists requests per crawler, its share of all requests, and when it was first and last seen in UTC. A bot that was seen once, long ago, is less relevant than one that returns daily. A refusal summary beneath the table shows how often your server turned each bot away.

Most Requested Pages

This table shows which URLs AI bots read most. Pages you want quoted in AI answers should appear here. If your key product or guide pages are missing while old archive pages dominate, check your internal links and sitemap. You can run an AI crawler view on a missing page to see what a crawler would read on it.

Requests per Day

Daily counts reveal spikes. A bot that sends hundreds of requests in a few hours may strain a small server, which is a good reason to set rate limits. Dates are always in UTC, so a late-evening visit in your timezone may land on the next day. For a long log, the daily table may list only the most recent days.

Who Should Use This Analyzer?

Anyone who needs real crawler data will get value from it. Typical situations include:

  • SEO specialists auditing whether AI search bots can reach key landing pages before an AI visibility project.
  • Publishers who want to know whether training crawlers are reading paywall-adjacent or premium content.
  • Developers and sysadmins investigating a traffic spike and checking whether bots caused it.
  • Agencies that need a CSV of AI crawler activity to include in a client report.
  • Site owners who changed robots.txt and want to confirm that bots respond as expected.

Best Practices, Common Mistakes and Troubleshooting

A few habits make the report far more reliable.

  • Use a long enough period. A single day can miss bots that visit weekly. Try at least two weeks when your host allows it.
  • Include the user agent field. If your log format omits it, no crawler can be identified. A message saying no crawler was found often means this, not that bots stayed away.
  • Unzip compressed logs. A .gz or .zip file will not be read. Extract it first.
  • Check the recognized format. The tool shows which format it detected, such as Apache or Nginx, JSON lines or IIS (W3C extended). If a line cannot be parsed, it is still searched for a crawler name and counted by name only, so details like page and status may be missing.
  • Compare against search engines. Leave the search engine option on to see how AI bot volume compares with Googlebot and bingbot on the same site.
  • Keep query strings only when needed. Turn the option on if parameters create distinct content, such as filtered category pages. Leave it off to avoid splitting one page into many rows.

Once you know who is visiting, act on it. Use an AI robots.txt generator to allow or block specific bots, then verify the rules with an AI crawler access checker for any page. If you also want to track visits that arrive from AI assistants in analytics, an AI traffic regex generator helps you build filters for ChatGPT, Perplexity and Claude referrals.

Limitations and Accuracy Notes

The analyzer reports what your log says, and that has limits. A user agent can be faked, so the tool counts requests that claim to be each crawler. It does not check IP addresses against the operators' published ranges, so verify IPs separately before you block a range or accuse a company of misbehaving.

Only crawlers the tool recognizes are counted. Names that exist only in robots.txt, such as Google-Extended, never appear in a log, so you will not see them here. A CDN may also answer requests before they reach your origin server, so an origin log can undercount bot traffic. Export logs from the CDN layer if you use one.

Finally, a log shows what bots requested, not whether they used your content. Treat the report as proof of access, not proof of citation. To judge how ready a page is to be quoted, pair it with an AI search readiness check, and browse the AI SEO tools hub for related checks.

Upload your access log or paste a few lines into the tool on this page and you will see which AI crawlers visit, what they read and where your server says no. Start with the sample if you want to see the report first, then run your own log.

Frequently Asked Questions

How do I see which AI bots crawl my site?
Upload your server access log to the AI Bot Log Analyzer and it lists every AI crawler it recognizes. You see requests per crawler, the pages each one read and when it was first and last seen. Export the log from your host, server or CDN first.
What log formats does the AI Bot Log Analyzer accept?
The tool accepts Apache and Nginx access logs, IIS W3C extended logs and JSON lines such as Cloudflare Logpush exports. Files ending in .log, .txt, .json, .jsonl or .ndjson work. Unzip .gz or .zip files first, or paste a few lines instead.
How can I tell if my server is blocking GPTBot or ClaudeBot?
Open the Crawlers section and check the refusal summary for each bot. A crawler with many 403 or 429 responses is being blocked or rate limited. If that was not intended, review your firewall, CDN bot rules and robots.txt.
Why does the tool say no crawler was found in my log?
The message means either no recognized AI crawler visited during that period, or your log does not record user agents. Try a longer date range, then confirm each log line includes the user agent field, usually at the end.
Can the AI Bot Log Analyzer prove a request really came from OpenAI or Anthropic?
No. The tool counts requests that claim to be a crawler by user agent, and a user agent can be faked. It does not check IP addresses against the operators' published ranges, so verify IPs separately before blocking.
Is the AI Bot Log Analyzer free to use?
Yes, the AI Bot Log Analyzer is free and does not require sign-in. You can analyze a log file or pasted lines, load the sample, and download the Crawlers CSV or Pages CSV for your own reports.
Recommended

Explore VPN options

A VPN changes your network exit. It does not guarantee anonymity.

Hide Your IP Encryption Network coverage
Protect My IP Now