Robots.txt Rule Tester: Check Any URL Free

A robots.txt rule tester checks whether a crawler may fetch a specific URL on your site. Enter a domain, a URL path and a user-agent, and this free tool reads the live robots.txt file. It then tells you whether the path is Allowed or Blocked and which rule made the call.

What Is a Robots.txt File and How Does It Work?

A robots.txt file is a plain text file at the root of a domain that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardized as RFC 9309, and well-behaved bots read it before crawling your pages. Think of it as house rules posted at the front door.

The file is split into groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler picks the group that names it most specifically and falls back to the * group when none does. It then follows only that group, so a Googlebot group does not inherit rules from the * group.

When several rules match a URL, the most specific one (the longest matching path) wins, according to Google's Search Central documentation. Paths are case-sensitive, and Google supports the * wildcard and the $ end-of-URL marker. Robots.txt controls crawling, not indexing, so a blocked URL can still show up in results if other pages link to it.

How to Use the Robots.txt Rule Tester

You can test any public URL path in under a minute by following these steps.

1. Type your site into Domain Name, for example example.com. If you paste a full address such as https://example.com/page, the tool keeps only the host name.
2. Enter the page you want to check in URL Path to Test, such as /blog/my-post. Leave the default / to test the homepage. A missing leading slash is added for you.
3. Open the User-Agent menu and pick a crawler. Options are grouped into Search Engines (Googlebot, Bingbot and others), AI Crawlers (GPTBot, ClaudeBot, PerplexityBot and others) and Wildcard (All Bots).
4. Click Test Rule. The button changes to Testing... while the tool loads your robots.txt file.
5. Read the verdict, the matched rule, the declared crawl delay, any sitemaps and the raw file.
6. Change the path or user-agent and run it again to compare how different bots are treated.

Worked Example: Testing Four Paths Against One File

Say example.com publishes a robots.txt file with a single * group containing these lines:

• Disallow: /blog/
• Allow: /blog/guides/
• Disallow: /cart

Here is what the tester returns for Googlebot on four paths, and why:

• /blog/my-post is Blocked. The Disallow rule for /blog/ is the only match.
• /blog/guides/robots-txt is Allowed. Both the Disallow and Allow rules match, but /blog/guides/ is 13 characters and /blog/ is 6, so the longer Allow rule wins.
• /cart-tips is Blocked. Rules are prefix matches, so /cart also catches /cart-tips, /cart/ and /cartridge.
• /about is Allowed with no matching rule. When nothing matches, crawling is allowed by default.

The /cart-tips result is the one that surprises people. If you only meant to block the cart folder, change the rule to /cart/ and test again.

How Do You Read the Results?

The result panel gives you a verdict, the reason behind it and some context from the same file.

Allowed or Blocked

This is the answer for the exact path and user-agent you chose. A Blocked result on a page you want in search is a problem to fix. A Blocked result on a login page, cart or internal search page is usually intentional.

Matched rule

The tool shows the rule that decided the outcome and labels it as allow or disallow. If you see the line saying no matching rule was found, the path is allowed by default and there is nothing to change.

Declared crawl delay

This shows a Crawl-delay value in seconds if the group for your chosen bot has one, or None. Crawler support varies. Google's documentation says Googlebot ignores the directive, while some other crawlers honor it.

Sitemaps Declared and Raw robots.txt

Any Sitemap lines found in the file are listed so you can confirm they point to the right addresses. The raw file is printed below so you can spot typos, stray wildcards or a leftover Disallow: / from a staging site.

Missing or unavailable files

If the file returns a client error such as 404, the tool reports crawling as allowed, because RFC 9309 treats a missing robots.txt as no restrictions. If the file cannot be retrieved because of a server error, a timeout or a rate-limit response, the tool says crawl permission is unverified. Fix that quickly, because Google's documentation notes that server errors on robots.txt can hold back crawling of the whole site.

When Is This Tool the Right Choice?

Use it any time you need a fast, rule-level answer about one URL and one bot.

• After a site launch or migration: confirm key templates such as product pages and blog posts are not blocked by a staging rule.
• When pages will not index: rule out robots.txt before you dig into other causes. A Google index checker can then show how much of the site is actually indexed.
• Managing AI crawlers: check whether GPTBot, ClaudeBot, PerplexityBot or CCBot can reach a section you want to protect or share.
• Client audits: test a competitor-free sample of important paths and attach the matched rule to your report.
• Writing new rules: after you draft a file with the robots.txt generator and publish it, verify that each rule behaves as intended.

Best Practices and Common Mistakes

Most robots.txt problems come from a handful of repeat mistakes, and each has a simple fix.

• Test more than one bot. A page can be Allowed for the * group and Blocked for a bot with its own group. Check Googlebot, Bingbot and the wildcard at least.
• Watch prefix matching. Disallow: /blog blocks /blog-news as well. Add the trailing slash when you mean a folder.
• Mind capitalization. /Blog/ and /blog/ are different paths. Test the exact casing your URLs use.
• Do not block CSS and JavaScript that pages need to render. Google's guidance is to let Googlebot fetch them.
• Do not use Disallow to remove a page from search. A blocked page cannot show its noindex tag to the crawler. Allow crawling and use noindex instead.
• Keep the file at the root. Only the file at the top of the host applies, and each subdomain needs its own.

If a result looks wrong, open the raw file in the results and look for a more specific group that names your bot. Also check the file's address with an HTTP header checker to confirm it returns a 200 status. If your homepage loads in the result instead of plain text, the file is probably being redirected to an HTML page.

Limitations to Keep in Mind

This tester reads the robots.txt file that is live on your domain right now. It cannot test a draft you have not published, so upload changes first, then run the check. It also shows how the file's rules apply to a path, not whether a bot will really obey them. Reputable crawlers follow robots.txt, but the file is a request and not a lock, so never rely on it to hide private content.

Google-Extended is listed in the dropdown so you can see the rules written for it, but it is a control token rather than a separate crawler. For a second view of the same file, try the robots.txt tester. To check many addresses for broken pages after changing rules, run them through the bulk HTTP status checker.

Knowing exactly which rule blocks or allows a URL saves hours of guesswork. Enter your domain and a path above, pick a crawler and click Test Rule. For more checks like this, browse the website audit and performance tools.