Robots.txt Tester: Free Online URL Checker

This robots.txt tester checks whether a URL is allowed or blocked for a crawler such as Googlebot or Bingbot. Enter a site, a path and a user-agent, and the tool reads the live robots.txt file, shows Allowed or Blocked and names the rule behind the answer. It is free, with no sign-in.

What Is a Robots.txt File and What Is It Used For?

A robots.txt file is a plain text file at the root of a host (for example, example.com/robots.txt) that tells crawlers which URL paths they may or may not request. It is part of the Robots Exclusion Protocol, and well-behaved search engine bots read it before crawling.

The file is organized into groups. Each group starts with a User-agent line, followed by Disallow and Allow rules that match the beginning of a URL path. Google's documentation also describes the * wildcard and the $ end-of-URL marker. A Sitemap line can point crawlers to your XML sitemap.

Robots.txt controls crawling, not indexing. Google's Search Central documentation explains that a blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of search, let it be crawled and use a noindex directive instead.

How to Use the Robots.txt Tester

You need three inputs and one click, and the whole check takes a few seconds.

1. Type the domain into Site URL, for example https://example.com. If you leave off the protocol, the tool assumes http.
2. Enter a path or a full address in URL or path to test, such as /admin or https://example.com/shop?page=2. A full URL is reduced to its path and query string, and a missing leading slash is added for you.
3. Choose a User-agent: Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, YandexBot, Baiduspider or * (the generic group for all other bots).
4. Click Test URL.
5. Read the verdict, the matched rule, the tested path and the full robots.txt content shown underneath.

The same steps also answer "how do I check the robots.txt of any website?" The result panel prints the site's current file, so you can read it even when you only came to look.

Worked Example: Testing Three URLs Against One File

Suppose the live file at shop.example.com contains these rules:

• User-agent: *
• Disallow: /cart/
• Disallow: /search
• Allow: /search/help

Here is what you would see for three tests, all with the * user-agent:

• /cart/checkout?step=2 returns Blocked, and the matched rule is Disallow: /cart/. The path begins with /cart/, so the rule applies even with a query string.
• /search/help returns Allowed. Both Disallow: /search and Allow: /search/help match, but Google's documented behavior is that the more specific (longer) rule wins.
• /blog/robots-guide returns Allowed because no rule matches it, and crawlers may fetch anything that is not disallowed.

The second case is the one that surprises people. Testing it shows exactly why a page inside a blocked folder can still be crawlable.

How Do You Read and Act on the Results?

The verdict reads Allowed or Blocked for the user-agent you picked, with the matched rule and the tested path beneath it. Use the rule to decide your next edit. Some results come from the robots.txt response itself rather than a rule:

If the server returns an HTML page instead of a text file, the tool reports that permission is unverified. That usually means a redirect to a home page, a login wall or a firewall page. Make sure the plain text file is served at the root path. If the tool cannot fetch the file at all, check that the domain is spelled correctly and the site is online.

Who Needs a Robots.txt Tester?

Anyone who edits, inherits or audits a robots.txt file benefits from checking real URLs against it. The most common situations are:

• SEO specialists confirming that important templates, such as product, category and blog URLs, are crawlable before or after a migration.
• Developers checking that a staging rule like Disallow: / did not reach the live site.
• Store owners verifying that cart, checkout and internal search paths are blocked while product pages stay open. Shopify, WordPress and Wix sites each generate default rules, so a quick test shows what your platform actually serves.
• Content teams finding out why an image is missing from Google Images by testing a path with Googlebot-Image.
• Site owners comparing bots by running the same path for Googlebot, Bingbot and * to see which group applies.

Best Practices, Common Mistakes and Troubleshooting

Why is my page allowed when I expected it to be blocked?

Usually a longer Allow rule overrides your Disallow, or the bot you selected follows a different group. A crawler obeys the most specific User-agent group that names it, and ignores the * group when it finds one. Test again with * and with the named bot, and compare the matched rules.

Why is a page blocked that I never meant to block?

Rules match from the start of the path, so Disallow: /pr also blocks /products/ and /press/. Add the trailing slash or use a more exact path. Paths are also case-sensitive, so /Admin and /admin are different.

Other habits worth building

• Do not block CSS or JavaScript files that pages need to render. Google recommends letting Googlebot fetch them.
• Remember that every subdomain and protocol has its own robots.txt. Testing example.com says nothing about blog.example.com.
• Do not rely on robots.txt to hide private content. The file is public, and anyone can read it.
• Retest your key URLs after every change. If you keep old versions, a code diff tool shows exactly which lines changed.
• Starting from scratch? The Robots.txt Generator builds a clean file you can then check here.

Limitations to Keep in Mind

This tester checks one path and one user-agent at a time, and it reads the file that is live on the site right now. If you have unpublished edits, test those drafts with the Robots.txt Rule Tester first, then upload the file and verify it here.

The dropdown covers seven common choices, so a niche crawler needs a manual check against the file shown in the result. The tool also says nothing about meta robots tags, X-Robots-Tag headers or whether a page is indexed. An Allowed verdict means crawling is permitted, not that the page will rank.

If your fix involves server-level changes such as moving URLs, the .htaccess redirect generator can help, and the Web Developer Tools category lists more utilities for site owners.

A few seconds with this robots.txt tester can catch the one stray rule that keeps a key page from being crawled. Enter your site, test your most important URLs for Googlebot and Bingbot, and fix any Blocked result you did not expect.