robots.txt looks like a simple list of Disallow lines, but the rule that decides whether a specific path is crawlable follows a precise precedence: the most specific pattern wins, not the first line, not Allow beating Disallow by default. A one-line file is easy to read by eye; a file with several user-agent groups and overlapping patterns is not — this tool runs the actual matching rules instead of asking you to trace them by hand.

The rules this follows

• Most specific pattern wins, measured by length — not by which line comes first in the file. Disallow: /blog/ combined with Allow: /blog/keep leaves everything under /blog/ blocked except that one longer, more specific path.
• On an exact-length tie, Allow wins. This is what makes the common Disallow: /*? + Allow: /*?$ pair work as intended.
• A specific user-agent group excludes the wildcard entirely — if a GPTBot group exists, GPTBot only reads its own group’s rules, never merging with whatever User-agent: * says.
• * matches any run of characters in a pattern; $ anchors the end of the path.

Why test by user-agent

Different crawlers are commonly given different rules on purpose — blocking AI training crawlers (GPTBot, ClaudeBot) from a whole site while leaving search crawlers (Googlebot, Bingbot) with normal access is one of the most common patterns in a modern robots.txt. Testing one path against several agents at once shows whether that split is actually working as intended.

What this tool doesn’t check

Whether a path is crawlable and whether it’s indexed are two different questions — a page robots.txt disallows can still show up in search results (without a snippet) if enough links point to it. To check indexability directly, see the Meta Robots & X-Robots-Tag Checker. For a full audit cross-referencing robots.txt against your actual sitemap, see the Robots.txt & Sitemap Checker.

Frequently asked

Does order matter in robots.txt?

Not for which rule wins — specificity does. Order matters only for readability and for which User-agent lines group together (consecutive User-agent lines share one group of rules).

What happens if a domain has no robots.txt at all?

A 404 on /robots.txt means no restrictions exist — every path is crawlable by every agent. This tool reports that explicitly rather than treating the fetch failure as an error.

Does this tool send my robots.txt anywhere?

No — fetching a live domain’s robots.txt makes one request to that domain; parsing and every path/agent test after that run entirely in your browser.