A robots.txt tester tells you your syntax parses. A sitemap validator tells you your XML is well-formed. Both can hand you a green tick while your pages sit unindexed, because the failure that hides them is not inside either file — it is the disagreement between them. This tool reads both and checks one against the other.

The two files, and what each one actually does

robots.txt is permission. It lives at the root of your domain and tells crawlers which paths they may fetch. It does not control indexing — a blocked page can still appear in results if other sites link to it, which is whynoindex and robots.txt are different tools for different jobs.

sitemap.xml is an invitation. It lists the URLs you want crawled, ideally with a lastmod date so Google knows what changed. It does not grant access and it does not override robots.txt.

That asymmetry is the whole problem. The sitemap says “please crawl this”; robots.txt says “you may not”. Permission wins, every time, and nothing in either file tells you the two are fighting.

The conflict, concretely

Here is a robots.txt that passes every syntax test:

User-agent: *
Disallow: /blog/

Sitemap: https://example.com/sitemap.xml

And a sitemap that is perfectly valid XML, listing two hundred blog posts under /blog/. Run a robots.txt tester: passes. Run a sitemap validator: passes. Submit the sitemap in Search Console: accepted, and every URL is reported as excluded with a message most people read past.

Two hundred articles, invisible, and both checkers said everything was fine. The checker above catches this by taking each URL out of your sitemap, running it through your robots.txt rules as Googlebot would, and naming the exact Disallow line that blocks it.

The other silent failure: a sitemap Google was never told about

Google looks for your sitemap in two places: whatever you submitted in Search Console, and the Sitemap: line in robots.txt. If you rely only on a one-time manual submission, you are one Search Console mishap away from Google losing track of it — and a sitemap that is never re-read is a sitemap that never reports your new pages.

One line fixes it, and it is group-independent, so it can sit anywhere in the file:

Sitemap: https://example.com/sitemap.xml

The check flags the case where /sitemap.xml exists and serves real URLs but robots.txt never mentions it. That combination is common and it is entirely self-inflicted.

How the rules are actually resolved

Most people assume robots.txt is read top to bottom and the first match wins, or thatDisallow beats Allow. Neither is true. Google resolves by specificity: the longest matching pattern wins, regardless of order.

User-agent: *
Disallow: /blog/
Allow: /blog/keep-this

Here /blog/keep-this is crawlable — the Allow pattern is longer — and everything else under /blog/ is blocked. Swap the two lines and nothing changes. On an exact length tie, Allow wins, which is what makes the common Disallow: /*? plus Allow: /*?$ pair behave the way people expect.

Two wildcards matter: * matches any run of characters, and $ anchors the end of the path. So Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?v=2. Getting this wrong in either direction is how sites either block more than they meant to or believe they blocked something they did not.

One more that surprises people: a group matching your specific user-agent replaces the * group rather than adding to it. If User-agent: * says Disallow: / and a later User-agent: Googlebot group only disallows /admin, Googlebot is free to crawl everything except /admin. It does not inherit the site-wide block.

What this checker reports

• Googlebot access at the root — a site-wide Disallow: / is terminal, so it is graded as such and nothing else can compensate for it.
• Whether robots.txt declares a sitemap, and whether that sitemap is the one that actually exists.
• Whether the sitemap parses — a 200 response that yields no <loc> entries looks identical to a working sitemap from the outside. A sitemap index is expanded one level.
• Every sitemap URL, checked against your robots.txt rules, with the blocking line named. This is the check the other tools skip.
• Cross-host entries — URLs in your sitemap pointing at another domain. Google ignores those unless that host is verified in the same property.

Fixing what it finds

Sitemap URLs blocked by robots.txt

Decide which file is wrong. If those pages should be indexed, narrow the Disallow — often it was written to keep a crawler out of one subdirectory and caught the whole tree. If they should not be indexed, take them out of the sitemap; a sitemap listing pages you do not want crawled sends Google a mixed signal on every read.

No Sitemap: line

Add it. Then confirm in Search Console that the sitemap has actually been downloaded, not merely submitted — those are different states, and a sitemap can sit accepted-but-unread for weeks.

Sitemap returns nothing

Usually one of three things: the URL 404s or redirects, the response is HTML rather than XML (a common outcome when a framework serves the app shell for unknown routes), or the sitemap is generated on request and times out before it finishes. All three look like a healthy sitemap in a browser tab.

Frequently asked

Does robots.txt stop a page from being indexed?

No. It stops crawling. A page blocked in robots.txt can still be indexed from external links, and because Google cannot fetch it, it cannot see a noindex tag either. To keep something out of the index, allow the crawl and serve noindex.

Do I need a sitemap at all?

For a small, well-linked site, no — crawlers will find your pages through links. It matters as you grow, when pages are poorly linked, or when you publish often and want new URLs found quickly rather than eventually.

Where does Google's own robots.txt tester fit?

Google retired the standalone robots.txt tester in Search Console; there is now a robots.txt report showing the file as fetched. It is useful for confirming what Google has, and it does not cross-check your sitemap against it.

How often is the result cached?

A domain's result is cached for six hours. Ship a fix and the check will still show the old state for a while — the files themselves update immediately, so verify a change by loading /robots.txt directly.