Technical SEO
Robots.txt Rule Tester
Paste a robots.txt file, pick a crawler, and test any path. See exactly which rule wins, and catch mistakes that quietly block search engines or waste crawl budget.
Runs entirely in your browser. Nothing you paste here is sent anywhere except an anonymized usage count.
Paste the exact contents of your robots.txt file.
Path and optional query string only, no domain, e.g. /blog/post-1?ref=x.
How this works, and how to fix common mistakes
Real crawlers pick the single most specific User-agent group that matches their name (an exact or longest-prefix match beats "*"), then, inside that group only, apply the Allow or Disallow rule whose path pattern is the longest match, with Allow winning ties. This tool runs that exact algorithm — it does not just look for the first Disallow it finds.
"*" inside a rule matches any sequence of characters, and a trailing "$" anchors the rule to the end of the path. Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?ref=1 — a common source of unexpectedly-crawlable PDFs.
An empty "Disallow:" line means no restriction at all — it does not block anything. And once a crawler matches a specific group (like "Googlebot"), the rules under "User-agent: *" are ignored entirely for that crawler; they are never merged.
Common mistakes this tool flags: a leftover Disallow: / from a staging site (blocks everything), rules placed before any User-agent line (silently ignored), Crawl-delay (ignored by Google and Bing), and Sitemap URLs that are not absolute.