Crawl

Robots.txt & Sitemap Auditor

Just paste a domain to check robots.txt and crawl the linked sitemaps (root/blog/blacklist/forum variants). Shows blocked URLs, orphaned sitemaps, and malformed entries. Client-side only lah—can be limited by CORS.

Audit

Site URL (e.g., https://example.com)

What we check

• robots.txt availability and Disallow/Allow rules.
• Sitemap discovery (robots.txt + common paths like /sitemap.xml, /blog/sitemap.xml).
• Count of URLs per sitemap and malformed entries.
• Flags blocked URLs that also show up in sitemaps.