AI Search / GEO
AI Crawler & llms.txt Checker
See which AI crawlers can read your page, and why. Most sites block the AI engines that cite them while leaving the ones that train on them wide open — this finds that.
Checks the page you enter, plus that origin's robots.txt, llms.txt and llms-full.txt.
What this tool checks
Search is no longer only Google. ChatGPT, Claude, Perplexity, Gemini and Copilot answer questions by fetching pages, and each of them uses a different, separately-named crawler. A single line in robots.txt decides whether you are in those answers.
The four kinds of AI crawler (and why the difference matters)
Blocking is not one decision. Treating every AI bot the same is what causes the most common mistake on the web right now: sites block the crawlers that would have cited them and sent traffic back, while the crawlers that quietly train on their content are still allowed.
Training crawlers
They add your pages to a model's training corpus. Blocking them protects your content and costs you no traffic. Examples: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended.
AI search crawlers
They build the index an AI answer engine cites from. Blocking them removes you from those answers — this is the one that costs real traffic. Examples: OAI-SearchBot, PerplexityBot, Applebot, DuckAssistBot.
User-triggered fetches
A real person asked an assistant about your page and it goes to read it. Blocking these means the assistant tells your own visitor it cannot open your link. Examples: ChatGPT-User, Claude-User, Perplexity-User.
Dataset crawlers
Public or commercial crawl corpora that many models ingest later. Examples: CCBot (Common Crawl), Diffbot, AI2Bot, Webzio-Extended.
What is llms.txt?
llms.txt is a proposed standard (llmstxt.org): a Markdown file at the root of your domain that gives an AI assistant a clean, curated map of your site — an H1 with your name, a one-line summary in a blockquote, then sections of links. It is not a ranking factor and no engine is obliged to read it, but it is cheap to publish and it is how you control the wording an assistant uses to describe you. This tool validates the structure and generates a starter file from your own sitemap.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No — and this is the most common misunderstanding. GPTBot only feeds model training. ChatGPT's live answers use OAI-SearchBot (the search index) and ChatGPT-User (when someone opens your link in a chat). You can block GPTBot and still be cited by ChatGPT, as long as those two stay allowed.
Does Google-Extended affect my Google Search ranking?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Normal Search crawling and ranking are governed by Googlebot, which is a separate token. Blocking Google-Extended does not remove you from Google Search.
Do AI crawlers actually obey robots.txt?
The named ones from major operators publish their tokens and are documented as obeying it. robots.txt is voluntary, though — it is a request, not a wall. Crawlers that ignore it, or that do not identify themselves, need a firewall or WAF rule instead.
Why does my page score low even though nothing is blocked?
Access is only half of it. AI crawlers do not run JavaScript, so if your text only appears after JS runs, they see an empty page. The score also rewards server-rendered text, JSON-LD structured data, a single clear H1, and a title plus description.
Is llms.txt required?
No. It is a proposal, not a standard any engine enforces, and no major engine has committed to reading it. It costs one file to publish and it is the only place you get to write the summary an assistant may reuse verbatim, so the downside is small.