AI Search / GEO
AI Crawler & llms.txt Checker
See which AI crawlers can read your page, and why. Most sites block the AI engines that cite them, but leave the ones that train on them wide open — this one catches that.
Checks the page you key in, plus that origin robots.txt, llms.txt and llms-full.txt.
What this tool checks
Search is not only Google already. ChatGPT, Claude, Perplexity, Gemini and Copilot answer questions by fetching pages, and each one uses a different crawler with its own name. One line in robots.txt decides whether you show up in those answers.
The four kinds of AI crawler (and why the difference matters)
Blocking is not one decision only. Treating every AI bot the same is what causes the most common mistake on the web right now: sites block the crawlers that would have cited them and sent traffic back, while the crawlers that quietly train on their content still can come in.
Training crawlers
They add your pages to a model training corpus. Blocking them protects your content and costs you zero traffic. Examples: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended.
AI search crawlers
They build the index an AI answer engine cites from. Blocking them takes you out of those answers — this is the one that really costs you traffic. Examples: OAI-SearchBot, PerplexityBot, Applebot, DuckAssistBot.
User-triggered fetches
A real person asked an assistant about your page, so it goes and reads it. Block these, and the assistant will tell your own visitor it cannot open your link. Examples: ChatGPT-User, Claude-User, Perplexity-User.
Dataset crawlers
Public or commercial crawl corpora that many models take in later. Examples: CCBot (Common Crawl), Diffbot, AI2Bot, Webzio-Extended.
What is llms.txt?
llms.txt is a proposed standard (llmstxt.org): a Markdown file at the root of your domain that gives an AI assistant a clean, curated map of your site — an H1 with your name, a one-line summary in a blockquote, then sections of links. It is not a ranking factor and no engine must read it, but it is cheap to publish and it is how you control the wording an assistant uses to describe you. This tool checks the structure and generates a starter file from your own sitemap.
Frequently asked questions
Blocking GPTBot means I disappear from ChatGPT or not?
No — and this is the most common misunderstanding. GPTBot only feeds model training. The live ChatGPT answers use OAI-SearchBot (the search index) and ChatGPT-User (when somebody opens your link inside a chat). You can block GPTBot and still get cited by ChatGPT, so long as those two stay allowed.
Does Google-Extended affect my Google Search ranking?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Normal Search crawling and ranking go by Googlebot, which is a separate token. Blocking Google-Extended does not take you out of Google Search.
AI crawlers really follow robots.txt one?
The named ones from the big operators publish their tokens and are documented as obeying it. But robots.txt is voluntary — it is a request, not a wall. Crawlers that ignore it, or that do not identify themselves, need a firewall or WAF rule instead.
Why is my page score low even though nothing is blocked?
Access is only half the story. AI crawlers do not run JavaScript, so if your text only appears after JS runs, they see an empty page. The score also rewards server-rendered text, JSON-LD structured data, one clear H1, and a title plus description.
Is llms.txt compulsory?
No. It is a proposal, not a standard any engine enforces, and no major engine has committed to reading it. It costs one file to publish and it is the only place you get to write the summary an assistant may reuse word for word, so the downside is quite small.