What the AI crawler check tells you
Search engines, AI companies and AI assistants all send crawlers to read websites. Each one has a short name, its product token, and the robots.txt file at the root of your site tells each of them which pages they may fetch. The rules are easy to get wrong: one line meant for an AI crawler can quietly block a search engine, and a typo means a rule isn't applied at all. This check reads your robots.txt the way the standard says crawlers should, and shows for every crawler on our list whether the page you entered is allowed, and which rule decided it.
robots.txt shows what a site asks for, not what crawlers do. It is a published preference that reputable crawlers say they honour, not a lock.
What we check
robots.txt. We fetch it once and parse it with our own implementation of RFC 9309, the robots.txt
standard: groups and their user-agent lines, Allow and Disallow rules with * and $, the longest-match rule
and Allow winning ties. A missing file allows everything; a server error tells crawlers to crawl nothing; a
file served as a web page or longer than 500 KB is a problem. We list lines crawlers can't read, the sitemaps
the file names and any Content-Signal preferences.
Crawlers by purpose. For each crawler on our list we evaluate the rules for your exact page and group the results three ways:
- Search engines, such as Googlebot, Bingbot, Applebot and DuckDuckBot, which build search results.
- AI training, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta's training crawler and Amazonbot, which collect pages to train models.
- AI assistants and AI search, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and DuckAssistBot, which read pages to answer a question or cite them.
For each one you see allowed or blocked, and the deciding rule with its line number, or why no rule applied.
The page's own directives. A robots meta tag or an X-Robots-Tag header can say noindex, or noai
and noimageai, for every crawler or for one by name. We list what the page says.
llms.txt. Whether the site publishes an llms.txt, a proposed Markdown summary for AI tools, and whether it starts with a heading as the proposal expects.
Reading the result
Each group starts with a verdict: all allowed, some blocked or all blocked. The table under it lists every crawler with its vendor, the verdict for your page and the rule that decided it, such as "Disallow: / (line 4)". Crawlers that fetch only when a person asks are marked. Below the groups you'll find robots.txt itself, the page's directives and llms.txt.
A blocked search engine is a warning, because it usually means a page drops out of search by accident. A blocked AI crawler is information: it's a legitimate choice, and we show it so you can confirm it's the one you meant to make.
How we fetch
We fetch robots.txt, the page and llms.txt as DNSSuperBot, three requests plus any redirects. We fetch exactly the page you ask about, as a browser would, and never crawl further. If robots.txt can't be read, the result is marked partial and we don't guess at crawler verdicts.
Choosing what to allow
Decide per purpose rather than per vendor. To stay in search but keep pages out of AI training, block the
training tokens by name and leave User-agent: * allowing everything. To be cited by AI assistants, keep
their search and fetch tokens allowed. Check again whenever you change robots.txt; the result shows the exact
line each crawler follows. If you publish llms.txt, keep it short and point it at the pages you most want
AI tools to read, such as documentation, pricing and contact details.