AI crawler access

See which search engines, AI training crawlers and AI assistants a site's robots.txt allows, and whether it has llms.txt.

A page URL or a domain; a domain means its home page.

A few free checks a day without an account. Sign in to use your credits.

Check many at once

Bulk checks aren't unlocked yet

Bulk checks unlock after your first credit purchase or plan. Single checks work as usual.

Sign up free

What the AI crawler check tells you

Search engines, AI companies and AI assistants all send crawlers to read websites. Each one has a short name, its product token, and the robots.txt file at the root of your site tells each of them which pages they may fetch. The rules are easy to get wrong: one line meant for an AI crawler can quietly block a search engine, and a typo means a rule isn't applied at all. This check reads your robots.txt the way the standard says crawlers should, and shows for every crawler on our list whether the page you entered is allowed, and which rule decided it.

robots.txt shows what a site asks for, not what crawlers do. It is a published preference that reputable crawlers say they honour, not a lock.

What we check

robots.txt. We fetch it once and parse it with our own implementation of RFC 9309, the robots.txt standard: groups and their user-agent lines, Allow and Disallow rules with * and $, the longest-match rule and Allow winning ties. A missing file allows everything; a server error tells crawlers to crawl nothing; a file served as a web page or longer than 500 KB is a problem. We list lines crawlers can't read, the sitemaps the file names and any Content-Signal preferences.

Crawlers by purpose. For each crawler on our list we evaluate the rules for your exact page and group the results three ways:

  • Search engines, such as Googlebot, Bingbot, Applebot and DuckDuckBot, which build search results.
  • AI training, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta's training crawler and Amazonbot, which collect pages to train models.
  • AI assistants and AI search, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and DuckAssistBot, which read pages to answer a question or cite them.

For each one you see allowed or blocked, and the deciding rule with its line number, or why no rule applied.

The page's own directives. A robots meta tag or an X-Robots-Tag header can say noindex, or noai and noimageai, for every crawler or for one by name. We list what the page says.

llms.txt. Whether the site publishes an llms.txt, a proposed Markdown summary for AI tools, and whether it starts with a heading as the proposal expects.

Reading the result

Each group starts with a verdict: all allowed, some blocked or all blocked. The table under it lists every crawler with its vendor, the verdict for your page and the rule that decided it, such as "Disallow: / (line 4)". Crawlers that fetch only when a person asks are marked. Below the groups you'll find robots.txt itself, the page's directives and llms.txt.

A blocked search engine is a warning, because it usually means a page drops out of search by accident. A blocked AI crawler is information: it's a legitimate choice, and we show it so you can confirm it's the one you meant to make.

How we fetch

We fetch robots.txt, the page and llms.txt as DNSSuperBot, three requests plus any redirects. We fetch exactly the page you ask about, as a browser would, and never crawl further. If robots.txt can't be read, the result is marked partial and we don't guess at crawler verdicts.

Choosing what to allow

Decide per purpose rather than per vendor. To stay in search but keep pages out of AI training, block the training tokens by name and leave User-agent: * allowing everything. To be cited by AI assistants, keep their search and fetch tokens allowed. Check again whenever you change robots.txt; the result shows the exact line each crawler follows. If you publish llms.txt, keep it short and point it at the pages you most want AI tools to read, such as documentation, pricing and contact details.

Questions

Does robots.txt stop every AI crawler?
No. robots.txt shows what a site asks for, not what crawlers do. Well-behaved crawlers from the vendors we list say they follow it, but it can't enforce anything. Some vendors say fetches a person asks for directly may not follow it; we mark those crawlers in the result.
If I block AI training crawlers, do I disappear from search?
Not if you block the right tokens. Google-Extended and Applebot-Extended control only whether Google and Apple may use pages for AI; Googlebot and Applebot still index for search. The check groups crawlers by purpose so you can see the difference at a glance.
Why does the page address matter?
robots.txt rules match paths, so a crawler can be allowed on your home page and blocked on /blog/. We evaluate every rule for the exact page you enter, including its query string. Enter just the domain to check the home page.
Where does the crawler list come from?
We write it ourselves from each vendor's own documentation and record when every entry was last checked. We leave out crawlers whose vendor doesn't document a robots.txt token, and we review the list every month because new crawlers appear often.
How much does a check cost?
In the browser you get a few free checks a day without an account. Through the API or MCP a check costs 1 credit, and a cached answer costs the same. Answers are cached for 15 minutes. Checks that fail on our side are free.