AI crawler access
See which search engines, AI training crawlers and AI assistants a site's robots.txt allows, and whether it has llms.txt.
GET /v1/ai-crawlers
- Price
- Costs 1 credit per call; cached answers cost the same. Calls that fail upstream (502) are free.
- Cache
- Answers are cached for up to 15 minutes, per account except for a site's home page; send fresh=true to ask the source again.
- MCP tool
- check_ai_crawlers
Query parameters
| Name | Type | Description |
|---|---|---|
| url (required) | string | A page URL or a domain; a domain means its home page. |
| fresh | boolean | Skip the cache and ask the source again; costs the same. |
Response
| Name | Type | Description |
|---|---|---|
| tool (required) | string | |
| target (required) | string | |
| status (required) | "ok" | "partial" | "error" | |
| fetched_at | string | |
| cached | boolean | |
| credits_used | integer | Credits this call took from your balance. 0 for free tools, for a replay of an earlier call, and for a free check run on the website without signing in |
| summary (required) | string | |
| score | integer or null | |
| findings | array of object | |
| credits_remaining (required) | integer | Your balance after this call. It can be below zero after a refund; free tools still work then |
| data (required) | object | This tool's result |
| data.url (required) | string | |
| data.robots_url (required) | string | |
| data.robots_status (required) | "ok" | "missing" | "server_error" | "unreachable" | |
| data.robots_http_status (required) | integer or null | |
| data.sitemaps (required) | array of string | |
| data.invalid_lines (required) | array of integer | |
| data.crawlers (required) | array of object | |
| data.crawlers.token (required) | string | |
| data.crawlers.vendor (required) | string | |
| data.crawlers.purpose (required) | "search" | "training" | "retrieval" | |
| data.crawlers.user_triggered (required) | boolean | |
| data.crawlers.allowed (required) | boolean | |
| data.crawlers.reason (required) | "rule" | "no_matching_rule" | "no_group" | "missing" | "server_error" | "unreachable" | |
| data.crawlers.rule (required) | string or null | |
| data.crawlers.rule_line (required) | integer or null | |
| data.crawlers.group_agents (required) | array of string or null | |
| data.groups (required) | object | |
| data.content_signals (required) | array of object | |
| data.page_directives (required) | array of string | |
| data.llms_txt (required) | object | |
| data.llms_txt.url (required) | string | |
| data.llms_txt.present (required) | boolean | |
| data.llms_txt.status (required) | integer or null | |
| data.llms_txt.starts_with_h1 (required) | boolean | |
| data.llms_txt.bytes (required) | integer or null | |
| data.catalog_checked (required) | string |
Errors
- 400 Invalid domain, URL or parameter, a TLD this tool can't check, or an X-Request-Id already used for another call (`invalid_domain`, `invalid_url`, `invalid_request`, `unsupported_tld`)
- 401 Missing, unknown or revoked API key, or one whose creator is no longer a member of the account (`unauthorized`)
- 402 Balance below the call's price (`insufficient_credits`)
- 403 Account suspended, the key's creator is over the plan's seats, or an anonymous check without a valid proof-of-work or for a tool that needs an account, or bulk checks before a first purchase (`account_suspended`, `plan_limit`, `challenge_required`, `sign_in_required`, `bulk_locked`)
- 429 Too many requests, today's free checks are used up, or too many website scans at once; wait for the number of seconds in the Retry-After header (`rate_limited`, `free_limit_reached`, `too_many_jobs`). Not charged
- 500 Unexpected error (`internal_error`)
- 502 Every upstream source failed; not charged (`source_unavailable`). A Retry-After header says how long to wait when the source asked us to
- 503 Too many open event streams on this server, the checked site is getting too many checks, or the browser or link checker is busy; try again after the seconds in the Retry-After header (`busy`, `site_busy`, `browser_busy`, `crawl_busy`). Not charged
Example
curl
curl "https://api.dnssuper.com/v1/ai-crawlers?url=example.com" \
-H "Authorization: Bearer $DNSSUPER_API_KEY"Python
import os
import httpx
client = httpx.Client(
base_url="https://api.dnssuper.com",
headers={"Authorization": f"Bearer {os.environ['DNSSUPER_API_KEY']}"},
timeout=60,
)
response = client.get("/v1/ai-crawlers", params={
"url": "example.com",
})
response.raise_for_status()
result = response.json()
print(result["summary"], result["credits_used"])TypeScript
const response = await fetch(`https://api.dnssuper.com/v1/ai-crawlers?${new URLSearchParams({"url":"example.com"})}`, {
headers: { Authorization: `Bearer ${process.env.DNSSUPER_API_KEY}` },
});
if (!response.ok) throw new Error((await response.json()).error.message);
const result = await response.json();
console.log(result.summary, result.credits_used);Example response
{
"tool": "ai-crawler-checker",
"target": "https://example.com/blog/post",
"status": "ok",
"fetched_at": "2026-11-14T10:00:00Z",
"cached": false,
"credits_used": 1,
"summary": "Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.",
"findings": [
{
"id": "ai_crawlers.robots_invalid",
"severity": "warn",
"title": "robots.txt has problems",
"detail": "Lines 16 aren't valid.",
"fix": "Fix the listed lines, serve it as text/plain and keep it under 500 KB."
},
{
"id": "ai_crawlers.search_blocked",
"severity": "warn",
"title": "Search engine crawlers are blocked",
"detail": "Blocked: Bingbot.",
"fix": "Allow search crawlers unless you want out of search results."
},
{
"id": "ai_crawlers.page_noindex",
"severity": "warn",
"title": "The page asks search engines not to index it",
"detail": "",
"fix": "Remove noindex if the page should appear in search."
},
{
"id": "ai_crawlers.training_blocked",
"severity": "info",
"title": "AI training crawlers are blocked",
"detail": "Blocked: GPTBot, CCBot.",
"fix": "Nothing to do if that's your choice."
},
{
"id": "ai_crawlers.retrieval_blocked",
"severity": "info",
"title": "AI assistant fetchers are blocked",
"detail": "Blocked: PerplexityBot.",
"fix": "Nothing to do if that's your choice."
},
{
"id": "ai_crawlers.noai_directive",
"severity": "info",
"title": "The page asks not to be used for AI",
"detail": "noai, GPTBot: noimageai.",
"fix": "Nothing to do; not every crawler honours noai."
},
{
"id": "llms_txt.missing",
"severity": "info",
"title": "No llms.txt",
"detail": "",
"fix": "Nothing to do; llms.txt is an optional proposal."
}
],
"data": {
"url": "https://example.com/blog/post",
"robots_url": "https://example.com/robots.txt",
"robots_status": "ok",
"robots_http_status": 200,
"sitemaps": [
"https://example.com/sitemap.xml"
],
"invalid_lines": [
16
],
"crawlers": [
{
"token": "Googlebot",
"vendor": "Google",
"purpose": "search",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Bingbot",
"vendor": "Microsoft",
"purpose": "search",
"user_triggered": false,
"allowed": false,
"reason": "rule",
"rule": "Disallow: /",
"rule_line": 7,
"group_agents": [
"Bingbot"
]
},
{
"token": "Applebot",
"vendor": "Apple",
"purpose": "search",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "DuckDuckBot",
"vendor": "DuckDuckGo",
"purpose": "search",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Google-Extended",
"vendor": "Google",
"purpose": "training",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "GPTBot",
"vendor": "OpenAI",
"purpose": "training",
"user_triggered": false,
"allowed": false,
"reason": "rule",
"rule": "Disallow: /",
"rule_line": 4,
"group_agents": [
"GPTBot",
"CCBot"
]
},
{
"token": "ClaudeBot",
"vendor": "Anthropic",
"purpose": "training",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "CCBot",
"vendor": "Common Crawl",
"purpose": "training",
"user_triggered": false,
"allowed": false,
"reason": "rule",
"rule": "Disallow: /",
"rule_line": 4,
"group_agents": [
"GPTBot",
"CCBot"
]
},
{
"token": "Applebot-Extended",
"vendor": "Apple",
"purpose": "training",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Meta-ExternalAgent",
"vendor": "Meta",
"purpose": "training",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Amazonbot",
"vendor": "Amazon",
"purpose": "training",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "OAI-SearchBot",
"vendor": "OpenAI",
"purpose": "retrieval",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 13,
"group_agents": [
"OAI-SearchBot"
]
},
{
"token": "ChatGPT-User",
"vendor": "OpenAI",
"purpose": "retrieval",
"user_triggered": true,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Claude-SearchBot",
"vendor": "Anthropic",
"purpose": "retrieval",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Claude-User",
"vendor": "Anthropic",
"purpose": "retrieval",
"user_triggered": true,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "PerplexityBot",
"vendor": "Perplexity",
"purpose": "retrieval",
"user_triggered": false,
"allowed": false,
"reason": "rule",
"rule": "Disallow: /blog/",
"rule_line": 10,
"group_agents": [
"PerplexityBot"
]
},
{
"token": "Perplexity-User",
"vendor": "Perplexity",
"purpose": "retrieval",
"user_triggered": true,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "Meta-ExternalFetcher",
"vendor": "Meta",
"purpose": "retrieval",
"user_triggered": true,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "DuckAssistBot",
"vendor": "DuckDuckGo",
"purpose": "retrieval",
"user_triggered": false,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
},
{
"token": "MistralAI-User",
"vendor": "Mistral AI",
"purpose": "retrieval",
"user_triggered": true,
"allowed": true,
"reason": "rule",
"rule": "Allow: /",
"rule_line": 18,
"group_agents": [
"*"
]
}
],
"groups": {
"search": "mixed",
"training": "mixed",
"retrieval": "mixed"
},
"content_signals": [],
"page_directives": [
"noindex",
"noai",
"GPTBot: noimageai"
],
"llms_txt": {
"url": "https://example.com/llms.txt",
"present": false,
"status": 404,
"starts_with_h1": false
},
"catalog_checked": "2026-11-15"
},
"credits_remaining": 498
}MCP
{
"tool": "check_ai_crawlers",
"arguments": {
"url": "example.com"
}
}Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.
Search: mixed (blocked: Bingbot)
AI training: mixed (blocked: GPTBot, CCBot)
AI assistants: mixed (blocked: PerplexityBot)
robots.txt: found, 1 invalid line
Page directives: noindex, noai, GPTBot: noimageai
llms.txt: none
- warn: robots.txt has problems. Fix: Fix the listed lines, serve it as text/plain and keep it under 500 KB.
- warn: Search engine crawlers are blocked. Fix: Allow search crawlers unless you want out of search results.
- warn: The page asks search engines not to index it. Fix: Remove noindex if the page should appear in search.
- info: AI training crawlers are blocked
+3 more findings (pass detail='full' to see all)
Credits: 1 credit used, 498 remaining | fresh | checked 2026-11-14 10:00 UTC