Docs menu

AI crawler access

See which search engines, AI training crawlers and AI assistants a site's robots.txt allows, and whether it has llms.txt.

GET /v1/ai-crawlers

Price
Costs 1 credit per call; cached answers cost the same. Calls that fail upstream (502) are free.
Cache
Answers are cached for up to 15 minutes, per account except for a site's home page; send fresh=true to ask the source again.
MCP tool
check_ai_crawlers

Run it in the browser

Query parameters

Query parameters
NameTypeDescription
url (required)stringA page URL or a domain; a domain means its home page.
freshbooleanSkip the cache and ask the source again; costs the same.

Response

Response fields
NameTypeDescription
tool (required)string
target (required)string
status (required)"ok" | "partial" | "error"
fetched_atstring
cachedboolean
credits_usedintegerCredits this call took from your balance. 0 for free tools, for a replay of an earlier call, and for a free check run on the website without signing in
summary (required)string
scoreinteger or null
findingsarray of object
credits_remaining (required)integerYour balance after this call. It can be below zero after a refund; free tools still work then
data (required)objectThis tool's result
data.url (required)string
data.robots_url (required)string
data.robots_status (required)"ok" | "missing" | "server_error" | "unreachable"
data.robots_http_status (required)integer or null
data.sitemaps (required)array of string
data.invalid_lines (required)array of integer
data.crawlers (required)array of object
data.crawlers.token (required)string
data.crawlers.vendor (required)string
data.crawlers.purpose (required)"search" | "training" | "retrieval"
data.crawlers.user_triggered (required)boolean
data.crawlers.allowed (required)boolean
data.crawlers.reason (required)"rule" | "no_matching_rule" | "no_group" | "missing" | "server_error" | "unreachable"
data.crawlers.rule (required)string or null
data.crawlers.rule_line (required)integer or null
data.crawlers.group_agents (required)array of string or null
data.groups (required)object
data.content_signals (required)array of object
data.page_directives (required)array of string
data.llms_txt (required)object
data.llms_txt.url (required)string
data.llms_txt.present (required)boolean
data.llms_txt.status (required)integer or null
data.llms_txt.starts_with_h1 (required)boolean
data.llms_txt.bytes (required)integer or null
data.catalog_checked (required)string

Errors

  • 400 Invalid domain, URL or parameter, a TLD this tool can't check, or an X-Request-Id already used for another call (`invalid_domain`, `invalid_url`, `invalid_request`, `unsupported_tld`)
  • 401 Missing, unknown or revoked API key, or one whose creator is no longer a member of the account (`unauthorized`)
  • 402 Balance below the call's price (`insufficient_credits`)
  • 403 Account suspended, the key's creator is over the plan's seats, or an anonymous check without a valid proof-of-work or for a tool that needs an account, or bulk checks before a first purchase (`account_suspended`, `plan_limit`, `challenge_required`, `sign_in_required`, `bulk_locked`)
  • 429 Too many requests, today's free checks are used up, or too many website scans at once; wait for the number of seconds in the Retry-After header (`rate_limited`, `free_limit_reached`, `too_many_jobs`). Not charged
  • 500 Unexpected error (`internal_error`)
  • 502 Every upstream source failed; not charged (`source_unavailable`). A Retry-After header says how long to wait when the source asked us to
  • 503 Too many open event streams on this server, the checked site is getting too many checks, or the browser or link checker is busy; try again after the seconds in the Retry-After header (`busy`, `site_busy`, `browser_busy`, `crawl_busy`). Not charged

Example

curl

curl
curl "https://api.dnssuper.com/v1/ai-crawlers?url=example.com" \
  -H "Authorization: Bearer $DNSSUPER_API_KEY"

Python

Python (httpx)
import os

import httpx

client = httpx.Client(
    base_url="https://api.dnssuper.com",
    headers={"Authorization": f"Bearer {os.environ['DNSSUPER_API_KEY']}"},
    timeout=60,
)
response = client.get("/v1/ai-crawlers", params={
    "url": "example.com",
})
response.raise_for_status()
result = response.json()
print(result["summary"], result["credits_used"])

TypeScript

TypeScript (fetch)
const response = await fetch(`https://api.dnssuper.com/v1/ai-crawlers?${new URLSearchParams({"url":"example.com"})}`, {
  headers: { Authorization: `Bearer ${process.env.DNSSUPER_API_KEY}` },
});
if (!response.ok) throw new Error((await response.json()).error.message);
const result = await response.json();
console.log(result.summary, result.credits_used);
Example response
Example response
{
  "tool": "ai-crawler-checker",
  "target": "https://example.com/blog/post",
  "status": "ok",
  "fetched_at": "2026-11-14T10:00:00Z",
  "cached": false,
  "credits_used": 1,
  "summary": "Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.",
  "findings": [
    {
      "id": "ai_crawlers.robots_invalid",
      "severity": "warn",
      "title": "robots.txt has problems",
      "detail": "Lines 16 aren't valid.",
      "fix": "Fix the listed lines, serve it as text/plain and keep it under 500 KB."
    },
    {
      "id": "ai_crawlers.search_blocked",
      "severity": "warn",
      "title": "Search engine crawlers are blocked",
      "detail": "Blocked: Bingbot.",
      "fix": "Allow search crawlers unless you want out of search results."
    },
    {
      "id": "ai_crawlers.page_noindex",
      "severity": "warn",
      "title": "The page asks search engines not to index it",
      "detail": "",
      "fix": "Remove noindex if the page should appear in search."
    },
    {
      "id": "ai_crawlers.training_blocked",
      "severity": "info",
      "title": "AI training crawlers are blocked",
      "detail": "Blocked: GPTBot, CCBot.",
      "fix": "Nothing to do if that's your choice."
    },
    {
      "id": "ai_crawlers.retrieval_blocked",
      "severity": "info",
      "title": "AI assistant fetchers are blocked",
      "detail": "Blocked: PerplexityBot.",
      "fix": "Nothing to do if that's your choice."
    },
    {
      "id": "ai_crawlers.noai_directive",
      "severity": "info",
      "title": "The page asks not to be used for AI",
      "detail": "noai, GPTBot: noimageai.",
      "fix": "Nothing to do; not every crawler honours noai."
    },
    {
      "id": "llms_txt.missing",
      "severity": "info",
      "title": "No llms.txt",
      "detail": "",
      "fix": "Nothing to do; llms.txt is an optional proposal."
    }
  ],
  "data": {
    "url": "https://example.com/blog/post",
    "robots_url": "https://example.com/robots.txt",
    "robots_status": "ok",
    "robots_http_status": 200,
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ],
    "invalid_lines": [
      16
    ],
    "crawlers": [
      {
        "token": "Googlebot",
        "vendor": "Google",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Bingbot",
        "vendor": "Microsoft",
        "purpose": "search",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 7,
        "group_agents": [
          "Bingbot"
        ]
      },
      {
        "token": "Applebot",
        "vendor": "Apple",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "DuckDuckBot",
        "vendor": "DuckDuckGo",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Google-Extended",
        "vendor": "Google",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "GPTBot",
        "vendor": "OpenAI",
        "purpose": "training",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 4,
        "group_agents": [
          "GPTBot",
          "CCBot"
        ]
      },
      {
        "token": "ClaudeBot",
        "vendor": "Anthropic",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "CCBot",
        "vendor": "Common Crawl",
        "purpose": "training",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 4,
        "group_agents": [
          "GPTBot",
          "CCBot"
        ]
      },
      {
        "token": "Applebot-Extended",
        "vendor": "Apple",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Meta-ExternalAgent",
        "vendor": "Meta",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Amazonbot",
        "vendor": "Amazon",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "OAI-SearchBot",
        "vendor": "OpenAI",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 13,
        "group_agents": [
          "OAI-SearchBot"
        ]
      },
      {
        "token": "ChatGPT-User",
        "vendor": "OpenAI",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Claude-SearchBot",
        "vendor": "Anthropic",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Claude-User",
        "vendor": "Anthropic",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "PerplexityBot",
        "vendor": "Perplexity",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /blog/",
        "rule_line": 10,
        "group_agents": [
          "PerplexityBot"
        ]
      },
      {
        "token": "Perplexity-User",
        "vendor": "Perplexity",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Meta-ExternalFetcher",
        "vendor": "Meta",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "DuckAssistBot",
        "vendor": "DuckDuckGo",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "MistralAI-User",
        "vendor": "Mistral AI",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      }
    ],
    "groups": {
      "search": "mixed",
      "training": "mixed",
      "retrieval": "mixed"
    },
    "content_signals": [],
    "page_directives": [
      "noindex",
      "noai",
      "GPTBot: noimageai"
    ],
    "llms_txt": {
      "url": "https://example.com/llms.txt",
      "present": false,
      "status": 404,
      "starts_with_h1": false
    },
    "catalog_checked": "2026-11-15"
  },
  "credits_remaining": 498
}

MCP

MCP tool call
{
  "tool": "check_ai_crawlers",
  "arguments": {
    "url": "example.com"
  }
}
Compact MCP output
Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.
Search: mixed (blocked: Bingbot)
AI training: mixed (blocked: GPTBot, CCBot)
AI assistants: mixed (blocked: PerplexityBot)
robots.txt: found, 1 invalid line
Page directives: noindex, noai, GPTBot: noimageai
llms.txt: none
- warn: robots.txt has problems. Fix: Fix the listed lines, serve it as text/plain and keep it under 500 KB.
- warn: Search engine crawlers are blocked. Fix: Allow search crawlers unless you want out of search results.
- warn: The page asks search engines not to index it. Fix: Remove noindex if the page should appear in search.
- info: AI training crawlers are blocked
+3 more findings (pass detail='full' to see all)
Credits: 1 credit used, 498 remaining | fresh | checked 2026-11-14 10:00 UTC