# AI crawler access API

See which search engines, AI training crawlers and AI assistants a site's robots.txt allows, and whether it has llms.txt.

`GET /v1/ai-crawlers`

- Price: Costs 1 credit per call; cached answers cost the same. Calls that fail upstream (502) are free.
- Cache: Answers are cached for up to 15 minutes, per account except for a site's home page; send fresh=true to ask the source again.
- MCP tool: `check_ai_crawlers`

Which search, AI training and AI assistant crawlers robots.txt allows for a page, page directives and llms.txt. Costs 1 credit per call; cached answers cost the same. Calls that fail upstream (502) are free. Answers are cached for up to 15 minutes, per account except for a site's home page; send fresh=true to ask the source again. A retry with the same `X-Request-Id` returns the stored result of the first call for 24 hours without charging again, marked with the `X-Replayed: true` header.

## Query parameters

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | yes | A page URL or a domain; a domain means its home page. |
| `fresh` | boolean | no | Skip the cache and ask the source again; costs the same. |

## Response

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `tool` | string | yes |  |
| `target` | string | yes |  |
| `status` | "ok" | "partial" | "error" | yes |  |
| `fetched_at` | string | no |  |
| `cached` | boolean | no |  |
| `credits_used` | integer | no | Credits this call took from your balance. 0 for free tools, for a replay of an earlier call, and for a free check run on the website without signing in |
| `summary` | string | yes |  |
| `score` | integer or null | no |  |
| `findings` | array of object | no |  |
| `credits_remaining` | integer | yes | Your balance after this call. It can be below zero after a refund; free tools still work then |
| `data` | object | yes | This tool's result |
| `data.url` | string | yes |  |
| `data.robots_url` | string | yes |  |
| `data.robots_status` | "ok" | "missing" | "server_error" | "unreachable" | yes |  |
| `data.robots_http_status` | integer or null | yes |  |
| `data.sitemaps` | array of string | yes |  |
| `data.invalid_lines` | array of integer | yes |  |
| `data.crawlers` | array of object | yes |  |
| `data.crawlers.token` | string | yes |  |
| `data.crawlers.vendor` | string | yes |  |
| `data.crawlers.purpose` | "search" | "training" | "retrieval" | yes |  |
| `data.crawlers.user_triggered` | boolean | yes |  |
| `data.crawlers.allowed` | boolean | yes |  |
| `data.crawlers.reason` | "rule" | "no_matching_rule" | "no_group" | "missing" | "server_error" | "unreachable" | yes |  |
| `data.crawlers.rule` | string or null | yes |  |
| `data.crawlers.rule_line` | integer or null | yes |  |
| `data.crawlers.group_agents` | array of string or null | yes |  |
| `data.groups` | object | yes |  |
| `data.content_signals` | array of object | yes |  |
| `data.page_directives` | array of string | yes |  |
| `data.llms_txt` | object | yes |  |
| `data.llms_txt.url` | string | yes |  |
| `data.llms_txt.present` | boolean | yes |  |
| `data.llms_txt.status` | integer or null | yes |  |
| `data.llms_txt.starts_with_h1` | boolean | yes |  |
| `data.llms_txt.bytes` | integer or null | yes |  |
| `data.catalog_checked` | string | yes |  |

## Errors

- `400` Invalid domain, URL or parameter, a TLD this tool can't check, or an X-Request-Id already used for another call (`invalid_domain`, `invalid_url`, `invalid_request`, `unsupported_tld`)
- `401` Missing, unknown or revoked API key, or one whose creator is no longer a member of the account (`unauthorized`)
- `402` Balance below the call's price (`insufficient_credits`)
- `403` Account suspended, the key's creator is over the plan's seats, or an anonymous check without a valid proof-of-work or for a tool that needs an account, or bulk checks before a first purchase (`account_suspended`, `plan_limit`, `challenge_required`, `sign_in_required`, `bulk_locked`)
- `429` Too many requests, today's free checks are used up, or too many website scans at once; wait for the number of seconds in the Retry-After header (`rate_limited`, `free_limit_reached`, `too_many_jobs`). Not charged
- `500` Unexpected error (`internal_error`)
- `502` Every upstream source failed; not charged (`source_unavailable`). A Retry-After header says how long to wait when the source asked us to
- `503` Too many open event streams on this server, the checked site is getting too many checks, or the browser or link checker is busy; try again after the seconds in the Retry-After header (`busy`, `site_busy`, `browser_busy`, `crawl_busy`). Not charged

## Example

```sh
curl "https://api.dnssuper.com/v1/ai-crawlers?url=example.com" \
  -H "Authorization: Bearer $DNSSUPER_API_KEY"
```

```python
import os

import httpx

client = httpx.Client(
    base_url="https://api.dnssuper.com",
    headers={"Authorization": f"Bearer {os.environ['DNSSUPER_API_KEY']}"},
    timeout=60,
)
response = client.get("/v1/ai-crawlers", params={
    "url": "example.com",
})
response.raise_for_status()
result = response.json()
print(result["summary"], result["credits_used"])
```

```ts
const response = await fetch(`https://api.dnssuper.com/v1/ai-crawlers?${new URLSearchParams({"url":"example.com"})}`, {
  headers: { Authorization: `Bearer ${process.env.DNSSUPER_API_KEY}` },
});
if (!response.ok) throw new Error((await response.json()).error.message);
const result = await response.json();
console.log(result.summary, result.credits_used);
```

Example response:

```json
{
  "tool": "ai-crawler-checker",
  "target": "https://example.com/blog/post",
  "status": "ok",
  "fetched_at": "2026-11-14T10:00:00Z",
  "cached": false,
  "credits_used": 1,
  "summary": "Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.",
  "findings": [
    {
      "id": "ai_crawlers.robots_invalid",
      "severity": "warn",
      "title": "robots.txt has problems",
      "detail": "Lines 16 aren't valid.",
      "fix": "Fix the listed lines, serve it as text/plain and keep it under 500 KB."
    },
    {
      "id": "ai_crawlers.search_blocked",
      "severity": "warn",
      "title": "Search engine crawlers are blocked",
      "detail": "Blocked: Bingbot.",
      "fix": "Allow search crawlers unless you want out of search results."
    },
    {
      "id": "ai_crawlers.page_noindex",
      "severity": "warn",
      "title": "The page asks search engines not to index it",
      "detail": "",
      "fix": "Remove noindex if the page should appear in search."
    },
    {
      "id": "ai_crawlers.training_blocked",
      "severity": "info",
      "title": "AI training crawlers are blocked",
      "detail": "Blocked: GPTBot, CCBot.",
      "fix": "Nothing to do if that's your choice."
    },
    {
      "id": "ai_crawlers.retrieval_blocked",
      "severity": "info",
      "title": "AI assistant fetchers are blocked",
      "detail": "Blocked: PerplexityBot.",
      "fix": "Nothing to do if that's your choice."
    },
    {
      "id": "ai_crawlers.noai_directive",
      "severity": "info",
      "title": "The page asks not to be used for AI",
      "detail": "noai, GPTBot: noimageai.",
      "fix": "Nothing to do; not every crawler honours noai."
    },
    {
      "id": "llms_txt.missing",
      "severity": "info",
      "title": "No llms.txt",
      "detail": "",
      "fix": "Nothing to do; llms.txt is an optional proposal."
    }
  ],
  "data": {
    "url": "https://example.com/blog/post",
    "robots_url": "https://example.com/robots.txt",
    "robots_status": "ok",
    "robots_http_status": 200,
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ],
    "invalid_lines": [
      16
    ],
    "crawlers": [
      {
        "token": "Googlebot",
        "vendor": "Google",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Bingbot",
        "vendor": "Microsoft",
        "purpose": "search",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 7,
        "group_agents": [
          "Bingbot"
        ]
      },
      {
        "token": "Applebot",
        "vendor": "Apple",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "DuckDuckBot",
        "vendor": "DuckDuckGo",
        "purpose": "search",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Google-Extended",
        "vendor": "Google",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "GPTBot",
        "vendor": "OpenAI",
        "purpose": "training",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 4,
        "group_agents": [
          "GPTBot",
          "CCBot"
        ]
      },
      {
        "token": "ClaudeBot",
        "vendor": "Anthropic",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "CCBot",
        "vendor": "Common Crawl",
        "purpose": "training",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /",
        "rule_line": 4,
        "group_agents": [
          "GPTBot",
          "CCBot"
        ]
      },
      {
        "token": "Applebot-Extended",
        "vendor": "Apple",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Meta-ExternalAgent",
        "vendor": "Meta",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Amazonbot",
        "vendor": "Amazon",
        "purpose": "training",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "OAI-SearchBot",
        "vendor": "OpenAI",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 13,
        "group_agents": [
          "OAI-SearchBot"
        ]
      },
      {
        "token": "ChatGPT-User",
        "vendor": "OpenAI",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Claude-SearchBot",
        "vendor": "Anthropic",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Claude-User",
        "vendor": "Anthropic",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "PerplexityBot",
        "vendor": "Perplexity",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": false,
        "reason": "rule",
        "rule": "Disallow: /blog/",
        "rule_line": 10,
        "group_agents": [
          "PerplexityBot"
        ]
      },
      {
        "token": "Perplexity-User",
        "vendor": "Perplexity",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "Meta-ExternalFetcher",
        "vendor": "Meta",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "DuckAssistBot",
        "vendor": "DuckDuckGo",
        "purpose": "retrieval",
        "user_triggered": false,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      },
      {
        "token": "MistralAI-User",
        "vendor": "Mistral AI",
        "purpose": "retrieval",
        "user_triggered": true,
        "allowed": true,
        "reason": "rule",
        "rule": "Allow: /",
        "rule_line": 18,
        "group_agents": [
          "*"
        ]
      }
    ],
    "groups": {
      "search": "mixed",
      "training": "mixed",
      "retrieval": "mixed"
    },
    "content_signals": [],
    "page_directives": [
      "noindex",
      "noai",
      "GPTBot: noimageai"
    ],
    "llms_txt": {
      "url": "https://example.com/llms.txt",
      "present": false,
      "status": 404,
      "starts_with_h1": false
    },
    "catalog_checked": "2026-11-15"
  },
  "credits_remaining": 498
}
```

## MCP

```json
{
  "tool": "check_ai_crawlers",
  "arguments": {
    "url": "example.com"
  }
}
```

```text
Search: mixed; AI training: mixed; AI assistants: mixed. robots.txt shows what a site asks for, not what crawlers do.
Search: mixed (blocked: Bingbot)
AI training: mixed (blocked: GPTBot, CCBot)
AI assistants: mixed (blocked: PerplexityBot)
robots.txt: found, 1 invalid line
Page directives: noindex, noai, GPTBot: noimageai
llms.txt: none
- warn: robots.txt has problems. Fix: Fix the listed lines, serve it as text/plain and keep it under 500 KB.
- warn: Search engine crawlers are blocked. Fix: Allow search crawlers unless you want out of search results.
- warn: The page asks search engines not to index it. Fix: Remove noindex if the page should appear in search.
- info: AI training crawlers are blocked
+3 more findings (pass detail='full' to see all)
Credits: 1 credit used, 498 remaining | fresh | checked 2026-11-14 10:00 UTC
```
