Info

AI training crawlers are blocked

The short version

Nothing to do if that's your choice.

What it means

For the page you checked, robots.txt disallows at least one AI training crawler (such as GPTBot, ClaudeBot, Google-Extended or CCBot) or AI assistant fetcher (such as ChatGPT-User or PerplexityBot). The result shows the deciding rule and line for each.

Why it matters

This is information, not a problem. Blocking training crawlers asks vendors not to use your pages to train models; blocking assistant fetchers means AI tools can't read or cite the page when people ask about it. robots.txt shows what a site asks for, not what crawlers do, and vendors document which tokens they honour. Google-Extended and Applebot-Extended only control AI use; they don't affect search.

How to fix it

  1. Decide per purpose: training, AI search and assistants can be handled separately.
  2. To block training only, add groups such as User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended, User-agent: CCBot followed by Disallow: /.
  3. To allow assistants to cite you, leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed.
  4. Check vendors' documentation every few months; new tokens appear.

Check it worked

Run the AI crawler check again. The groups should match what you intended.