What it means
For the page you checked, robots.txt disallows at least one AI training crawler (such as GPTBot, ClaudeBot, Google-Extended or CCBot) or AI assistant fetcher (such as ChatGPT-User or PerplexityBot). The result shows the deciding rule and line for each.
Why it matters
This is information, not a problem. Blocking training crawlers asks vendors not to use your pages to train models; blocking assistant fetchers means AI tools can't read or cite the page when people ask about it. robots.txt shows what a site asks for, not what crawlers do, and vendors document which tokens they honour. Google-Extended and Applebot-Extended only control AI use; they don't affect search.
How to fix it
- Decide per purpose: training, AI search and assistants can be handled separately.
- To block training only, add groups such as
User-agent: GPTBot,User-agent: ClaudeBot,User-agent: Google-Extended,User-agent: CCBotfollowed byDisallow: /. - To allow assistants to cite you, leave
OAI-SearchBot,Claude-SearchBotandPerplexityBotallowed. - Check vendors' documentation every few months; new tokens appear.
Check it worked
Run the AI crawler check again. The groups should match what you intended.