What it means
For the page you checked, robots.txt disallows at least one search engine crawler, such as Googlebot or Bingbot. The result shows the rule and line that decide it for each crawler.
Why it matters
A page search crawlers can't fetch can't be indexed properly, so it drops out of search results
or appears without a description. Often this is an accident: a rule meant for AI crawlers or a staging
site, such as User-agent: * with Disallow: /, that also catches search engines. robots.txt shows what
a site asks for; well-behaved crawlers follow it.
How to fix it
- Find the deciding rule and line in the result.
- If the block is meant for AI training crawlers, name them in their own group, for example
User-agent: GPTBotthenDisallow: /, and leaveUser-agent: *allowing everything. - Remove
Disallow: /copied from a staging site. - To keep a page out of search while still letting it be crawled, use
<meta name="robots" content="noindex">instead; a crawler blocked by robots.txt never sees that tag.
Check it worked
Run the AI crawler check again. The search engines group should show all allowed, and the finding should be gone.