What it means
Your robots.txt has lines crawlers can't read, it is served as an HTML page instead of plain text, or it is larger than 500 KB, the most RFC 9309 asks crawlers to read. The detail says which.
Why it matters
Crawlers skip lines they can't parse, so a misspelled Disalow or a missing colon means the rule
isn't applied and pages you meant to keep out are crawled. An HTML page at /robots.txt usually means
the file doesn't exist and the site shows its home page instead, so no rule applies at all.
How to fix it
- Write each rule as
Field: valuewith a colon:User-agent:,Allow:,Disallow:,Sitemap:. - Start each group with one or more
User-agent:lines, followed by its rules. - Serve the file as
text/plainand return 404, not your home page, if it doesn't exist. - Keep it small: use wildcards (
*and$) instead of listing thousands of paths.
Check it worked
Run the AI crawler check again. No invalid lines should be listed, and the finding should be gone.