Warning

robots.txt has problems

The short version

Fix the listed lines, serve it as text/plain and keep it under 500 KB.

What it means

Your robots.txt has lines crawlers can't read, it is served as an HTML page instead of plain text, or it is larger than 500 KB, the most RFC 9309 asks crawlers to read. The detail says which.

Why it matters

Crawlers skip lines they can't parse, so a misspelled Disalow or a missing colon means the rule isn't applied and pages you meant to keep out are crawled. An HTML page at /robots.txt usually means the file doesn't exist and the site shows its home page instead, so no rule applies at all.

How to fix it

  1. Write each rule as Field: value with a colon: User-agent:, Allow:, Disallow:, Sitemap:.
  2. Start each group with one or more User-agent: lines, followed by its rules.
  3. Serve the file as text/plain and return 404, not your home page, if it doesn't exist.
  4. Keep it small: use wildcards (* and $) instead of listing thousands of paths.

Check it worked

Run the AI crawler check again. No invalid lines should be listed, and the finding should be gone.