Warning

robots.txt returns a server error

The short version

Fix robots.txt; crawlers treat a server error as "crawl nothing".

What it means

Requesting /robots.txt on your site returned a 5xx status. Under RFC 9309, crawlers treat a server error on robots.txt as "everything is disallowed" until it answers again.

Why it matters

While robots.txt fails, search engines stop crawling your site, so new and updated pages don't reach search results. Google allows a short grace period and then treats a long outage as if the site asked to be removed. Usually the cause is a framework route or a firewall rule that doesn't expect the file.

How to fix it

  1. Open https://example.com/robots.txt yourself and check the status.
  2. If you have no robots.txt, make the server answer 404 for it, which means "everything allowed".
  3. If you want one, serve a static file; the simplest is two lines: User-agent: * and Allow: /.
  4. Check that your firewall, bot protection or rate limiting doesn't answer 503 to crawlers on this path.

Check it worked

Run the AI crawler check again. robots.txt should show as found or not found, and the finding should be gone.