Warning

robots.txt returns a server error

The short version

Fix robots.txt; crawlers stop when it errors.

What it means

When we fetched /robots.txt, the server answered with a 5xx error. The file itself may be fine; the problem is that the server couldn't deliver it. A missing file (404) is not an error: it simply means every crawler may visit every page.

Why it matters

Search engines treat a robots.txt that fails with a server error as "don't crawl anything". Google pauses crawling of the whole site while it lasts, and after a long outage it may drop pages from its results. One misconfigured rule on a server or CDN can have that effect on every page you have.

How to fix it

  1. Open /robots.txt in a browser and note the status code.
  2. Check that the file exists where your web server expects it, or that the route generating it works.
  3. Look at your server, application or CDN logs for errors on that address.
  4. If you don't need any rules, a plain 404 is fine; a static file with User-agent: * and Allow: / works too.
  5. Make sure bot protection doesn't answer crawlers on that path with an error.

Check it worked

Run the SEO audit again on the same address. The finding should be gone and the grade should rise.