What it means
When we fetched /robots.txt, the server answered with a 5xx error. The file itself may be fine; the problem is that the server couldn't deliver it. A missing file (404) is not an error: it simply means every crawler may visit every page.
Why it matters
Search engines treat a robots.txt that fails with a server error as "don't crawl anything". Google pauses crawling of the whole site while it lasts, and after a long outage it may drop pages from its results. One misconfigured rule on a server or CDN can have that effect on every page you have.
How to fix it
- Open
/robots.txtin a browser and note the status code. - Check that the file exists where your web server expects it, or that the route generating it works.
- Look at your server, application or CDN logs for errors on that address.
- If you don't need any rules, a plain 404 is fine; a static file with
User-agent: *andAllow: /works too. - Make sure bot protection doesn't answer crawlers on that path with an error.
Check it worked
Run the SEO audit again on the same address. The finding should be gone and the grade should rise.