How our checker crawls

DNSSuperBot and the DNS Super browser

What DNSSuperBot fetches, why, how it identifies itself, how fast it goes and how to control it with robots.txt.

DNS Super checks websites when one of our users asks it to. Requests from our checker carry the name DNSSuperBot; pages loaded in our headless browser carry DNSSuper. This page explains what each one does, so you can recognise it in your logs and decide what to allow.

What DNSSuperBot fetches

DNSSuperBot fetches exactly the page a user asked about, and the few files a check needs to read: for example /robots.txt, /.well-known/security.txt or a sitemap. It does this for single checks such as the security headers checker, the redirect checker or the social preview, the way a browser or a link preview would. It never fills in forms, never logs in, never retries past a bot challenge and never sends cookies between requests.

The broken link checker is different: it crawls a site. That crawler obeys your robots.txt under the product token DNSSuperBot, paces its requests, and stops at the number of pages the user paid for.

How it identifies itself

Plain requests send this user agent, unchanged:

DNSSuperBot/0.1 (+https://dnssuper.com/bot)

Pages loaded in our sandboxed browser (rendered SEO audits, colour contrast checks and cookie scans) send the browser's usual user agent with this added at the end:

DNSSuper/1.0 (+https://dnssuper.com/bot)

We never pretend to be another browser or another company's crawler.

How to control it

To keep the crawler out of part of your site, add a group for its token to robots.txt:

User-agent: DNSSuperBot
Disallow: /private/

Single checks fetch only the page a user named, so they don't read robots.txt rules, just as a browser doesn't. Every site also gets a limit: across all our users and tools we make at most 120 requests a minute to any one host, and fewer for a single check.

Where requests come from

Our requests come from our own servers. The addresses will be listed here before the checker runs against public sites.

Contact

If our checker causes a problem, or you want us to stop checking your site, email abuse@dnssuper.com. To report a vulnerability in DNS Super itself, use our security page.