DNS Super checks websites when one of our users asks it to. Requests from our checker carry the name DNSSuperBot; pages loaded in our headless browser carry DNSSuper. This page explains what each one does, so you can recognise it in your logs and decide what to allow.
What DNSSuperBot fetches
DNSSuperBot fetches exactly the page a user asked about, and the few files a check needs to read: for
example /robots.txt, /.well-known/security.txt or a sitemap. It does this for single checks such as the
security headers checker, the redirect checker or the social preview, the way a browser or a link preview
would. It never fills in forms, never logs in, never retries past a bot challenge and never sends cookies
between requests.
The broken link checker is different: it crawls a site. That crawler obeys your robots.txt under the product
token DNSSuperBot, paces its requests, and stops at the number of pages the user paid for.
How it identifies itself
Plain requests send this user agent, unchanged:
DNSSuperBot/0.1 (+https://dnssuper.com/bot)
Pages loaded in our sandboxed browser (rendered SEO audits, colour contrast checks and cookie scans) send the browser's usual user agent with this added at the end:
DNSSuper/1.0 (+https://dnssuper.com/bot)
We never pretend to be another browser or another company's crawler.
How to control it
To keep the crawler out of part of your site, add a group for its token to robots.txt:
User-agent: DNSSuperBot
Disallow: /private/
Single checks fetch only the page a user named, so they don't read robots.txt rules, just as a browser
doesn't. Every site also gets a limit: across all our users and tools we make at most 120 requests a minute
to any one host, and fewer for a single check.
Where requests come from
Our requests come from our own servers. The addresses will be listed here before the checker runs against public sites.
Contact
If our checker causes a problem, or you want us to stop checking your site, email abuse@dnssuper.com. To report a vulnerability in DNS Super itself, use our security page.