The lack of a free Reverse DNS (rDNS) lookup database has made large-scale DNS research harder. To address this, we used ZDNS, an open-source, high-performance DNS toolkit developed by Stanford University, to create our own rDNS database. To reduce UDP timeout issues during rDNS operations, we devised a scan-ordering approach that randomised the IP space and improved the efficiency of the scanning process.
Leveraging ZDNS for rDNS Lookups Across the Internet
Understanding rDNS is useful for internet operations and research. Active DNS measurement helps us inspect how providers advertise the use of their IP address space. One of the components of this ecosystem is Reverse DNS (rDNS), which serves an important role in IP database categorisation and ASN (Autonomous System Number) classification. However, running rDNS across the entire internet is not a trivial task.
Previously, Rapid7 provided a free database for rDNS lookups, but it has discontinued the offering. This situation has prompted the need to create our own database, calling for a robust, efficient, and scalable tool to accomplish the task. ZDNS was the right fit.
Introducing ZDNS
ZDNS, a part of the ZMap.io project, is a capable tool developed by Stanford University to support scalable and reproducible DNS research. ZDNS is an open-source DNS measurement framework specifically optimised for large-scale DNS research on the public internet. It can resolve 50 million domains in 10 minutes and query the PTR records of the complete public IPv4 address space in approximately 12 hours.
This high-performance toolkit offers a modular interface, enabling researchers to safely implement new functionalities. Its architecture is designed to expose DNS lookup chains by performing recursive resolution. ZDNS supports a command-line interface and outputs results in JSON, a machine-parsable format.
Enhancements by ZDNS
ZDNS's architecture and feature set are tailored to the challenges of extensive DNS research. Its guiding principles are that the DNS lookup chain is exposed, and that the tool is safe, easy to use, and extensible.
ZDNS's performance optimisations make it a suitable tool for DNS experiments that require querying a large number of names. Parallelism, UDP socket reuse, and selective caching are some of the critical performance optimisations that enable ZDNS to efficiently handle large volumes of DNS queries.
ZDNS's scalability, execution time, and success rate have been evaluated against several existing tools, showcasing its performance. For instance, when it comes to exposing the DNS lookup chain, ZDNS is 85 times faster than Dig. ZDNS also outperforms other higher-performance tools, achieving 2.6 to 3.6 times more successful queries per second and experiencing about 30% less packet drop than MassDNS.
Our rDNS Journey
When we started scanning the whole internet with rDNS, we hit a practical roadblock: UDP timeouts made the scans slow. The system spent too much time waiting for responses from parts of the internet that were either empty or broken.
We used two changes. Firstly, instead of scanning the internet's addresses in order, we mixed them up and scanned randomly. This spread out our requests and stopped the system from getting stuck on troublesome ranges. Secondly, we checked smaller sections of the internet first, so we did not waste time waiting for big chunks of the internet that weren't responding.
With these changes, we scanned the whole internet in 13 days, finding over a billion addresses. The main lesson was straightforward: scan order matters when timeout behaviour dominates runtime.
The result depended on scan order
ZDNS gave us the throughput and structured output we needed, but the tool alone did not make the scan finish. The decisive change was to stop walking adjacent address ranges in sequence. Randomising the work kept slow or empty ranges from dominating a worker's runtime and brought the full scan down to 13 days.
ZDNS is available on GitHub, and the IMC 2022 paper covers its architecture and benchmark results. If you repeat this kind of measurement, treat target ordering, timeout policy, responsible scan rates, and restartability as part of the design rather than as command-line details.
-
Izhikevich, L., Akiwate, G., Berger, B., Drakontaidis, S., Ascheman, A., Pearce, P., Adrian, D., & Durumeric, Z. (2022). ZDNS: a fast DNS toolkit for internet measurement. In Proceedings of the 22nd ACM Internet Measurement Conference (pp. 33-43). https://doi.org/10.1145/3517745.3561434 ↩
-
ZMap Project. (n.d.). ZDNS. GitHub. Retrieved 2023-05-15 13:00, from https://github.com/zmap/zdns. ↩
