Peakhour.IO - Technical

ZDNS - scan the entire internet

2023-06-20T13:00:00+10:00

The lack of a free Reverse DNS (rDNS) lookup database has made large-scale DNS research harder. To address this, we used ZDNS, an open-source, high-performance DNS toolkit developed by Stanford University, to create our own rDNS database. To reduce UDP timeout issues during rDNS operations, we devised a scan-ordering approach that randomised the IP space and improved the efficiency of the scanning process.

Leveraging ZDNS for rDNS Lookups Across the Internet

Understanding rDNS is useful for internet operations and research. Active DNS measurement helps us inspect how providers advertise the use of their IP address space. One of the components of this ecosystem is Reverse DNS (rDNS), which serves an important role in IP database categorisation and ASN (Autonomous System Number) classification. However, running rDNS across the entire internet is not a trivial task.

Previously, Rapid7 provided a free database for rDNS lookups, but it has discontinued the offering. This situation has prompted the need to create our own database, calling for a robust, efficient, and scalable tool to accomplish the task. ZDNS was the right fit.

Introducing ZDNS

ZDNS, a part of the ZMap.io project, is a capable tool developed by Stanford University to support scalable and reproducible DNS research. ZDNS is an open-source DNS measurement framework specifically optimised for large-scale DNS research on the public internet. It can resolve 50 million domains in 10 minutes and query the PTR records of the complete public IPv4 address space in approximately 12 hours.

This high-performance toolkit offers a modular interface, enabling researchers to safely implement new functionalities. Its architecture is designed to expose DNS lookup chains by performing recursive resolution. ZDNS supports a command-line interface and outputs results in JSON, a machine-parsable format.

Enhancements by ZDNS

ZDNS's architecture and feature set are tailored to the challenges of extensive DNS research. Its guiding principles are that the DNS lookup chain is exposed, and that the tool is safe, easy to use, and extensible.

ZDNS's performance optimisations make it a suitable tool for DNS experiments that require querying a large number of names. Parallelism, UDP socket reuse, and selective caching are some of the critical performance optimisations that enable ZDNS to efficiently handle large volumes of DNS queries.

ZDNS's scalability, execution time, and success rate have been evaluated against several existing tools, showcasing its performance. For instance, when it comes to exposing the DNS lookup chain, ZDNS is 85 times faster than Dig. ZDNS also outperforms other higher-performance tools, achieving 2.6 to 3.6 times more successful queries per second and experiencing about 30% less packet drop than MassDNS.

Our rDNS Journey

When we started scanning the whole internet with rDNS, we hit a practical roadblock: UDP timeouts made the scans slow. The system spent too much time waiting for responses from parts of the internet that were either empty or broken.

We used two changes. Firstly, instead of scanning the internet's addresses in order, we mixed them up and scanned randomly. This spread out our requests and stopped the system from getting stuck on troublesome ranges. Secondly, we checked smaller sections of the internet first, so we did not waste time waiting for big chunks of the internet that weren't responding.

With these changes, we scanned the whole internet in 13 days, finding over a billion addresses. The main lesson was straightforward: scan order matters when timeout behaviour dominates runtime.

The result depended on scan order

ZDNS gave us the throughput and structured output we needed, but the tool alone did not make the scan finish. The decisive change was to stop walking adjacent address ranges in sequence. Randomising the work kept slow or empty ranges from dominating a worker's runtime and brought the full scan down to 13 days.

ZDNS is available on GitHub, and the IMC 2022 paper covers its architecture and benchmark results. If you repeat this kind of measurement, treat target ordering, timeout policy, responsible scan rates, and restartability as part of the design rather than as command-line details.

Izhikevich, L., Akiwate, G., Berger, B., Drakontaidis, S., Ascheman, A., Pearce, P., Adrian, D., & Durumeric, Z. (2022). ZDNS: a fast DNS toolkit for internet measurement. In Proceedings of the 22nd ACM Internet Measurement Conference (pp. 33-43). https://doi.org/10.1145/3517745.3561434 ↩
ZMap Project. (n.d.). ZDNS. GitHub. Retrieved 2023-05-15 13:00, from https://github.com/zmap/zdns. ↩

Double MAD?

2023-05-15T13:00:00+10:00

This article explores the use of Double Median Absolute Deviation (Double MAD) for [anomaly detection](/learning/threat-detection/what-is-anomaly-detection/) in time series
data, particularly in skewed or non-symmetric distributions. Double MAD, which calculates two median absolute
deviations — one for data below the median and one for data above — provides a more nuanced approach than traditional
MAD, allowing for accurate detection of anomalies even in skewed data distributions. We also delve into its application
in identifying slow abuse, like bots, by catching lower range anomalies. However, it's important to note Double MAD's
limitations such as not capturing seasonal data shape and trends over time. A comparison is also drawn with the Z-score
method, highlighting that the choice between the two depends on the nature of your data. The article provides insights
into the practical implementation of Double MAD and its potential to improve your data analysis toolkit.

Operational systems increasingly rely on time-series data for decisions. Anomaly detection is one practical use: by identifying patterns that deviate from the norm, businesses can investigate potential issues early or understand unexpected opportunities.

One useful technique for anomaly detection is the Median Absolute Deviation (MAD) and, more specifically, its extension, the Double MAD. This article explains where Double MAD fits in time-series anomaly detection and how it can help identify anomalous clients.

Understanding MAD and Double MAD

MAD, a robust measure of variability, is less susceptible to outliers than standard deviation. It calculates the median of absolute deviations from the data's median, often providing a better representation of 'normal' behaviour in datasets with skewed distributions or outliers.

Double MAD is an extension of MAD, where two MADs are calculated — one for the data below the median and another for the data above. This split gives the detection process a better fit for asymmetric data, which is common in real-world time series data.

Why Double MAD?

While MAD provides a robust way to understand the 'normal' range of a dataset, it assumes a symmetric distribution of data around the median, which may not always hold true. Double MAD is useful where that assumption breaks down, offering an improved anomaly detection process for skewed or asymmetric datasets.

In time-series analysis, especially with 24-hour cycles like web traffic or server usage, patterns can exhibit seasonality and trend components. These patterns can often be asymmetric, making Double MAD a valuable tool for capturing the variability in different parts of the data.

Using Double MAD in Anomaly Detection

The Double MAD implementation provided uses Rust, a system programming language known for speed and memory safety. The code calculates the lower and upper MAD values, along with their respective thresholds. Anomalies can then be detected by comparing each data point to these thresholds.

An anomaly is defined as a data point that deviates significantly from the expected range. If a data point falls below the lower MAD threshold or above the upper one, it can be flagged as an anomaly. This approach is especially effective when handling datasets with high variability or extreme values.

Double MAD for Anomalous Client Detection

Beyond time-series data, Double MAD can also be instrumental in identifying anomalous behaviour among clients. By comparing each client's behaviour against the Double MAD of the time-series data, teams can pinpoint clients that deviate from the norm.

For instance, in the context of web service usage, an anomalous client might be one that is sending an unusually high or low number of requests. By using Double MAD, you can flag such outliers and take appropriate action, such as investigating potential misuse or reaching out to understand and address any issues they may be facing.

Detecting Lower-Range Anomalies: A Case of Slow Abuse

An interesting application of Double MAD is in detecting lower-range anomalies, a pattern often associated with slow abuse such as bots or Distributed Denial of Service (DDoS) attacks. These abuses are characterised by an unusually low frequency of activity that is consistent over a prolonged period. This consistent, low-level activity can fly under the radar of typical anomaly detection systems.

By setting a lower MAD threshold, Double MAD can effectively detect these lower-range anomalies, providing early warning of slow abuse. Its ability to detect both high and low anomalies makes Double MAD a flexible tool for anomaly detection.

The Math Behind Double MAD

To illustrate the power of Double MAD, let's consider a dataset from a right-skewed distribution. Applying the conventional MAD approach might lead to false positives where normal data points are marked as outliers. This is because MAD uses a symmetric interval around the median, which doesn't account for the skewed nature of our data.

With Double MAD, we instead calculate two MADs — one for the data below the median (MAD-lower) and another for the data above (MAD-upper). Outlier thresholds are then defined using these two MADs. The lower threshold is calculated as the median minus a multiplier (k) times MAD-lower. The upper threshold is the median plus k times MAD-upper.

This approach takes into account the asymmetric nature of our data, providing more accurate anomaly detection. For example, in a right-skewed distribution, Double MAD would correctly identify only the extreme right tail values as outliers without incorrectly flagging data points on the left tail.

Wrapping Up

Accurate anomaly detection matters when teams rely on time-series data to operate and investigate systems. The Double MAD approach provides a robust method for this, allowing businesses to better understand their data, spot potential issues early, and make more informed decisions.

Whether you're monitoring web traffic, server usage, or client behaviour, leveraging Double MAD can offer valuable insights and help ensure your operations continue to run smoothly. The ability to detect both high and low anomalies makes it especially powerful, providing protection against potential threats like slow abuse.

Understanding and implementing Double MAD gives your data analysis toolkit a more complete view of asymmetric data and helps you detect potential anomalies earlier.

Double MAD vs the Rest

2023-05-15T13:00:00+10:00

Limitations of Double MAD and Comparison with Z-Score

Double MAD is useful for anomaly detection, but it has clear limits. One is that it does not account for the shape of seasonal data. Time series data often show cyclical patterns by time of day, week, or year. For instance, web traffic to an e-commerce site might spike during holidays and dip on off-peak days.

Double MAD can capture shifts in the median of these data, but it does not consider the shape or pattern within these cycles. It might therefore miss anomalies that occur within a specific season, or flag normal seasonal variations as anomalies.

Another limitation is that Double MAD does not account for trends over time. If your time series data shows a consistent increase or decrease, Double MAD might misinterpret this trend as a series of anomalies.

Double MAD vs. Z-Score

In anomaly detection, Double MAD is often compared with the more traditional Z-score method. A Z-score measures how many standard deviations a data point is from the mean. It assumes that the data follows a Gaussian (or normal) distribution, which often does not hold true for real-world data.

Double MAD, on the other hand, is a non-parametric method that does not make assumptions about the distribution of data. This makes it more robust to outliers and skewed distributions.

However, Z-score has an advantage when data follows a Gaussian distribution, or when the data size is large enough for the Central Limit Theorem to take effect. It also accounts for the mean and standard deviation, giving it an edge in datasets where these measures are informative.

In contrast, Double MAD is more robust for datasets with outliers or skewed distributions, as it uses the median and absolute deviations from the median, which are less sensitive to extreme values.

Both Double MAD and Z-score have strengths, and the choice between them should be guided by the nature of your data. Understanding these nuances helps you choose the method that fits your specific use case.

Efficiently Generating and Printing All IPv4 Addresses in a Random Order

2023-05-15T13:00:00+10:00

In this article, we explored an efficient way to generate and print all possible IPv4 addresses in a random order using
a Linear Congruential Generator (LCG). The LCG, a pseudorandom number generator, helps generate the full range of IP
addresses without consuming vast amounts of memory, making this approach suitable for systems with memory constraints.
We also provided a Python script demonstrating the concept, along with a test case to verify its correctness.

We then delved into the importance of randomising IP addresses, highlighting its critical role in areas like security
testing, load balancing, enhancing privacy, and web scraping. However, while using this technique, it's essential to
respect privacy and legality, as misuse can lead to legal repercussions.

In summary, the ability to generate and print all IPv4 addresses in a random order is a powerful tool, especially in the
realm of networking and cybersecurity, and can be achieved efficiently using the LCG approach.

In networking, some tasks require generating and printing every possible IPv4 address. Doing that in random order without a large memory footprint is less straightforward. The IPv4 address space contains 2^32, or 4,294,967,296, values. Storing all of them in memory at once is not feasible for most systems.

This article uses a Linear Congruential Generator (LCG) to generate the full range without holding it in memory.

Linear Congruential Generator

A Linear Congruential Generator is a type of pseudorandom number generator that can run without storing the whole sequence. It generates each next value from a linear equation based on the previous value. The basic form of the LCG is:

X_(n+1) = (a*X_n + c) mod m

Here, a, c, and m are constants, and X_n is the nth number in the sequence. The initial seed or starting point of the sequence is X_0.

If we choose parameters such that the period of the LCG is maximum (equal to the modulus), and the modulus equals the range of numbers we're generating (the number of possible IPv4 addresses in this case), then the LCG should generate each number in the range exactly once before repeating.

Here is that idea in Python:

import ipaddress

def lcg(modulus, a, c, seed):
    """Linear congruential generator."""
    while True:
        seed = (a * seed + c) % modulus
        yield seed

start_ip_str = '0.0.0.0'
end_ip_str = '255.255.255.255'

start_ip = int(ipaddress.IPv4Address(start_ip_str))
end_ip = int(ipaddress.IPv4Address(end_ip_str))

modulus = end_ip - start_ip + 1
a = 1664525
c = 1013904223
seed = 1  # Arbitrary seed

generator = lcg(modulus, a, c, seed)

for _ in range(modulus):
    ip_int = start_ip + next(generator)
    ip = ipaddress.IPv4Address(ip_int)
    print(ip)

The script first defines the parameters of the LCG. a, c, and seed are set to values used in Numerical Recipes' LCG, a well-known and widely used LCG. The modulus is set to the total number of possible IPv4 addresses.

The function lcg() is implemented as a Python generator, yielding the next number in the sequence each time it is called.

The loop then generates and prints each IP address. It adds the output of the LCG to the start IP address, converts it back to an IP address string, and prints it.

This script generates and prints each IP address in random (more precisely, pseudorandom) order using very little memory. Each IP address is printed exactly once, assuming the period of the LCG is maximum.

The point is that a small pseudorandom number generator can walk a large range without materialising the whole list. The code can still be tweaked and optimised for specific requirements and constraints.

The Importance of Randomising IP Addresses

Randomising IP addresses has practical uses in several networking workflows:

1. Security Testing and Penetration Testing

In cybersecurity, randomising IP addresses can help simulate attacks on a network from various sources. By using a range of IP addresses in no particular order, penetration testers can mimic the unpredictable nature of real-world cyber threats and build more robust test scenarios.

2. Load Balancing and Network Traffic Simulation

Randomising IP addresses is also useful in network traffic simulations. Network engineers and administrators can use this approach to test network resilience and capacity. By sending requests to servers from randomised IP addresses, they can evaluate how well their load balancing strategies are functioning and whether the network can handle high traffic loads from various sources.

3. Anonymity and Privacy

In some cases, randomising IP addresses can help with privacy and anonymity. While it is not a foolproof method, using a different IP address for each request can make it more challenging for online trackers to monitor user activity. It is a common practice among privacy-focused internet users and is also used in some VPN (Virtual Private Network) services.

4. Web Scraping

Web scraping is another area where randomising IP addresses is useful. To prevent being blocked by anti-bot measures, web scrapers often need to rotate their IP addresses. By using a different IP address for each request, they can avoid being detected and blocked by the sites they are scraping.

Randomising IP addresses can be useful in these cases, but privacy and legality still matter. Unauthorised network scanning, privacy breaches, and cyberattacks are illegal and punishable under law.

Generating and printing every possible IPv4 address in a random order is a valuable technique with various applications, from network testing to privacy enhancement. With the Linear Congruential Generator approach, we can do it efficiently.

Scaling anomaly detection with RRCF

2023-05-15T13:00:00+10:00

As data volumes grow, the anomaly detection process has to scale with them. RRCF is efficient, but large, high-dimensional datasets can still create performance challenges. The following strategies focus on reducing dimensionality, smoothing bursts of input, and distributing independent work.

Compute Summary Statistics Instead of Shingling

Shingling transforms a single time series into a multivariate one by stacking lagged versions of the data. This can help capture temporal dependencies, but it also increases the dimensionality of the points inserted into each tree, which can slow the algorithm down.

An alternative is to compute summary statistics that capture the types of anomalies you are looking for. For instance, if you are detecting spikes, the data points could consist of second central differences. If you are looking for long-term trends, the data points could consist of rolling means at different window sizes. This reduces the dimension of the points inserted into each tree, improving performance.

Buffer Input and Compute Rolling Summary Statistics

When data arrives too quickly to be inserted into the trees directly, buffer the input and compute rolling summary statistics (mean, median, max, etc.). This reduces the number of points that need to be inserted into the trees and helps the algorithm keep up with the streaming data.

Parallelisation

RRCF can be parallelised, which is particularly useful when dealing with multiple independent time series. Different RRCF instances can be run for each time series, using separate processes or server instances. This distributes the computational load and can improve performance.

For instance, if you have 10 independent time series, you can run 10 instances of RRCF in parallel, each focusing on one time series. This scales the anomaly detection process to handle larger volumes of data.

Conclusion

Scaling RRCF for large datasets usually means reducing the work each tree has to do, controlling input volume, and parallelising where the data allows it. Summary statistics, input buffering, and independent RRCF instances can help manage high-dimensional data and high data velocities without changing the underlying anomaly detection goal.

Applied RRCF - thresholding techniques.

2023-05-15T13:00:00+10:00

Once we've applied the RRCF algorithm to our streaming data, the resulting scores measure how anomalous each data point is. To classify data points as "normal" or "anomalous", we still need to set a threshold. This defines the level of deviation considered anomalous and controls how often anomalies are over-identified or missed.

Why is Thresholding Needed?

Thresholding matters in anomaly detection because it separates normal and anomalous behaviour. Without a threshold, the scores still indicate relative degrees of anomalousness, but they do not provide a clear dividing line between normal points and anomalies.

Set the threshold too low and normal data points may be misclassified as anomalies, increasing false positives. Set it too high and actual anomalies may be missed, increasing false negatives.

How to Set the Threshold?

There are several ways to set a threshold for RRCF scores, including the Median Absolute Deviation (MAD), Min/Max, and others. The right method depends on the characteristics of the data and the specific use case.

Median Absolute Deviation (MAD)

The Median Absolute Deviation is a robust measure of variability in a data set. For RRCF scores, MAD can be used to set a threshold. A typical approach is to set the threshold as some multiple of the MAD above the median. This approach is robust to outliers and can be useful when the data has heavy-tailed distributions.

Min/Max

Another approach is to use the minimum and maximum RRCF scores to set the threshold. This could mean setting the threshold as a percentage of the range between the minimum and maximum scores. The method is straightforward, but it can be sensitive to extreme score values.

Z-Score

Several other methods can be used to set the threshold, depending on the data. These include statistical techniques such as setting the threshold based on standard deviations from the mean, using quartiles of the data, or using machine learning techniques to dynamically adjust the threshold based on observed data.

Conclusion

Thresholding gives anomaly detection a clear boundary between normal and anomalous scores, which helps identify potential issues such as cyber threats or system errors. The choice of thresholding method depends on the use case and the characteristics of the data. Whatever method is used, the threshold needs to balance anomaly detection against the risk of false positives and false negatives.