For years, rate limiting has been a foundational tool for protecting websites and APIs from abuse. The concept is simple: limit the number of requests a single "user" can make in a given period. If a user exceeds the limit (e.g., 10 login attempts in a minute), they are temporarily blocked.
The critical question, however, has always been: how do you identify a "user"? Traditionally, the answer was the IP address. The assumption was that one IP address equaled one user. In the early days of the internet, this was a reasonable approximation. Today, it's a dangerously outdated assumption that leaves systems vulnerable to modern attacks.
The IP address is no longer a reliable identifier for a single user or device. Here's why:
- Proxy Networks: Attackers don't use a single IP address. They use vast residential proxy networks to rotate their requests through thousands or even millions of different IP addresses, making each request look like it's coming from a new user.
- Shared IPs (CGNAT): On the flip side, a single IP address can represent thousands of legitimate users. Mobile carriers use Carrier-Grade NAT (CGNAT) to have many mobile devices share the same public IP. Similarly, an entire office building or university campus might appear to the internet as a single IP.
- Distributed Attacks: Modern automated attacks, like Layer 7 DDoS or credential stuffing, are inherently distributed. Attackers use botnets or proxy networks to spread their attack across a huge number of IPs, ensuring that no single IP ever exceeds a traditional rate limit.
Blocking a shared IP because of one bad actor can result in massive collateral damage, denying access to thousands of legitimate users. Conversely, failing to see that thousands of IPs are part of a single coordinated attack means the attack succeeds. In short, traditional IP-based rate limiting is no longer effective.
The New Way: Advanced Rate Limiting
Advanced Rate Limiting solves this problem by moving beyond the IP address. Instead of grouping requests by a single, unreliable identifier, it allows you to count requests using more stable and meaningful characteristics of the connection or the software making it.
This new approach groups requests using identifiers like:
- TLS/HTTP2 Fingerprints: Every client application (like a browser or a script) has a unique "fingerprint" based on how it initiates a secure connection (TLS) or communicates over HTTP/2. This fingerprint remains consistent even as an attacker rotates through thousands of IP addresses. By rate limiting based on the TLS fingerprint, you can track and block the underlying automation tool itself, not just the IPs it's using.
- Device Characteristics: A fingerprint can be constructed from a range of attributes, including the device's operating system, browser version, and more. This allows for the detection of repeated requests coming from the same class of device.
- A Combination of Headers: For authenticated APIs, you can group requests by an Authorization header or API key, ensuring fair usage and preventing abuse by a single authenticated client.
Practical Use Cases
The power of advanced rate limiting becomes clear when applied to real-world threats:
-
Mitigating Distributed Credential Stuffing: An attacker using a tool like OpenBullet launches a credential stuffing attack against your login page, rotating through thousands of residential proxy IPs. Traditional rate limiting is useless. However, the OpenBullet software has a consistent TLS fingerprint. By setting a rule to limit failed login attempts per TLS fingerprint, you can instantly detect and block the entire distributed attack, regardless of how many IPs are involved.
-
Protecting APIs from Abuse: A partner is abusing their API key, sending far too many requests and degrading service for other users. By rate limiting based on the
Authorizationheader, you can enforce usage limits on a per-client basis, ensuring fair access for everyone without affecting other users. -
Stopping Content Scrapers: A scraper is hammering your e-commerce site to steal pricing data. They are using a botnet to distribute the requests across hundreds of IPs. However, the scraping script they are using has a unique combination of a user-agent and a TLS fingerprint. Advanced rate limiting can count requests based on this combined signature and block the scraper, protecting your intellectual property.
In today's threat landscape, where attackers are always distributed, your defenses must be able to see the single actor behind the many IPs. Advanced rate limiting provides that crucial visibility, making it an essential component of any modern application security strategy.
