Peakhour.IO - DDoS

HTTP/2 Rapid Reset Attack Deepdive

2023-10-12T00:00:00+11:00

Distributed Denial of Service (DDoS) attack vectors keep changing. The recent spike in HTTP/2-based DDoS attacks has been notable for its volume, with some attacks surpassing 398 million requests per second. Peakhour observed these attacks and worked through how to understand and mitigate them. This article explains how they work and what operators can do to reduce exposure.

The Rise of HTTP/2 in DDoS Attacks

HTTP/2 was designed to make web traffic more efficient. The same features that improve performance for legitimate users can also be abused in DDoS traffic.

Much of HTTP/2's efficiency lies in "stream multiplexing." It allows multiple messages to be sent over a single TCP connection. While HTTP/1.1 processes each request serially, HTTP/2 can manage multiple concurrent streams on a single connection. This means a client can send multiple requests in a single round trip, increasing how much work each connection can drive.

The 'Rapid Reset' Attack Explained

The "Rapid Reset" attack is a specific DDoS technique built around HTTP/2. The attacker starts by opening multiple streams, much like in a standard HTTP/2 attack. However, instead of waiting for responses, they cancel each request immediately.

The client does this by sending a RST_STREAM frame, indicating that a previous stream should be cancelled. The rapid request-and-reset sequence means the server spends resources processing the request, only for it to be cancelled before a response is generated. This tactic amplifies the server's workload without the attacker needing to wait for responses, which increases the pressure each connection can place on the server.

Variants of the Rapid Reset Attack

Attackers also used variations of the Rapid Reset attack:

One variant involves delaying the reset action. The attacker opens multiple streams, waits, then cancels the streams and instantly opens new ones. This method can evade some rate-based defences.
Another variant avoids stream cancellations. Instead, the attacker tries to open more streams than the server allows. This aims to keep the server continually busy, processing a near-constant flow of requests.

Effective Mitigation Techniques

Mitigation is not as simple as blocking individual malicious requests. A more effective approach is to close the entire TCP connection when malicious activity is detected. The HTTP/2 protocol supports connection termination through the GOAWAY frame. This feature needs to be used aggressively to prevent Rapid Reset attacks, rather than relying on the more passive, standard implementation.

Deciding which connections to treat as malicious is a challenge. One potential strategy is to monitor connection statistics. If a connection exceeds a set threshold of cancelled requests, it might be deemed malicious. Responses to suspect activity could range from sending a GOAWAY frame to terminating the TCP connection.

For the non-cancelling variant, the best approach is to shut down connections that breach the concurrent stream limit, either immediately or after a few violations.

Broader Protocol Implications

These attack techniques are specific to HTTP/2, but the wider protocol lesson still matters. The HTTP/3 (QUIC) protocol isn't directly vulnerable in the same way. As a precaution, server implementations should consider limiting the work done by a single connection.

The Importance of Industry Collaboration

When the threat of the Rapid Reset attack became apparent, the industry collaborated to address the issue. The vulnerability was disclosed to key HTTP/2 implementers, helping to devise and distribute effective countermeasures. The vulnerability is logged against CVE-2023-44487.

HTTP/2 'Rapid Reset' DDoS attacks pose a serious risk to services using the protocol. To reduce exposure, service providers should promptly apply available software patches and updates.

Learn how Peakhour's Application Security Platform helps protect against Layer 7 DDoS attacks, including the HTTP/2 Rapid Reset vulnerability. Contact our team to secure your infrastructure.

Enterprise DDoS Protection

2023-06-19T00:00:00+10:00

Cyber threats continue to grow in complexity and volume, and Layer 7 attacks remain especially difficult to defend against¹. Each layer presents its own set of vulnerabilities for threat actors to exploit. The 7th layer, or application layer, handles application-specific communications. That makes it a useful target because modern applications are complex and varied.

Defending against Layer 7 attacks requires continuous tuning and adaptation¹. Microsoft highlighted the issue in June 2023, when it reported a traffic surge that temporarily affected the availability of some of its services¹.

Microsoft's Layer 7 DDoS Attacks

Microsoft's security team detected and tracked DDoS activity from a threat actor it called Storm-1359. The actor used a mix of resources, including multiple virtual private servers (VPS), rented cloud infrastructure, open proxies, and DDoS tools¹.

The activity did not target layers 3 or 4. It targeted layer 7, where requests can look like regular traffic and arrive from source IPs distributed around the world.

The Attack Methods

Storm-1359 used several attack types, including:

HTTP(S) Flood Attack: The attacker aimed to exhaust system resources with a high load of SSL/TLS handshakes and HTTP(S) request processing. This attack led the application backend to run out of compute resources such as CPU and memory¹.
Cache Bypass: The attacker attempted to overload the origin servers by bypassing the CDN layer¹.
Slowloris: In this case, the client opens a connection to a web server, requests a resource, such as an image, but fails to acknowledge the download or accepts it slowly. This causes the web server to keep the connection open and hold the requested resource in memory¹. Strengthening Layer 7 Protections

Microsoft mitigated the majority of disruptions by hardening its Layer 7 protections. It fine-tuned Azure Web Application Firewall (WAF) to better defend customers from the impact of similar DDoS attacks¹.

Azure Web Application Firewall, ModSecurity, and DDoS Attacks

Azure Web Application Firewall (WAF), part of Microsoft's security architecture, is built upon ModSecurity⁴, a well-established open-source Web Application Firewall (WAF) module¹. The DDoS attack Microsoft faced highlighted potential limitations in using ModSecurity, or any conventional WAF, as the primary defence mechanism against such threats.

ModSecurity's Limitations in DDoS Defence

ModSecurity is effective against a variety of web application threats, but it has limitations when dealing with DDoS attacks:

Lack of Scalability: ModSecurity is not inherently scalable. It can struggle to handle the enormous traffic volume associated with DDoS attacks.
Delayed Response: ModSecurity's rule-based approach can result in slower response times to evolving DDoS threats. While it can block threats based on established rules, it can take time to identify and create rules for new or uncommon attack patterns.
Operational Complexity: ModSecurity requires substantial expertise and constant fine-tuning to remain effective, potentially slowing down response times during a fast-paced DDoS attack.

These limitations were visible during the DDoS attack Microsoft experienced. Even though Microsoft utilised ModSecurity via Azure WAF, the time it took for Azure to respond underlines the challenge of using traditional WAFs for this class of attack¹.

The Role of Residential Proxy Networks in Layer 7 DDoS Attacks

Residential proxy networks create a specific problem in the defence against Layer 7 DDoS attacks³. These networks use IP addresses tied to physical locations, often originating from typical home or office internet connections. That makes it harder to separate legitimate traffic from malicious traffic.

Unlike traditional proxy or VPN networks, where traffic can be blocked or rate-limited based on their recognisable IP ranges, residential proxy networks blend in with legitimate users. That complicates identifying and blocking malicious requests, as any blocking or limiting measures could affect legitimate traffic from residential IPs.

A Potential Solution

In this context, fingerprinting can help distinguish between legitimate clients and malicious actors. Fingerprinting involves gathering data points from each client request, including user agent, IP address, headers, cookies, and more. The combination of these data points creates a unique 'fingerprint' for each client.

By analysing these fingerprints, it is possible to detect anomalous request patterns and potentially identify malicious clients hidden behind residential IPs. Fingerprinting can improve the accuracy of identifying malicious traffic, but it is not foolproof and should sit inside a broader, layered defence strategy.

Implementing effective fingerprinting also requires substantial technical expertise and resources. The measures need to avoid degrading user experience or breaching privacy regulations.

The Need for Specialised Rate Limiting Services

A specialised rate limiting service could have offered a faster and more effective response to the DDoS attack. Rate limiting restricts the number of requests that an IP address can make within a specific time period².

Such a service offers several advantages when defending against DDoS attacks:

Rapid Response: Rate limiting can provide a quick initial defence against a DDoS attack by immediately limiting traffic from suspicious IP addresses.
Flexibility: Rate limiting rules can be applied to factors such as IP addresses, URL, headers, response codes, and more, creating more granular defence mechanisms.
Reduced Load: By limiting the rate of requests, these services can reduce the load on the server, preserving resources for legitimate traffic.

Advanced Rate Limiting and Custom Keys

One way to defend against these attacks is through advanced rate limiting². Rate limiting restricts the number of requests an IP address, URL, or another custom key can make in a set time period. This can stop a single actor from flooding a network with traffic.

Criteria Used in Rate Limiting

Rate limits can be defined using different criteria:

IP Address
URL
Query String
Headers
Response Codes
GeoIP Information: ASN or Country Code
Parsed User Agent Information: Different rules for search engines vs. generic 'bots'
Fingerprints: TCP, TLS or H2 fingerprints can uniquely identify the connecting software
Meta Information: From bot protection service²

This allows rate limiting to 'bucket' requests using different criteria, effectively rate limiting a larger group of connections.

The Role of Anomaly Detection

Anomaly detection is another useful tool against these attacks. It identifies patterns or events that deviate from the norm and may indicate suspicious activity. Detecting those anomalies quickly can help teams respond faster, identify a suitable rate limit key and stop the potential attack.

Caching as a Mitigation Strategy

Caching is an effective mitigation strategy for Layer 7 attacks. It stores static responses to requests, reducing load on the server by serving those responses instead of processing each request individually. In a DDoS scenario, where a flood of requests is sent to the server, caching can help maintain availability. Ignoring client-provided 'Cache Control' headers such as 'max-age=0' or 'no-cache' can be effective because these headers are typically used to bypass a CDN.

Recommendations for Defence Against Layer 7 Attacks

Use anomaly detection to identify an active attack.
Use Layer 7 protection services, including rate limiting, with past 99th percentile hit rates as a starting point.
Apply bot mitigation techniques, as most Layer 7 attacks originate from bots.
Use IP reputation as an early warning sign, as many IPs have been involved in attacks before.
Block, limit, or redirect traffic from outside a defined geographic region.
Rate limit or block requests from data centre and hosting ASNs.
Create custom WAF rules to automatically block and rate limit HTTP or HTTPS attacks with known signatures.
Use effective CDN caching and ignore client-presented Cache-Control headers.

Defending against Layer 7 attacks requires several controls working together. Rate limiting, anomaly detection, and effective caching all have a role.

Peakhour's advanced rate limiting and DDoS mitigation strategies help protect applications from sophisticated Layer 7 attacks. Contact our team to strengthen your defences.

Layer 7 DDoS Protection

2023-02-07T13:00:00+11:00

In previous blog posts we've covered the benefits that full page caching has on page load performance. We also covered how caching pages lowers origin server utilisation, so the site can handle more customers. A lesser-known side benefit is protection against Layer 7 denial of service (DoS) attacks.

What is a Denial of Service attack?

The goal of a DoS attack is to make a network resource, such as a website or networked service, unavailable to users by overwhelming it with excessive traffic or requests. DoS attacks can be launched from a single source or from multiple sources (known as a distributed denial of service, or DDoS attack).

Typically, an attacker floods the web server with HTTP or API requests to try to overwhelm it. An attacker might also launch a 'slow attack', especially if they find a weak point in the application that consumes a lot of server resources for a single request. One example is repeatedly using a site search function. By slowing down the rate at which requests are sent, the attacker can bypass common rate-limiting and traffic-shaping mechanisms that are designed to block high-volume traffic spikes.

DoS conditions can also be inadvertent. A CMS like Magento or WordPress on an underpowered server can be overwhelmed, or slowed to a crawl, by so-called 'grey' bots. Examples include Semrush, Ahrefs, dotbot, and MJ12Bot. These spiders can crawl a site aggressively enough to bring it to its knees. Even Bing or Google can negatively impact a site.

What is Layer 7?

In networking parlance, Layer 7 refers to the actual application running on a web server. It is part of the OSI (Open Systems Interconnection) model used to describe the various functions in transmitting data over a network. The model is as follows:

Layer 1 -> Physical Layer, responsible for transmitting raw bits of data, e.g., a wire or wireless link
Layer 2 -> Data Link, responsible for transmitting data frames between network devices and detecting transmission errors.
Layer 3 -> Network Layer, responsible for routing data packets between networks and determining the best path for transmission.
Layer 4 -> Transport Layer, responsible for end-to-end communication between applications, providing reliable data transmission and flow control.
Layer 5 -> Session Layer, responsible for establishing, maintaining, and terminating communication sessions between applications.
Layer 6 -> Presentation Layer, responsible for formatting data to be presented to the application layer and for converting data from the application layer into a standardised format for transmission.
Layer 7 -> Application Layer, responsible for providing services to the user, such as file transfer, email, and web access.

How Full Page Caching helps

Full page caching helps by reducing the number of dynamic requests to the server. Caching the entire page allows the server to serve pre-generated content to visitors, rather than generating it dynamically each time a page is requested. This reduces the server's processing load and helps prevent excessive requests from overloading the server or making it unavailable.

A real world example

As part of a series of Peakhour recommendations to 'harden' its website against layer 7 attacks, a Peakhour client (a government site) applied limited full page caching with a time to live of 10 minutes. This means that Peakhour will serve a page from its Edge cache for 10 minutes before hitting the origin again for a new version.

Not long after implementation, that change was tested: a DDoS originating from hundreds of IPs across multiple countries hammered a set of 5 pages. The attack was spread over several days with bursts of activity lasting about 15 minutes. With the first hit, the page was cached and every subsequent hit was served from our high-performance Edge cache. No slowdown was observed and the attackers gave up.

Real attack on a Peakhour client; the spikes formed in 15 minute bursts.

Real page load times measured in the client browser

But my site is dynamic...

Many websites have a small dynamic component on the page. For ecommerce sites this might be the mini cart in the top right showing the number of items in the cart, or it might be some personalisation for a user. Often these dynamic parts of the page can be rendered in the browser using Ajax or local storage, rather than rendered on the server. By moving the dynamic components to the browser, the full page becomes cacheable. Another option is to use Edge Side Includes (ESI), which enables the majority of the page to be cached in the CDN while the dynamic parts are fetched separately before serving the full page to the user.

Peakhour can help move dynamic page components from the server to the browser and cache more at the edge. We also have a range of CMS plugins that do just that. Contact us if you want some help.

Final Thoughts

Layer 7 DoS attacks have become more common than traditional DoS attacks, as they typically require far fewer resources from the attacker. Reducing dynamic requests to the origin using full page caching is a useful but underappreciated way to mitigate them. If you're concerned about website security, resilience, and performance, Peakhour helps you cache more so you can protect, accelerate, and scale your website.

Intelligent Rate Limiting

2022-05-16T13:00:00+10:00

Rate limits protect web applications from clients making excessive requests. Peakhour.IO supports rate limits with flexible controls for selecting which clients are limited and which type of limit applies.

What kinds of attacks are stopped by rate limiting

When an application is protected with rate limiting, the main attack patterns are:

Brute force and enumeration attacks
Denial of Service (DoS) and Distributed Denial of Service (DDoS)
Site scraping
Vulnerability scanners

What else can rate limiting protect

Public APIs and authenticated APIs can be abused or misused. Sensible rate limit policies on these endpoints can reduce attack traffic and help maintain service availability. Rate limiting can protect:

APIs
Overzealous 'good bots'

How does it work?

Rate limiting focuses on a connecting client and their IP address. The following measures can be used to track client requests for rate limiting:

Concurrent connections
Connections per interval
Hits per interval
HTTP 4xx responses per interval
HTTP 5xx responses per interval
Custom criteria

How granular can rate limiting be?

Using wirefilter rules, rate limiting can identify clients from both the HTTP request and response, allowing rate limits to be separated by endpoint or behaviour. For example, the URL /api can be rate limited separately from the /login endpoint. Rate limits can also be set on response codes; for example, the endpoint /search can be protected from scraping by rate limiting clients with excessive 4xx response codes.

What types of criteria can be used to define rate limits?

Rate limits can include any information defined in an HTTP request and response, including:

IP address
URL
Query string
Headers
Response codes
GeoIP information such as ASN or country code
Parsed user agent information allowing different rules for search engines vs generic 'bots'
Metadata we make available from our BOT protection service

Defining your rate limits

Picking sensible rate limits is difficult without adequate analytics on how the web application is typically used. The Peakhour dashboard includes rate-based analytics to help with setup.

Peakhour's Application Security Platform combines high-performance delivery and cache capabilities with security controls for applications and APIs. It maintains caching performance while applying advanced threat protection. Contact our team to discuss how rate limiting can improve application performance and security posture.

Rate limiting

2022-05-16T13:00:00+10:00

Rate limits are a useful control for protecting a web application from abuse. When setting them for a web application, the key elements to consider are:

What endpoints on the web application require protecting?
Do different endpoints require separate handling?
What is the normal request rate for the entire application over a time period?
How many concurrent connections are typically used by your clients?
What errors does your API endpoint return in response to requests?

Before setting those policies, it helps to understand how rate limits protect an application from abuse or misuse, the types of attacks they can reduce, and how the rate limiting algorithm makes decisions.

What kinds of attacks are stopped by rate limiting?

When protecting an application with rate limiting, common attack scenarios include:

Brute force and enumeration attacks
Denial of Service (DoS) and Distributed Denial of Service (DDoS)
Site scraping

What else can rate limiting protect?

Public APIs and authenticated APIs can be subject to both abuse and misuse. Sensible rate limit policies can be applied on these endpoints to help prevent attacks and maintain service availability. Rate limiting can help protect these endpoints.

How does rate limiting work with user logins?

A well designed web application should allow only a limited number of failed login attempts before locking an account and requiring a password reset. This is designed to protect against brute force attacks against an account. Bots commonly attempt to brute force logins to WordPress and other popular web applications. Determined attackers can also attempt to brute force API login endpoints.

Rate limiting on a login page can be applied to the IP address of a user attempting to log in. By rate limiting by IP address, you can limit both password brute force attacks and simpler username enumeration attempts.

Using Peakhour.IO rate limiting, responses to requests can be monitored and IPs blocked for administrator-defined periods. This saves origin server resources and stops repeated attempts before they reach the application.

How could API rate limiting work?

APIs are ubiquitous across the modern web. Single Page Applications (SPAs) can be built almost entirely on REST or GraphQL APIs, while legacy applications often use form submits. Even when browsing this blog, you have consumed a range of APIs.

Because APIs are often publicly available, rate limits are commonly used to reduce abuse. Rate limiting for APIs can protect against malicious attacks. An attacker could script a bot to perform many API calls and make the service unavailable for other users, causing unplanned downtime - a layer 7 DoS or DDoS attack.

APIs

Public and private APIs can be subject to abuse or misuse. Public APIs are discoverable by anyone and can be scripted for data mining or attacks. Rate limiting these endpoints based on fair use policies is commonplace. Keeping track of this within an endpoint can be expensive, so handling it through Peakhour can offload that work from developers.

Overzealous 'good bots'

Peakhour has seen websites where up to 65% of requests come from automated bots. These bots are typically indiscriminate when mining information, and they do not carry the operational cost when your site slows down or fails. Rate limiting good bots separately from your main users helps ensure these crawlers do not stop your site from generating revenue.

How is rate limiting implemented?

Rate limiting is typically implemented using several common methods:

Fixed window

Window-based rate limiting is the simplest to understand. Fixed window limits are easy to define, such as 5,000 requests per 60 minutes. Fixed window rate limiting is subject to spikes at the edges of the window. For example, 5,000 requests in the first 5 minutes of the window may overwhelm a service.

Sliding window

A sliding window keeps much of the simplicity of a fixed window, but uses a rolling window. This allows bursts to be smoothed.

Token bucket

A token bucket is an algorithm where tokens are placed into a fixed-capacity bucket. Tokens could be defined as bytes transferred or hits to an API. When a request is considered for rate limiting, tokens are removed from the bucket. If the bucket has a sufficient quantity of tokens, the request can proceed. If there are insufficient tokens, the request is considered to be non-conforming. Non-conforming requests are dropped.

Leaky bucket

Leaky buckets are a mirror image of token buckets. Instead of removing tokens from a bucket, tokens are added to a bucket. Tokens are removed from the bucket (leaks) at a fixed rate. When a request is considered for rate limiting, it is compared to the number of tokens in the bucket. If the bucket is full, the request is considered non-conforming and is dropped.

If rate limiting is something you need to do to protect and secure your website, reach out to see how we can help.

Learn how Peakhour's Application Security Platform can improve your application's performance and security. Contact our team to get started.