Peakhour.IO - Bots

The BOTS Act and the War on Ticket Scalping

2025-09-17T00:00:00+10:00

For popular concerts and events, tickets can disappear in seconds. Often the competition is not just other fans, but ticket scalping bots built to buy faster than a human can, then push those tickets into secondary markets at heavily inflated prices. The United States introduced the Better Online Ticket Sales (BOTS) Act to address that conduct. Whether it is working is less clear. Recent investigations into Ticketmaster suggest enforcement is still catching up.

Understanding the BOTS Act

Enacted in 2016, the Better Online Ticket Sales (BOTS) Act is a U.S. federal law aimed at giving consumers a fairer chance when buying tickets online. The law makes it illegal to use software, or "bots," to bypass security measures on ticket-selling websites.

Specifically, the BOTS Act prohibits:

Circumventing a security measure, access control system, or other technological control used by a ticket issuer to enforce purchasing limits.
Using a bot to purchase tickets in excess of posted limits.
Selling tickets that were acquired in violation of these rules, if the seller knew or should have known they were obtained illegally.

The aim was to stop scalpers using automated tools to buy tickets in bulk, so fans had a fairer chance of purchasing tickets at face value.

Australia's Approach

Ticket scalping bots are not only a U.S. problem. Other countries have also used legislation to protect consumers. In Australia, the state of New South Wales (NSW) became the first to outlaw ticket bots in 2017.

The legislation introduced heavy penalties, with fines up to $110,000 for corporations using bots to snap up tickets. The law also went further than the US BOTS Act by capping ticket resale prices at no more than 10% above the original price. At the time, a representative from TEG, the owner of Australia's largest ticket seller Ticketek, stated that bots accounted for up to 70% of activity on their website, showing how widespread the problem had become.

Ticketmaster Under the Microscope

Despite the BOTS Act being in place for years, ticket scalping persists. U.S. regulators have now turned their attention to the platforms themselves. In September 2025, reports emerged that the U.S. Federal Trade Commission (FTC) was investigating whether Ticketmaster, a subsidiary of Live Nation, was doing enough to comply with the BOTS Act.

The probe is reportedly examining whether the ticket seller has a financial incentive to allow resellers to circumvent its own rules. Ticketmaster denies wrongdoing and claims to block hundreds of millions of bots daily, but the investigation raises a direct question: are ticket platforms motivated to eliminate a secondary market from which they can also profit?

If the FTC pursues a case and finds Ticketmaster in violation, the penalties could be substantial. The law allows for fines of up to $53,000 per violation, which could translate to billions of dollars in penalties and put the rest of the industry on notice.

Renewed Political Pressure

Ticket scalping has also drawn direct political attention. In March 2025, President Donald Trump signed an executive order aimed at cracking down on ticket scalpers and price-gouging practices. The order specifically calls on the FTC to increase its enforcement of the BOTS Act, a piece of legislation that, despite being law since 2016, has been used to prosecute offenders only once.

The executive order also pushes for greater price transparency throughout the ticket-buying process and calls for investigations into unfair and anti-competitive practices in the secondary market. This follows similar efforts from the Biden administration, which had focused on eliminating "junk fees" and promoting all-in pricing. The bipartisan pressure points to a growing consensus that the current state of the ticketing industry is failing consumers.

The order received broad support from across the industry, with Live Nation, StubHub, and the National Independent Venue Association (NIVA) all voicing their approval. That support reflects a clear industry position: stronger enforcement of existing laws like the BOTS Act is a necessary step in protecting fans from scalpers.

Fighting on Two Fronts

While regulators are questioning Ticketmaster's efforts, the company has also used legal channels to fight bot operators. A notable case involved Prestige Entertainment, a ticket broker that Ticketmaster accused of using sophisticated bots to buy large quantities of tickets for resale.

In that lawsuit, Ticketmaster alleged that Prestige used bots and dummy accounts to bypass security measures like CAPTCHAs and purchase limits. The case produced some interesting legal arguments, including a novel copyright claim. Ticketmaster argued that in order to develop the bots, the software creators must have illegally copied code from Ticketmaster's website and app.

The legal battle ended in a settlement in 2019, with Prestige Entertainment permanently banned from using bots on Ticketmaster's platforms. The case shows that while Ticketmaster is under scrutiny itself, it has also pursued those who exploit its system. That leaves Ticketmaster fighting on two fronts: against bot operators, and now against regulators questioning whether its own enforcement is strong enough.

An Endless Cat and Mouse Game

The ticketing industry is locked in a constant technical arms race. As platforms like Ticketmaster develop more sophisticated security measures, bot creators find new ways to circumvent them. They use tools like residential proxies to mimic human users and distribute their attacks across thousands of IP addresses, making them difficult to detect.

Ticketmaster states it blocks 200 million bots a day, a fivefold increase from 2019. That figure shows the scale of the problem. The financial incentive for scalpers is large, and they are willing to invest heavily in technology to stay ahead.

For consumers, this means sold-out events and high resale prices are likely to continue. The BOTS Act provides a legal framework to punish offenders, but its effectiveness depends on enforcement and cooperation from major players like Ticketmaster. The current FTC investigation may show how much practical force the law has.

The CAPTCHA Conundrum

2025-09-01T00:00:00+10:00

Remember the distorted, wavy letters you had to decipher to prove you were not a robot? That was CAPTCHA, which stands for "Completely Automated Public Turing test to tell Computers and Humans Apart." First developed in the early 2000s, its purpose was simple: create a challenge that humans could solve, but automated scripts, or bots, would struggle with.

For years, CAPTCHA was used across common web actions, from creating an account to posting a comment. But bots have improved faster than the test, while users have become less tolerant of friction. CAPTCHA now often fails in both directions: it interrupts legitimate users and can still be solved by automated campaigns.

The User Experience Problem: Killing Conversions

The biggest issue with visible CAPTCHAs is the friction they add to the user journey. Customers expect checkout, login, and form flows to be quick. Asking them to stop and solve a puzzle creates a clear point for abandonment.

The data is hard to ignore:

A landmark Stanford University study found that adding a CAPTCHA can reduce form conversions by up to 40%.
Research from bot management firm HUMAN Security revealed that 40% of real shoppers have abandoned a purchase because of CAPTCHA frustration.
Other analyses have shown that simply adding a CAPTCHA can lead to a 3.2% higher bounce rate and an overall 3-5% drop in conversions.

For an e-commerce business, losing up to 40% of potential sales at the final checkout step is not a marginal UX issue. These numbers map directly to revenue from legitimate customers who were annoyed, delayed, or unable to solve the puzzle. The impact is even worse for users with disabilities, for whom many visual CAPTCHAs are nearly impossible to complete.

The Security Problem: A Speed Bump for Bots

While CAPTCHAs frustrate legitimate users, they are often only a small obstacle for modern bots. A market exists for bypassing them at scale.

Attackers now use automated CAPTCHA-solving services, often called "CAPTCHA farms." These services use a combination of machine learning algorithms and low-wage human workers to solve CAPTCHAs in real time for a fraction of a cent per puzzle.

An attacker using an automation tool like OpenBullet can integrate with these services via a simple API call. When the bot encounters a CAPTCHA, it sends the puzzle to the solving service and receives the solution seconds later. In many cases, these services have a higher success rate at solving CAPTCHAs than actual humans.

That breaks the original CAPTCHA model. The test designed to block bots can now create a false sense of security while actively harming the experience for real users.

The Modern Alternative: Invisible Challenges

If visible CAPTCHAs are broken, what's the alternative? Modern bot management uses invisible challenges to verify users without causing friction. Instead of actively testing the user, these systems analyse background data to distinguish humans from bots.

This is achieved through a multi-layered approach:

Behavioural Analysis: These systems track subtle indicators of human behaviour, like mouse movements, typing cadence, and touchscreen interactions. Bots, even sophisticated ones, struggle to mimic these patterns consistently.
Network and Browser Fingerprinting: By analysing hundreds of data points from the browser and network connection, these systems can identify the tell-tale signs of automation, such as the use of data centre IPs, proxy networks, or inconsistencies in the browser fingerprint.
Machine Learning: Machine learning models are trained on vast datasets of human and bot traffic. They can identify complex patterns and adapt in real time to new and evolving bot techniques.

With this approach, most legitimate users never see a challenge at all. Their journey remains uninterrupted. Only when the system detects highly suspicious activity is a challenge presented, so security controls are applied where the risk justifies the friction. For sites that depend on checkout completion, account protection, or lead capture, that is a better trade-off than showing every user another puzzle.

Key Considerations for Effective Bot Management

2025-09-01T00:00:00+10:00

Introduction

Bots account for a large share of web traffic. Recent studies estimate that nearly 50% of all internet traffic is generated by automated programs. Some bots are necessary for the web to function, such as search engine crawlers, but a significant portion are malicious. These "bad bots" are used for content scraping, credential stuffing, spam, and DDoS attacks.

As bot operators become more sophisticated, bot management needs to cover detection, classification, and response. This article outlines the main considerations for security teams protecting intellectual property, online revenue, and user accounts.

The Goal: Accurate Bot Detection and Classification

The first step in effective bot management is separating legitimate users from automated threats. Identification is not enough on its own. Security teams also need accurate classification across good, bad, and "grey" bots.

Good Bots: Support normal internet operations, such as search engine crawlers (Googlebot, Bingbot) and performance monitoring bots.
Bad Bots: Carry out malicious activity such as content scraping, account takeover, and spamming.
Grey Bots: Serve a legitimate purpose but can cause problems when they crawl too aggressively, such as SEO and marketing bots (Ahrefs, SEMrush).

Effective detection usually needs more than basic signatures. A layered approach commonly includes:

Basic Protection: Targets simple bots using user agent checks and IP reputation databases.
Intermediate Protection: Uses JavaScript-based challenges and basic network fingerprinting, such as JA3/JA4, to detect less sophisticated bots.
Advanced Protection: Combines comprehensive network fingerprinting, behavioural analysis, and machine learning to identify sophisticated bots that mimic human behaviour, use residential proxies, or rely on anti-detect browsers.

Machine learning models help in this context because they can learn from changing bot strategies and inspect incoming traffic for subtle signs of automation.

The Method: Continuously Adaptive Detection and Response

Bot behaviour changes quickly. Threat actors modify tooling, traffic patterns, and infrastructure to avoid detection, so static defence rules degrade over time. Organisations need detection and response that can adapt as the attack changes.

That means correlating metadata with behavioural factors in real time, then applying the right response for the risk. When a bot attempts account takeover or data scraping, an adaptive response can act immediately to reduce the impact.

Effective adaptive responses include:

Advanced Rate Limiting: Goes beyond simple IP-based limits by grouping requests with more stable identifiers, such as TLS/HTTP2 fingerprints or device characteristics. This helps stop distributed attacks from tools like OpenBullet that rotate through thousands of IP addresses.
Web Application Firewalls (WAF): Provide an important first line of defence by filtering harmful Layer 7 traffic based on predefined rules.
Tarpitting: Slows malicious connections to increase cost and resource consumption for attackers.
Challenges: Traditional visible CAPTCHAs can harm user experience and are often solvable by modern bots. Invisible challenges can verify a legitimate browser environment with less friction.
Alternate Content Serving: Misleads scraping bots by serving alternate or cached content with incorrect information (e.g., higher prices), making their scraped data useless.

The same response process should also feed learning loops, building a repository of bot attack patterns that can train machine learning models and improve accuracy over time.

The Expected Outcomes: A Resilient Security Posture

An adaptive bot management strategy should support several practical outcomes:

Risk Mitigation: Reduce potential financial losses, service disruption, and data breaches associated with malicious bot activity such as credential stuffing, ad fraud, and inventory hoarding.
Improved User Experience: Keep disruption low for genuine users by using invisible challenges and behavioural analysis instead of frustrating CAPTCHAs, which can reduce conversions by up to 40%.
Intellectual Property Protection: Protect valuable content, pricing data, and other intellectual property from unauthorised scraping.
Online Revenue Security: Protect online revenue streams by preventing fraud, inventory scalping, and other malicious activity that targets e-commerce platforms.
Regulatory Compliance: Help organisations meet data protection and privacy regulations with a proactive bot management approach.

Conclusion: Fortifying Against Sophisticated Bots

Modern bot defence depends on accurate detection, precise classification, and adaptive response. Machine learning, comprehensive network fingerprinting, and behavioural analysis all contribute, but they work best as part of a layered control set.

With that approach, security teams can better protect intellectual property, online revenue, and user accounts from sophisticated bot activity.

The Bot Spectrum

2025-09-01T00:00:00+10:00

The word "bot" is often used as shorthand for unwanted automation: scripts trying to break into accounts, scrape content, or overwhelm websites. A large share of internet traffic does come from bad bots, but automated traffic is not automatically harmful. Some bots are part of how the web is discovered, monitored, and kept usable.

Effective bot management is not about blocking every automated request. It depends on accurate classification: separating good bots from bad bots, and recognising the "grey" bots that sit between them. That classification lets you apply controls that reduce risk without cutting off traffic that helps your site operate.

Good Bots: The Essential Workers of the Web

Good bots are automated programs that perform useful or necessary tasks. They are usually clear about who they are and respect the rules you set in your robots.txt file. Blocking them can damage search visibility, monitoring, or other business workflows.

Examples of Good Bots:

Search Engine Crawlers: Bots like Googlebot and Bingbot are the best-known good bots. They crawl and index your website's content, which is how your pages appear in search engine results. Blocking them would make your site invisible on Google.
Performance Monitoring Bots: These bots are used by services to check your website's uptime and performance from different locations around the world, and to alert you if your site goes down.
Copyright Bots: These bots scan the web for plagiarised content, helping to protect your intellectual property.

Management Strategy: Good bots should be identified and allowed to access your site freely. Verification techniques, such as reverse DNS lookups, can be used to confirm that a bot claiming to be Googlebot is actually coming from Google.

Bad Bots: The Malicious Actors

Bad bots are designed for malicious activity. They are a major reason bot management exists as a security function. These bots are deceptive, often hiding their identity and purpose, and they can be responsible for a wide range of costly and damaging activity.

Examples of Bad Bots:

Credential Stuffers: These bots use stolen usernames and passwords to carry out account takeover attacks.
Content and Price Scrapers: These bots steal your valuable content, product listings, and pricing data, often for use by competitors.
Spam Bots: These bots flood comment sections, forums, and contact forms with unwanted ads or malicious links.
Denial of Service (DDoS) Bots: These bots are part of a botnet used to overwhelm a website with traffic, causing it to slow down or crash.
Inventory Hoarding Bots: Common in e-commerce, these bots automatically add limited-edition products to shopping carts to prevent legitimate customers from buying them, often for resale at a higher price (scalping).

Management Strategy: Bad bots need to be accurately identified and blocked as quickly as possible, ideally at the network edge before they consume your server resources.

Grey Bots: The Nuanced Category

Grey bots are not inherently malicious, but their behaviour can still cause problems. They often serve a legitimate purpose, but become an issue when they crawl too aggressively, consume excessive bandwidth or server resources, and slow the site down for real users.

Examples of Grey Bots:

Aggressive SEO Tools: Bots from marketing tools like Ahrefs, SEMrush, and Majestic crawl websites to gather data for backlink analysis and competitive research. They can be useful, but their crawling can also be heavy.
Partner and Aggregator Bots: These could be bots from partner companies or price comparison websites that need to access your data. The activity may be legitimate, but it still needs to be managed.
Feed Fetchers: Bots that collect data for news aggregators or other applications fall into this category.

Management Strategy: Grey bots require more than a simple allow or block rule. The best strategy is often to rate-limit or tarpit them.

Rate-Limiting: This allows the bot to continue accessing your site, but slows it to a manageable level so it does not overwhelm your servers.
Tarpitting: This intentionally slows the connection for a specific bot, increasing the cost and time required to crawl your site and discouraging overly aggressive behaviour.

By classifying incoming bot traffic and applying the right control for each category, organisations can block threats, manage resource consumption, and allow the useful automation the modern web depends on.

How MTU Fingerprinting Identifies VPNs and Mobile Users

2025-01-15T14:00:00+11:00

For traffic analysis, it helps to know how a user reached the service. Are they on a home network, a mobile connection, or a VPN? Deep packet inspection is invasive, but TCP handshake metadata can still carry useful context about the Maximum Transmission Unit (MTU) a connection appears to be using. By analysing those inferred MTU values, we can build "fingerprints" that point to the underlying network technology carrying the connection.

This article looks at how common technologies affect MTU values and shows how a SQL query can turn that data into useful network labels.

What is MTU and Why Does it Change?

The Maximum Transmission Unit (MTU) is the largest data packet, or frame, that a network-connected device can transmit. On standard Ethernet networks, this value is typically 1500 bytes. Larger payloads have to be split into chunks that fit that limit.

Encapsulation and Tunneling

The value starts to shift when tunnelling protocols are involved, including those used by VPNs and mobile networks. These protocols wrap the original data packet inside another packet, a process called encapsulation. The outer packet has its own headers for routing and management.

This encapsulation "steals" space from the original 1500 bytes available on the physical network. If a tunnelling protocol adds 60 bytes of headers, for example, the maximum size for the original data packet is now 1440 bytes (1500 - 60).

The Problem with Fragmentation

What happens if a device tries to send a 1500-byte packet through this 1440-byte tunnel? The packet has to be broken into smaller pieces, a process called fragmentation. It works, but it is inefficient. Fragmentation consumes CPU resources on the router performing it, adds header overhead to each fragment, and requires the receiving device to reassemble the pieces. The result is lower speed and higher latency.

To avoid that penalty, operating systems and network devices reduce the MTU of the connection to account for the tunnel's overhead. The amount of the reduction follows from the tunnelling protocol in use. That predictable drop is the basis for MTU fingerprinting.

A Guide to Common MTU Values

Different technologies add different overheads, which produces distinct MTU values.

WireGuard

WireGuard is a modern VPN known for its efficiency, but it still adds overhead.

IPv4 Overhead: 60 bytes (20-byte IPv4 header + 8-byte UDP header + 32-byte WireGuard header).
IPv6 Overhead: 80 bytes (40-byte IPv6 header + 8-byte UDP header + 32-byte WireGuard header).

On a standard 1500-byte network, that produces predictable MTU values:

1500 - 60 = 1440 bytes (WireGuard over IPv4)
1500 - 80 = 1420 bytes (WireGuard over IPv6)

There is a special case with ISPs that use DS-Lite (Dual-Stack Lite) to carry IPv4 traffic over an IPv6 network. This adds another 40-byte IPv6 header, reducing the MTU further.

1420 - 40 = 1380 bytes (WireGuard over DS-Lite)

OpenVPN

OpenVPN is another common VPN solution, but its fingerprint is less tidy. Instead of setting a static interface MTU, OpenVPN often uses a feature called mssfix. This dynamically adjusts the Maximum Segment Size (MSS) value within the TCP headers of encapsulated packets to prevent fragmentation.

The MSS is the MTU minus the IP and TCP header sizes (typically 40 bytes for IPv4). The exact MSS value, and therefore the effective MTU, depends on OpenVPN's configuration, including the transport protocol (UDP or TCP), cipher, MAC algorithm, and compression. As noted by security researcher ValdikSS, these unique MSS values can be used to fingerprint a connection with high precision. For example, a common configuration might result in an MSS of 1369, which corresponds to an effective MTU of 1409 (1369 + 40).

For general analysis, connections with an MTU around 1400 or 1380 bytes often indicate OpenVPN or other VPN usage, especially when seen with other factors.

Mobile Networks (LTE & 5G)

Mobile networks also modify MTU values. When your phone connects to the internet, its data is tunnelled through the carrier's network using the GPRS Tunnelling Protocol (GTP). This encapsulation adds its own layer of headers.

As detailed by Nick vs Networking, the typical overhead for GTP traffic over an Ethernet transport network is 50 bytes:

14 bytes for the Ethernet header
20 bytes for the outer IPv4 header
8 bytes for the UDP header
8 bytes for the GTP header

For a mobile carrier using a standard 1500-byte MTU on its transport network, the maximum MTU available to the user's device is 1450 bytes (1500 - 50).

Mobile devices don't guess this value; they are explicitly told what MTU to use by the network during the connection setup process (via Protocol Configuration Options). Mobile operators have two choices to avoid fragmentation:

Increase Transport MTU: Enable jumbo frames (for example, 1600 bytes or more) on their internal network to accommodate the 50-byte overhead and still provide a full 1500-byte MTU to the user.
Lower Advertised MTU: Advertise a lower MTU to the user's device. This is why values such as 1450 are common. Some operators may configure a more conservative MTU, such as 1300 bytes, to maintain stability across all parts of their network.

Other Common Values

Standard Ethernet: The baseline is 1500 bytes.
PPPoE: Common for DSL connections, adds 8 bytes of overhead, resulting in an MTU of 1492 bytes.
IPv6 Minimum: The IPv6 specification mandates a minimum MTU of 1280 bytes, so this value is also a significant marker.

Analysis with SQL

With this context, we can analyse network logs to classify user connections. The following SQL query buckets and attributes MTU values from a large dataset, turning raw numbers into meaningful labels.

The query works in several stages:

Extract Data: It parses the MTU from a fingerprint string in the logs.
Bucket MTUs: It uses a CASE statement to group MTUs. Specific known values, such as 1500, 1440, 1420, and 1380, go into their own buckets. Jumbo frames (>1500) are grouped into 100-byte buckets, and everything else is grouped into 20-byte buckets.
Attribute Buckets: In the final SELECT, another CASE statement translates those numeric buckets into human-readable descriptions based on the fingerprints we've identified.

The Query

-- Bucketing logic and attribution informed by research from:
-- https://ripx80.de/posts/06-wg-mtu/ (WireGuard)
-- https://medium.com/@ValdikSS/detecting-vpn-and-its-configuration-and-proxy-users-on-the-server-side-1bcc59742413 (OpenVPN)
-- https://nickvsnetworking.com/mtu-in-lte-5g-transmission-networks-part-1/ (Mobile Networks)
WITH base_data AS (
    SELECT
        toInt32OrNull(splitByChar(':', splitByChar(',', synner_fingerprint)[1])[4]) AS mtu,
        toInt32OrNull(splitByChar(':', splitByChar(',', synner_fingerprint)[1])[5]) AS wsize,
        toInt32OrNull(splitByChar(':', splitByChar(',', synner_fingerprint)[2])[1]) AS scale,
        (tls.handshake_rtt_us - tcp.min_rtt_us) >= 65000 AS is_high_latency
    FROM logs.client_logs
    WHERE time >= '2025-07-01' AND shielded = 0
),
main_aggs AS (
    SELECT
        CASE
            WHEN mtu = 1500 THEN 1500
            WHEN mtu = 1440 THEN 1440
            WHEN mtu = 1420 THEN 1420
            WHEN mtu = 1380 THEN 1380
            WHEN mtu > 1500 THEN 1501 + intDiv(mtu - 1501, 100) * 100
            ELSE intDiv(mtu, 20) * 20
        END AS mtu_bucket,
        countIf(is_high_latency) AS high_latency_count,
        countIf(not is_high_latency) AS normal_latency_count,
        round(avg(wsize * pow(2, scale))) AS avg_real_wsize
    FROM base_data
    WHERE mtu IS NOT NULL AND wsize IS NOT NULL AND scale IS NOT NULL
    GROUP BY mtu_bucket
),
top_wsizes AS (
    SELECT
        mtu_bucket,
        groupArray((wsize, cnt)) AS top_wsizes
    FROM
    (
        SELECT
            CASE
                WHEN mtu = 1500 THEN 1500
                WHEN mtu = 1440 THEN 1440
                WHEN mtu = 1420 THEN 1420
                WHEN mtu = 1380 THEN 1380
                WHEN mtu > 1500 THEN 1501 + intDiv(mtu - 1501, 100) * 100
                ELSE intDiv(mtu, 20) * 20
            END AS mtu_bucket,
            wsize,
            count() AS cnt,
            row_number() OVER (PARTITION BY mtu_bucket ORDER BY cnt DESC) AS rn
        FROM base_data
        WHERE mtu IS NOT NULL AND wsize IS NOT NULL AND scale IS NOT NULL
        GROUP BY mtu_bucket, wsize
    )
    WHERE rn <= 5
    GROUP BY mtu_bucket
),
top_scales AS (
    SELECT
        mtu_bucket,
        groupArray((scale, cnt)) AS top_scales
    FROM
    (
        SELECT
            CASE
                WHEN mtu = 1500 THEN 1500
                WHEN mtu = 1440 THEN 1440
                WHEN mtu = 1420 THEN 1420
                WHEN mtu = 1380 THEN 1380
                WHEN mtu > 1500 THEN 1501 + intDiv(mtu - 1501, 100) * 100
                ELSE intDiv(mtu, 20) * 20
            END AS mtu_bucket,
            scale,
            count() AS cnt,
            row_number() OVER (PARTITION BY mtu_bucket ORDER BY cnt DESC) AS rn
        FROM base_data
        WHERE mtu IS NOT NULL AND wsize IS NOT NULL AND scale IS NOT NULL
        GROUP BY mtu_bucket, scale
    )
    WHERE rn <= 5
    GROUP BY mtu_bucket
)
SELECT
    CASE
        WHEN mtu_bucket IN (1500, 1440, 1420, 1380) THEN toString(mtu_bucket)
        WHEN mtu_bucket > 1500 THEN concat(toString(mtu_bucket), '-', toString(mtu_bucket + 99))
        ELSE concat(toString(mtu_bucket), '-', toString(mtu_bucket + 19))
    END AS mtu_range,
    CASE
        WHEN mtu_bucket = 1500 THEN 'Standard Ethernet'
        WHEN mtu_bucket = 1480 THEN 'Likely PPPoE (e.g., 1492)'
        WHEN mtu_bucket = 1460 THEN 'Likely DS-Lite/GRE Tunnel'
        WHEN mtu_bucket = 1440 THEN 'Likely Mobile LTE/5G (e.g., 1450) / WireGuard over IPv4'
        WHEN mtu_bucket = 1420 THEN 'WireGuard over IPv6'
        WHEN mtu_bucket = 1400 THEN 'Likely OpenVPN / Mobile'
        WHEN mtu_bucket = 1380 THEN 'Likely OpenVPN / WireGuard over DS-Lite / Mobile'
        WHEN mtu_bucket = 1300 THEN 'Likely Mobile LTE/5G configured'
        WHEN mtu_bucket = 1280 THEN 'IPv6 Minimum'
        WHEN mtu_bucket > 1500 THEN 'Jumbo Frame'
        ELSE 'Other'
    END AS mtu_attribution,
    high_latency_count,
    normal_latency_count,
    round(high_latency_count / (high_latency_count + normal_latency_count), 2) AS high_latency_ratio,
    top_wsizes,
    top_scales,
    avg_real_wsize
FROM main_aggs
LEFT JOIN top_wsizes USING (mtu_bucket)
LEFT JOIN top_scales USING (mtu_bucket)
WHERE (high_latency_count + normal_latency_count) > 10000
ORDER BY mtu_bucket
LIMIT 50 FORMAT Vertical

Why Jumbo Frames Matter

Jumbo frames (MTU values greater than 1500 bytes) are a useful edge case in MTU fingerprinting. These frames, typically ranging from 9000-9216 bytes, are primarily used in high-performance computing environments, data centres, and enterprise networks where throughput optimisation is important.

When we detect jumbo frame MTUs in our analysis, they often indicate:

Enterprise Users: Corporate networks frequently enable jumbo frames for internal communications
Data Centre Traffic: Cloud services and CDNs often use jumbo frames between their infrastructure
High-Performance Applications: Video streaming, large file transfers, and backup operations can benefit from larger frame sizes
Network Misconfiguration: Jumbo frames sometimes appear because of network equipment misconfiguration

The presence of jumbo frames can help distinguish consumer and enterprise traffic, adding useful context for traffic classification and security analysis.

Practical Use Cases and Applications

MTU fingerprinting is useful across several security and operational domains:

Security Applications

VPN Detection for Compliance: Organisations can identify employees bypassing corporate network policies with personal VPNs, supporting compliance with data governance requirements.

Bot Traffic Classification: Automated traffic from residential proxy networks often shows consistent MTU patterns that differ from genuine residential users, improving bot detection.

Threat Intelligence Enhancement: Correlating MTU patterns with other indicators helps build broader threat profiles and improves attack attribution.

Network Operations

Performance Optimisation: Understanding the MTU distribution of your user base helps optimise content delivery and reduce fragmentation-related performance issues.

Infrastructure Planning: MTU analysis reveals the underlying network technologies your users employ, informing CDN placement and capacity planning decisions.

Quality of Service: Different MTU patterns correlate with connection quality, enabling proactive support for users on constrained networks.

Business Intelligence

Market Analysis: Geographic and demographic patterns in MTU distribution reveal technology adoption trends and market characteristics.

User Experience Optimisation: Identifying users on mobile or constrained networks enables adaptive content delivery and interface optimisation.

Dynamic Analysis vs Static IP Databases

MTU fingerprinting is a dynamic signal, which makes it useful alongside static IP reputation databases. It has several practical advantages:

Real-Time Adaptation

Static IP databases go stale. A residential IP address might be flagged as malicious based on historical activity, but MTU fingerprinting analyses the current network configuration. This dynamic approach captures the infrastructure being used at the moment of connection, providing more accurate and timely intelligence.

Treat the value as path evidence

An inferred MTU can narrow the possible network paths behind a connection. It cannot prove that a user is on a particular VPN, mobile carrier or proxy. Different tunnels can produce the same value, endpoints can set their own MTU, and the capture point may not see the smallest link on the complete path.

That makes MTU useful beside IP, ASN, TLS, HTTP and behavioural evidence. It can help explain why two requests that share an IP look different, or why requests from rotating IPs still appear to use the same tunnel shape. Whether it improves a decision—and whether it changes false positives—has to be measured on local traffic.

Where MTU fits

MTU fingerprinting turns TCP handshake metadata into a supporting signal, not a network identity. Record the observed TCP value, inference method and capture point so an operator can see why a label was suggested.

Start by logging the signal on one route. Compare it with known mobile, residential, corporate and VPN traffic before it affects enforcement. If the cohorts overlap too heavily, keep the value for investigation rather than forcing it into a block rule.

Anti-Detect Browsers

2025-01-15T10:00:00+11:00

Anti-detect browsers matter to defenders because they attack the assumptions behind browser trust. Many bot and fraud controls look for consistency between the browser, network, session, and behaviour. Anti-detect tooling is designed to make automated or repeated activity look more like separate ordinary browser sessions.

This article is not a guide to using those tools. The defensive question is simpler: when a browser tries to look ordinary, what can still be observed safely, and how should that evidence affect a request decision?

Why They Create Risk

Anti-detect browsers are often discussed as a browser-fingerprinting problem, but the risk is wider than that. The same traffic may also involve residential proxies, credential lists, automation, and API requests that never run browser-side checks. A login attempt, account creation flow, product scrape, checkout request, or mobile API call may look valid at the protocol level while still being part of an automated campaign.

The hard part is that some signals are genuinely ambiguous. A changed browser, a new device, a shared network, or a privacy tool does not prove abuse. A defensive system has to treat those observations as risk inputs, not as standalone verdicts.

Browser Consistency Is Evidence, Not Identity

Anti-detect tooling tries to make browser-reported attributes look internally consistent. That weakens simple checks that only ask whether the browser appears plausible. Defenders need a broader view: does the claimed browser line up with the network stack, TLS and HTTP behaviour, session history, cookie continuity, route sequence, response-code pattern, and recent account behaviour?

That does not mean a fingerprint identifies a person. Fingerprints classify software, client behaviour, and connection characteristics. They can help separate likely automation from ordinary traffic, but they need to be combined with route, account, proxy, and behavioural context. The result should be a risk classification with evidence attached, not an unexplained block.

Residential Proxies Change the Decision

Residential proxies are a common companion signal because they make requests appear to come from consumer networks. That creates a false-positive problem. Real customers also use shared residential, mobile, office, carrier-grade NAT, and public Wi-Fi networks. Blocking every suspicious or shared source would damage legitimate traffic.

The safer approach is to use residential proxy detection as one input in the decision. A proxy signal on a public content page may be logged. The same signal on repeated login failures, account creation, checkout abuse, or sensitive APIs may justify a challenge, rate limit, or block. Context changes the action.

The API Gap

Browser-side checks are weakest where the browser is not present. Mobile apps, partner integrations, token routes, and direct API clients may not expose the same JavaScript or browser evidence that a web page does. Attackers do not need a convincing browser if the target workflow accepts valid-looking API requests.

That is why anti-detect risk belongs in the wider application security model. API routes need method, schema, authentication, token, request cadence, response-code, account, and bot context. If the only signal available is an IP address, the decision will be too blunt.

Observable Signals Defenders Can Use

The useful evidence is usually the mismatch between what the request claims to be and how it behaves over time. A browser may look plausible on one request, but the wider pattern can still show automation: repeated attempts across accounts, route sequences that normal users do not follow, cache-miss pressure on expensive pages, unusual response-code loops, or browser and network characteristics that drift in ways ordinary clients rarely do.

Bot Management works best when it combines these signals rather than chasing a single magic detector. IP intelligence, proxy classification, network and browser fingerprints, route-aware rates, API state, WAF findings, and behaviour should all feed the same action vocabulary: allow, challenge, rate limit, block, log, or review.

Safer Defensive Response

The defensive response should be proportionate. High-confidence exploit traffic can be blocked quickly. Uncertain browser or proxy evidence may be better challenged, rate limited, or logged until the pattern is clearer. Sensitive routes should have tighter policy than public content. Account-impacting actions should preserve enough evidence for review.

This is especially important for support teams. If a real customer is challenged or blocked, operators need to see which signal drove the action and which route was involved. Without that record, anti-bot policy becomes a black box.

Final Thoughts

Anti-detect browsers are a practical problem because they reduce the value of simple browser checks. They do not make traffic invisible. They leave request-path evidence in network behaviour, route sequences, account activity, API usage, proxy signals, and response patterns.

The right goal is not to identify a person from a fingerprint or to block every unusual browser. The goal is to classify risk with enough context to choose a safe action at the edge, then keep the evidence available for tuning and review.

Managing Bots For Application Security

2024-09-15T00:00:00+10:00

This guide separates bot management into three maturity levels: basic, intermediate, and advanced. The point is not to rank feature lists. It is to understand what kind of bot decision each level can safely make on the request path.

Bots now target revenue, data, accounts, inventory, APIs, and origin capacity. Some are obvious crawlers. Others run credential stuffing, account creation, scraping, inventory hoarding, click fraud, or Layer 7 pressure through traffic that looks close to normal. A useful bot control has to decide whether to allow, challenge, rate limit, block, log, or review a request without punishing legitimate users who happen to share a network or device pattern.

For more on account impact, read our article on the Business Impact of Credential Stuffing.

Basic Protection

Basic bot management is built around visible signals: user-agent checks, simple IP reputation, known bot signatures, and broad rate limits. These controls still have a job. They can manage well-behaved crawlers, block obvious automation, and stop a single noisy source from hammering a site.

The limitation is that basic controls assume the source or header tells most of the story. That breaks down when automation behaves like a browser, rotates infrastructure, or spreads requests across residential and shared networks. A flat IP limit may slow one scraper and still miss a credential attack distributed across many exits. It may also catch real users behind a busy office, mobile carrier, or public Wi-Fi network.

Basic protection is suitable when the risk is low, the main concern is crawler hygiene, and the business can tolerate coarse controls. It is not enough for account protection, high-value scraping targets, or API workflows where abuse can arrive through valid requests.

Intermediate Protection

Intermediate bot management adds more request and client evidence. JavaScript signals, header consistency, cookie behaviour, basic network fingerprints such as TLS fingerprinting, and route-level observations make the decision less dependent on the IP address alone. This level can catch automation that fails to behave like a normal browser or that exposes inconsistencies across requests.

It is a meaningful step up, but it still has limits. Anti-detect browsers and modern automation can keep browser signals plausible for long enough to run a campaign. Residential proxy networks can make source reputation noisy. API requests may not run browser-side checks at all. If the bot system treats browser, route, credential, and API context as separate problems, operators end up tuning several partial controls rather than one decision.

Intermediate protection works for general scraping, noisy automation, and non-persistent abuse. It starts to struggle when attackers adapt, slow down, distribute requests, or target sensitive routes where a small number of requests can cause business harm.

Advanced Protection

Advanced bot management is combined signal decisioning. The difference is not "more techniques" in a checklist. The difference is that IP intelligence, residential proxy status, network and browser fingerprints, route-specific rates, behaviour, credential risk, API context, WAF/WAAP findings, DDoS pressure, and logs feed the same action model.

That context changes the decision. A high request rate on a public image route is not the same as repeated failed logins. A suspicious proxy signal on a cached page is not the same as the same signal on account recovery. A browser fingerprint mismatch may be logged on a low-risk page but challenged when paired with exposed credentials and rapid account switching.

Advanced protection is designed for persistent abuse: credential stuffing, account takeover attempts, scraping at scale, inventory hoarding, fake account creation, API bot traffic, and bot-driven Layer 7 floods. It should support web, mobile, and API traffic, and it should preserve evidence so security, platform, and support teams can see which signal drove an action.

Choosing the Right Level

The right level depends on what the bot can damage. A brochure site may only need crawler management and basic rate limits. An ecommerce site needs protection for search, product, checkout, promotion, and account routes. A marketplace, bank, gaming platform, ticketing site, or API-heavy business usually needs route-aware decisions that combine proxy, fingerprint, credential, account, and behaviour context.

The false-positive risk matters just as much as the attack risk. Shared networks, carrier-grade NAT, privacy tools, corporate egress, and normal browser drift can all make a simple signal look suspicious. A mature bot programme does not block every unusual request. It uses uncertainty to pick safer actions: log, challenge, rate limit, or review before escalating to a block.

Peakhour's View

Peakhour's Bot Management connects bot decisions to the rest of the application security path. Residential Proxy Detection, IP Intelligence, Advanced Rate Limiting, API Security, WAAP/WAF controls, and Log Forwarding all support the same request outcome: allow, challenge, rate limit, block, log, or review with evidence.

That is the practical maturity model. Basic controls handle obvious bots. Intermediate controls add client and request evidence. Advanced controls combine signals into decisions that match the route, risk, and business impact.

How To Exclude Query String Parameters from Search Engine crawling

2024-05-21T13:00:00+10:00

Last year we wrote about the problem of excessive crawling from search engine spiders. Search engines such as Google and Bing aim to index as much content as possible. For ecommerce sites, this often means indexing pages with query string parameters used for sorting, filtering, or pagination. Those parameters help users navigate the site, but they can cause a few predictable crawler problems:

Over-Crawling: Search engines may spend too much time crawling similar pages with different parameters, wasting crawl budget.
Duplicate Content: Pages with different parameters can be treated as duplicate content, weakening SEO performance.
Server Load: Excessive crawling can increase server load, slow down your site, and affect user experience. Search engines typically account for 30-50% of page requests to an ecommerce store. Managing their crawling effectively can have a material effect on site speed and server spend.

Another common cause of over crawling is internal searches being indexed.

In our previous article we mentioned using the webmaster tools provided by Google and Microsoft to manage crawler behaviour by adding ignored parameters. Since then, both tools have been updated and no longer allow you to add parameters to ignore during a crawl.

Differences in Crawling and Indexing

Search engines maintain an 'index' of web pages. Pages in this index are what appear in search results. To maintain the index, the search engine crawls a website to 'discover' new content and keep existing entries up to date. Webmasters can control what gets indexed with tags or headers in their web pages. These include:

Canonical Tags can be used to indicate the preferred version of a page. This helps consolidate link 'juice' and tell the search engine which URL to index.
Noindex tags can be used to prevent specific pages from being indexed. This is useful for thank you pages, admin pages or any content you don't want to appear in search results.
Nofollow links can be used to indicate to a search engine not to pass on SEO value to the linked page.

However, controlling what does or does not get indexed does not prevent content from being crawled. The only way to do that is via the robots.txt file. You may be familiar with the Disallow directive in the robots.txt file, but you can also use wildcards to prevent crawling of url parameters.

An example...

Consider an ecommerce store that has a category page which can then be customised with the following parameters:

    orderBy
    colors
    brands
    page
    results

These may appear in any order, and the combinations can result in 100s or even 1000s of variations of essentially the same page. Google is fairly smart when presented with this scenario, but Bing.... Bing can crawl very aggressively and it likes to try everything. In our example above, we may want to stop crawling everything except the page number, in which case an effective way to control crawler behaviour would be:

    User-agent: *
    Disallow: /*?*orderBy=*
    Disallow: /*?*colors=*
    Disallow: /*?*brands=*
    Disallow: /*?*results=*

We can't really do this in a single Disallow because the parameters might be in any order. By including the ? in the url we're ensuring that the parameter names are only in the query string, not in the main url path. This prevents crawlers from wasting crawl budget and putting unnecessary load on server resources.

Final Thoughts

Search engines can often make up 30-50% of the overall page requests to a website. Managing their behaviour helps maximise useful crawling and minimise server utilisation. Keep an eye on your access logs for unwanted behaviour, and use robots.txt where it gives you the right level of control.

When Bots Break Bad

2023-05-16T13:00:00+10:00

Bots account for a large share of web traffic. Recent studies put automated traffic at nearly 50% of all internet requests. Some bots are useful, such as search engine crawlers that index your site. Some are clearly harmful, such as scrapers and sneaker bots. Others sit in a grey area, including backlink and marketing bots from services such as Ahrefs and SEMrush. Even useful bots can create problems when they crawl too hard. This article looks at the main bot types and how to manage them with robots.txt and bot management tools.

Understanding the Different Types of Bots

'Good Bots'

Good bots perform legitimate work. Search engine crawlers like Googlebot and Bingbot index webpages so search results can stay current and relevant. Other examples include uptime and performance monitoring bots.

'Bad Bots'

Bad bots harm websites, users, or both. Common examples include:

Scraping content, copying and repurposing data from websites.
Sneaker bots, automatically purchasing limited-edition products (like sneakers) before human users can.
Spam bots, posting unsolicited messages and advertisements in comment sections or forums.
Vulnerability Scanners, trying thousands of website URLs to find security vulnerabilities.
Account Takeover, attempting to gain access to existing user/admin accounts using either credential stuffing or brute-force attacks.

'Grey Bots'

Grey bots sit between good and bad. They often serve a useful purpose and may follow crawling directives in robots.txt, but they can still cause problems when they crawl too aggressively. Common examples include:

AhrefsBot: A backlink analysis bot used by Ahrefs, an SEO tool.
SEMrushBot: A bot used by SEMrush, another popular SEO and digital marketing tool.
MJ12bot: A bot used by Majestic, a service that provides backlink data and analysis.
ScreamingFrog: An SEO analyser run from a local desktop.

When Grey bots (and even Good Bots) go bad.

Left unattended, grey bots can create practical problems:

Slow page loading times, which affect user experience.
Strain on server resources, potentially causing crashes, downtime, and higher costs.
Distorted website analytics, when bot traffic is mistaken for human traffic.

Managing Grey Bots with Robots.txt

The robots.txt file is a simple text file that tells web crawlers which parts of your site they can or cannot access. You can use it to manage bot behaviour and protect your website from aggressive crawling. Useful controls include:

Disallowing specific bots: You can block specific bots from accessing your site by adding a "User-agent" and "Disallow" directive to your robots.txt file. For example:

User-agent: AhrefsBot
Disallow: /

Limiting crawl rate: You can ask bots to slow down their crawling by adding a "Crawl-delay" directive:

User-agent: SEMrushBot
Crawl-delay: 10

Not all bots will follow robots.txt. ScreamingFrog, for example, can be instructed to ignore robots.txt and crawl a site as quickly as possible. You would not want a competitor doing this to your site.

Bot Management Tools

In addition to robots.txt, bot management tools (like those provided by Peakhour) can protect your website from abusive bots. Good bot management tools automatically block most unwanted traffic using a combination of Threat Intelligence, Fingerprinting techniques, Reverse DNS verification, and Header Inspection.

Advanced techniques like rate limiting and machine learning can help identify more sophisticated bad bots.

Search Bots and Double Crawling

Search bots like Bingbot can sometimes blindly follow links and crawl the same page multiple times due to different URL parameters. This double, triple, or worse crawling can increase server load and make indexing less efficient. eCommerce sites are especially exposed because product catalogues often have several filtering paths. We've seen Bing go haywire on a number of sites. Most recently, it was issuing around 50,000 requests per day to the search function of a Magento 2 store while cycling through parameters. This dropped to 2-3k requests per day when fixed. On another store, Bing was responsible for nearly half of all page requests (40k page requests) on a busy OpenCart store. Configuring it to ignore parameters dropped this to around 4k per day.

Control the crawl path that exists now

Google retired Search Console's URL Parameters tool in 2022. Bing removed its equivalent control too. Do not follow old instructions that send you hunting through either webmaster console.

Google says its crawler now learns how URL parameters behave automatically. When that is not enough, use robots.txt rules, canonical URLs and clean internal links to make the intended crawl path clear. Our worked example shows how to exclude query-string combinations with robots.txt. Google's own notice explains why it retired the URL Parameters tool.

Check the result in access logs. The useful measure is not whether a rule exists; it is whether duplicate parameter combinations stop consuming requests while the pages you want indexed remain crawlable.

Watch the request volume

When good or grey bots crawl too aggressively, they can cause the same operational problems as malicious bots: overloaded servers, slower pages, and worse user experience. Monitor crawler request volume and server load, set narrow robots.txt rules, then verify the change in logs rather than assuming the crawler behaved as intended.

Why Manage Bots?

2020-11-30T13:00:00+11:00

Modern sophisticated bad bots often work around traditional security controls. They disrupt websites, mobile applications, and APIs. Malicious bot tactics include scraping user and pricing data, creating fake accounts, running advertising click fraud, exhausting online inventories, and taking websites offline with automated DDoS attacks.

About one-quarter of all website traffic in 2019 originated from bad bots, an increase of 18 percent over 2018. Advanced persistent bots (APBs) made up seventy-five percent of that bad bot traffic as they attempted to evade detection by cycling through random IP addresses, using anonymous/residential proxies, and changing their identities (user agent). The industries hit hardest by bad bots in 2019 included financial services, education, ecommerce, and government as well as media and airlines.

“Bot attack campaigns have become big business for threat actors, and major organizations are now fighting to support legitimate users and prospects while keeping attackers out of online applications and services,” says Paula Musich, Research Director, Enterprise Management Associates.

Bots have moved from simple scripts to distributed networks of automated agents that can mimic human interactions with machine learning techniques. They can avoid detection by network security technologies that have not kept pace with the way automated agents now operate.

Reducing the damage from bad bots means using security countermeasures that detect automated traffic and make attacks uneconomic, not just visible.

Bot Countermeasure Best Practices:

The following bad bot countermeasure practices cover network security, machine learning, and behavioural analysis. The aim is to reduce the economic harm that malicious bots inflict on businesses and end-users.

Web Application Firewalls

Web Application Firewalls (WAF) are a common first line of defence that filter out harmful Layer 7 web application (HTTP) traffic using rules or policies that protect organisations against Distributed Denial of Service (DDoS) bot attacks. WAFs also protect against cross-site forgery, cross-site-scripting (XSS), file inclusion, and SQL injection attacks. A WAF is considered a reverse proxy that protects servers and can be deployed as an appliance, server plug‑in, or filter, and customised by application type or use case. WAF rules can be updated or changed based on the type of bot attack.

IP Tracking and Reputation

Sophisticated bots can be detected with network forensics by inspecting web traffic and assessing whether requests come from actual users or bad bots. Requests can be analysed using data sources including Tor/proxy IPs, IP addresses, IP geo-location information, ISP information, and IP owners. Additional sources for real-time and near-time malicious IP threat data can come from network data, CERTs, MITRE and cooperating competitors.

Client/Device Fingerprinting

Fingerprinting attempts to identify devices, including PCs, Internet of Things (IoT) devices, mobile devices and servers, using data attributes that create real-time risk profiles to stop bot attacks. Using web page access data, a bot detection fingerprinting engine generates unique fingerprints for each end-user device and checks them against bad bots that use evasion techniques, including dynamic IP addresses and anonymous web proxies.

Machine Learning

Artificial Intelligence (AI) and machine learning algorithms are increasingly used to analyse malicious bot activity and make mitigation recommendations using data from sources such as user activity history, behavioural patterns and meta-data. Machine learning can use custom-tailored algorithms to target bots and iteratively process user data and identities to discern emerging bot attack patterns from very large amounts of real-time information.

Tarpitting

Tarpitting is a bot countermeasure that delays and slows down incoming malicious traffic from suspect connections. The technique is used to increase the financial and resource costs of bot attacks in an attempt to discourage malicious actors. Bad bot tar pits can delay bot request responses or take the bad bot IP address attack source offline completely. Innovative tarpitting techniques include requiring bad bots to solve computationally complex maths challenges to access resources or websites, thereby slowing down or stopping bot activity.

User Behavior Analysis

User interaction behaviour and identifying characteristics on a web page or mobile app differ from the behaviour of an automated malicious bot. Factors such as number of pages visited per session, time spent on each web page or within a mobile app and repeat visit frequency all help differentiate authentic users from bad bots. Defeating bad bots using Behavior Analysis involves creating a user model for individual sites with historical visitor data, then checking for anomalies that may indicate bad bot activity.

Intent-based Deep Behavior Analysis (IDBA)

Compared with Behavior Analysis, Intent-based Deep Behavior Analysis (IDBA) conducts behavioural analysis at the user intent level rather than the commonly used interaction-based behaviour analysis. IDBA consists of intent encoding, intent analysis, and adaptive learning. It also employs machine learning techniques to detect bad bots emulating on-site human behaviour interactions. Bad bot mitigation techniques include limiting attempts on login pages, web authentication pages and API call authentication pages.

Rate Limiting

Rate Limiting mitigates bad bots and DDoS attacks by restricting the amount of incoming traffic accepted by specific applications and API endpoints using pre-defined bandwidth limitation policies. Web applications, GET versus POST requests, APIs that receive queries, and login credentials can all be blocked if clients, IP addresses or IP and user-agent pairs violate Rate Limiting rules. Intellectual property scraping can also be protected by Rate Limiting policies that restrict repeated image or digital downloads.

Javascript Injection

JavaScript Injection techniques can help mitigate bad bot attacks in several ways. Scripts can be placed into web applications that “fingerprint” a user’s browser to distinguish humans versus bad bots emulating “human-like” mouse movements, keystrokes or clicks. Fingerprinting detection may also involve user agent identification, HTML5 canvas and audio fingerprinting, and protocol-level fingerprinting with TLS and HTTP2. JavaScript combined with browser cookies can also be used to identify anomalous behaviour from unwanted traffic or bad bots trending over time.

ANYCast DDoS Mitigation

Anycast is an IP addressing method that routes incoming traffic requests to the nearest location or “node.” Using ANYCast for selective routing enables network load resilience against DDoS attacks by routing high traffic across multiple servers and data centres. This prevents network resources from becoming overwhelmed with malicious or irrelevant traffic.

Alternative Content Serving

Serving Alternate and Cached Content when a bad bot is detected gives organisations a way to mislead bots without blocking them altogether. For instance, e-commerce sites may fool price scraping bots by serving alternative web pages that look like legitimate pages but with higher prices. Serving Cached Content when a bot is detected also minimises load on servers without affecting site performance.

Challenges

Requests from suspected bots can be redirected to Challenges or puzzles such as a CAPTCHA, also known as a Completely Automated Public Turing test, to help identify a bad bot versus a human. Online puzzles, such as letter matching, are easy for humans to solve but difficult for automated bots. reCAPTCHA, offered free from Google, is an advanced version of CAPTCHA puzzles that require users to identify text from real-world images such as street address signs, printed books or text from paper newspapers.

Final Thoughts

Bad bots hijack user accounts, create fake accounts, scrape websites for data and personal information, flood websites with traffic through automated distributed denial of service attacks and attack public-facing APIs using constantly changing techniques. They hide behind dynamic IP addresses, change their attack signatures, mimic human behaviours, and take over vast networks of hosts and IoT devices, creating zombie machines that distribute malware across the internet. Countermeasures ranging from Web Application Firewalls to sophisticated Machine Learning algorithms form an organisation's primary line of defence against bad bots.

Malicious Bot Threats

2020-08-12T13:00:00+10:00

Bots are software applications that automate repetitive tasks without human interaction. They have become part of the normal infrastructure of the internet. Some bots are useful; others are bad bots. The latter are the concern for application and security teams.

Bad bots keep changing and are increasingly difficult to detect. They can cause significant financial damage to organisations by disrupting online operations, overwhelming websites with traffic, and stealing information such as web content and ecommerce pricing data.

Bad Bot Types

Bad bots span a wide range of attack capabilities and scenarios. The following are the main categories these attacks fall into:

Spam Bots

Spam bots typically target blog comment sections, community portals and lead generation forms with 'garbage' or fake content. They can also insert unwanted ads, malicious phishing links and banners into real-time conversations to disrupt the service and attack users.

Scraping Bots

Price, content and inventory scraping bots steal prices and product listings. This can damage an ecommerce site's revenue stream and harm SEO rankings when duplicate content appears on competitor and bogus sites. These bots also scrape product reviews, news, product catalogues and user-generated content. Scraper bots can harvest email addresses, images and text from victim websites, then repurpose that material to pose as legitimate web pages.

Credential Stuffing Bots

Credential Stuffing Bots attempt to use login details from other sites, or run brute force guessing attacks against customer and admin accounts. If successful, they can make purchases, harvest personal information and purchase histories, make unauthorised cryptocurrency transactions, and transfer reward points and money to gift cards and air miles.

Ad Click Fraud Bots

Ad Click Fraud Bots can sabotage competitors by clicking on their ads to drive costs up and exhaust budget caps. They can also be used to scam advertisers with fake websites and ad clicks that pay the fraudster directly. In both scenarios, bots automatically generate interactions or 'clicks' with ads, promotions and media.

Credit Card Stuffing Bots

Carding bots make repeated attempts to authorise stolen credit card credentials. This can leave merchant payment processors with chargebacks and penalties, and may ultimately result in the victim merchant being prevented from accepting credit cards altogether.

Inventory Denial Bots

Cart Abandonment and Inventory Exhaustion bots automatically add hundreds of products to ecommerce shopping carts, then abandon them. This can block consumers from buying products, reduce sales, manipulate conversion rates and damage a brand’s reputation.

DDoS Bots and Botnets

Distributed Denial of Service (DDoS) attack bots and botnets are made up of thousands of compromised computers or Internet of Things (IoT) devices called "zombies". They can slow down a website or take it offline completely by flooding sites with massive amounts of artificially generated traffic. Researchers have found cybercriminals advertising DDoS services on the dark web with basic fees to attack unprotected sites ranging from $50 to $100, while an attack on a protected site can reach $400 or more.

Ticket Scalping Bots

Ticket scalping bots automatically buy tickets, enabling malicious users to resell them at a higher price. Examples include using a bot to purchase concert tickets for major events the minute they go on sale.

Fake Account Creation Bots

Fake Account Creation bots create fake accounts for criminal activities such as content spam, cryptocurrency laundering and malware distribution. Fake accounts can compromise brands and attack users with malware such as ransomware.

Hacker Bots

Hacker bots can distribute malware, attack websites and compromise entire networks by exploiting security vulnerabilities and injecting code into victim sites. Hacker bots can also perform DDoS attacks across web proxies with browser-like signatures to disrupt business operations.

Impersonator Bots

Impersonator bots copy human computer interactions and behaviours to fool users and bot mitigation defences while they conduct malicious activity. Impersonator bots also include propaganda bots that influence political opinions on platforms such as Facebook and Twitter. According to researchers at the University of Southern California who studied bot use during the 2016 U.S. Presidential election, “the presence of social media bots can indeed negatively affect democratic political discussion rather than improving it, which in turn can potentially alter public opinion.”

The Growing Threat

A report from Imperva found that roughly one-quarter of all website traffic in 2019 originated from bad bots, an increase of 18% over 2018. 75% of that bad bot traffic is made up by Advanced persistent bots (APBs) that attempt to evade detection by cycling through random IP addresses, using anonymous proxies, and changing their identities. The industries hardest hit by bad bots in 2019 included financial services, education, ecommerce and government, as well as media and airlines.

Companies offering "Bad Bots as-a-Service"* are also gaining ground. These data scraping services sell bots as easy-to-use packaged products that provide pricing and competitive intelligence, alternative data for finance, or competitive insights managed by Web Data Extraction Specialists and Data Scraping Specialists.

Malicious bot-for-hire services also offer personal and financial data harvesting, brute-force login services, ad click fraud, spamming services, transaction fraud services, and Distributed Denial of Service (DDoS) attacks.

Final Thoughts

Bad bot activity continues to increase, so websites need security controls that can identify and stop them. Our next article on bots will go over the common countermeasures used to combat bad bots.