Security Investigation and Incident Response

This tutorial teaches you how to effectively investigate security incidents using Peakhour's comprehensive logging and analytics capabilities. You'll learn to identify threats, analyze attack patterns, and respond to security events.

Duration: 45 minutes
Prerequisites: Understanding of firewall rules and log analysis
Learning Goals: Master security event investigation, threat pattern analysis, incident response workflows, and proactive threat hunting

What You'll Build: A complete security investigation methodology that enables rapid threat detection, detailed attack analysis, and effective incident response using Peakhour's security event data.

Understanding Security Event Types

Peakhour captures comprehensive security events across multiple protection layers, providing rich data for investigation and analysis.

Security Event Categories

WAF (Web Application Firewall) Events:

• SQL injection attempts
• Cross-site scripting (XSS) attacks
• Remote code execution attempts
• Path traversal attacks
• Protocol violations

Access Control Events:

• IP-based blocking (reputation, geographic)
• Rate limiting violations
• Custom firewall rule triggers
• Bot detection and challenges

Authentication Events (Edge Access):

• Login attempts and failures
• Multi-factor authentication events
• Policy evaluation results
• Session management events

Network-Level Events:

• DDoS mitigation
• Protocol-level attacks
• TLS/SSL anomalies
• Fingerprint-based detection

Set Up Your Investigation Dashboard

Access Security Analytics

1. Navigate to Analytics > Security Events
2. Select appropriate time range for investigation
3. Review the Security Posture Overview:
4. Total security events
5. Threat distribution by type
6. Geographic threat patterns
7. Action distribution (blocked, challenged, allowed)

Configure Investigation Views

Set up your analytics workspace:

Time Range Selection:

• Recent Activity: Last 24 hours for current threats
• Weekly Trends: 7 days for pattern analysis
• Monthly Overview: 30 days for baseline understanding
• Custom Range: Specific incident timeframes

Event Grouping Options:

• By IP Address: Individual attacker analysis
• By IP Class C: Network-based threat grouping
• By Country: Geographic threat analysis
• By ASN: Network provider analysis
• By WAF Rule: Attack type classification
• By URL: Target analysis

Initial Threat Assessment

Review the Threat Funnel for high-level security posture:

1. Country-Level Filtering: Geographic threat distribution
2. Threat Type Analysis: Attack category breakdown
3. Action Effectiveness: How threats were handled
4. Risk Score Calculation: Overall security risk assessment (0-100 scale)

Investigate High-Priority Threats

Identify Critical Security Events

Look for high-severity indicators:

Critical WAF Events:

• Severity: CRITICAL - Immediate investigation required
• Attack Tags: sql, rce, lfi, xss with high confidence
• Multiple Rule Matches: Coordinated attack attempts
• Successful Bypasses: Events that weren't blocked

Volume-Based Threats:

• High Request Rates: Potential DDoS or brute force
• Geographic Clustering: Coordinated attacks from specific regions
• ASN Concentration: Attacks from hosting providers or compromised networks

Drill Down into Specific Events

SQL Injection Attack Investigation

Example Investigation - SQL Injection Attack:

1. Identify the Threat:

   Event Type: WAF Block
   Severity: CRITICAL
   Rule ID: 942100
   Attack Type: SQL Injection
   Source IP: 203.0.113.45
   Country: China
   

2. Analyze Event Details:

   {
     "unique_id": "waf_20240808_143025_abc123",
     "time": "2024-08-08T14:30:25.123Z",
     "client": "203.0.113.45",
     "host": "yoursite.com", 
     "method": "POST",
     "path": "/wp-admin/admin-ajax.php",
     "query": "action=test&id=1' UNION SELECT * FROM users--",
     "geoip_country_code": "CN",
     "geoip_as_organization": "China Telecom",
     "waf": {
       "matched_rule": {
         "id": "942100", 
         "data": "union select",
         "formatted_msg": "SQL Injection Attack Detected",
         "tags": ["sql", "injection", "attack"],
         "severity": "CRITICAL"
       }
     }
   }
   

3. Request Context Analysis:

4. Target: WordPress admin functionality
5. Payload: Classic UNION-based SQL injection
6. Geographic: High-risk country origin
7. Network: Residential ISP (potential compromised host)

Investigate Attack Patterns

Look for Related Activity:

1. Same IP Analysis:
2. Filter events by client: 203.0.113.45
3. Review attack timeline and escalation

4. Check for reconnaissance attempts

5. Similar Attack Vectors:

6. Search for waf.matched_rule.tags: sql
7. Group by source IP to identify campaign scale

8. Analyze target URL patterns

9. Geographic Correlation:

10. Filter by geoip_country_code: CN
11. Look for coordinated attacks from same region
12. Check ASN distribution for botnet indicators

Analyze Attack Campaigns

Campaign Attribution

Identify Coordinated Attacks:

1. Time-Based Clustering:

   Investigation Query:
   Time Range: 2024-08-08 14:00 - 15:00 (Set in UI)
   Group By: ip_classc (Using Chart Builder)
   Filter: waf.matched_rule.tags:sql
   
   Results: 
   - 203.0.113.0/24: 45 attacks
   - 198.51.100.0/24: 23 attacks  
   - 192.0.2.0/24: 18 attacks
   

2. Attack Vector Analysis:

   Common Patterns:
   - Path: /wp-admin/*, /admin/*, /phpMyAdmin/*
   - Query: UNION SELECT variations
   - User-Agent: Python scripts, curl, automated tools
   - Timing: Rapid sequential requests (< 1 second intervals)
   

Threat Intelligence Integration

Leverage Built-in Reputation Data:

1. Check IP Reputation:
2. Review if IPs appear in $webattacks list
3. Cross-reference with $malware sources

4. Check $tor exit node involvement

5. ASN Analysis:

   High-Risk ASNs in Attack:
   - AS4134 (China Telecom): 34% of attacks
   - AS16509 (Amazon AWS): 28% of attacks
   - AS13335 (Cloudflare): 12% of attacks (likely proxied)
   

6. Geographic Threat Distribution:

   Country Breakdown:
   - China (CN): 45% of attacks
   - Russia (RU): 23% of attacks  
   - United States (US): 18% of attacks
   - Brazil (BR): 14% of attacks
   

Attack Timeline Reconstruction

Build Attack Chronology:

1. Reconnaissance Phase:

   14:25:30 - GET /robots.txt (200 OK)
   14:25:35 - GET /wp-admin/ (302 Redirect)
   14:25:38 - GET /phpinfo.php (404 Not Found)
   14:25:42 - GET /.env (404 Not Found)
   

2. Attack Escalation:

   14:26:15 - POST /wp-admin/admin-ajax.php (WAF Block - XSS)
   14:26:22 - POST /wp-admin/admin-ajax.php (WAF Block - SQL Injection)  
   14:26:28 - POST /contact.php (WAF Block - Command Injection)
   14:26:35 - Multiple rapid requests (Rate Limited)
   

3. Lateral Movement Attempts:

   14:27:10 - Subdomain scanning (admin.yoursite.com, api.yoursite.com)
   14:27:45 - Different attack vectors (LFI, RFI attempts)
   14:28:20 - Brute force login attempts
   

Advanced Log Analysis Techniques

Use Advanced Query Language

Advanced KQL Security Queries

KQL (Kibana Query Language) Queries: Use the query bar for advanced filtering. For aggregations like counting and grouping, use the Chart Builder in the UI after applying your filters.

1. Find Attack Campaigns:

   # Filter for SQL injection attempts from specific countries
   block.by:waf AND waf.matched_rule.tags:sql AND geoip_country_code:("CN" OR "RU" OR "BR")
   

   After filtering, use the Chart Builder to group by client and geoip_as_organization to find the top attackers.

2. Analyze Failed Authentication:

   # Filter for failed Edge Access authentications
   event_type:Login AND success:false
   

   To find IPs with excessive failures, use the Chart Builder to group by client_ip and count the events.

3. Detect Reconnaissance Activity:

   # Look for systematic probing of admin areas
   method:GET AND path:/wp-admin/*
   

   To identify clients with high request counts, use the Chart Builder to group by client and count events.

Custom Field Analysis

Deep Dive into Request Components:

1. User Agent Analysis:

   Suspicious Patterns:
   - Python-requests/2.25.1 (automation)
   - curl/7.68.0 (command line tools)  
   - Mozilla/5.0 (Windows NT 6.1; rv:60.0) (outdated/fake)
   - Empty or very short user agents
   

2. Header Analysis:

   "request_headers": {
     "host": "yoursite.com",
     "user-agent": "sqlmap/1.4.9",
     "accept": "*/*",
     "connection": "close",
     "content-type": "application/x-www-form-urlencoded"
   }
   

   Red Flags: Missing common headers, sqlmap signature, unusual content types

3. Payload Analysis:

   Common Attack Payloads:
   - SQL: ' UNION SELECT 1,2,3,4-- 
   - XSS: <script>alert('XSS')</script>
   - RCE: $(whoami)
   - LFI: ../../../../etc/passwd
   

Fingerprint-Based Detection

Advanced Threat Detection:

1. TLS Fingerprinting:

   Suspicious TLS Patterns:
   - Fingerprint: tls_fp_12345 (common in attack tools)
   - Missing SNI extension
   - Unusual cipher suite selection
   

2. HTTP/2 Fingerprinting:

   Attack Tool Signatures:
   - h2_fingerprint: h2_attack_tool_v1
   - Abnormal SETTINGS frame parameters
   - Missing expected HTTP/2 features
   

Real-Time Threat Hunting

Set Up Continuous Monitoring

Configure Real-Time Alerts:

1. Critical Event Monitoring:

   Alert: High-Severity WAF Events
   Query: block.by:waf AND waf.matched_rule.severity:CRITICAL
   Threshold: > 5 events in 5 minutes
   Action: Immediate notification
   

2. Geographic Anomaly Detection:

   Alert: Unusual Country Activity
   Query: geoip_country_code NOT IN ("US", "CA", "GB")
   Threshold: > 100 requests from new country
   Action: Investigation required
   

Proactive Threat Hunting

Hunt for Advanced Threats:

1. Low-and-Slow Attacks: To detect attacks that stay below rate limits, you can filter out rate-limited events and then use the UI to look for suspicious patterns.

   # Exclude rate-limited traffic
   NOT block.by:rate_limit
   

   Use the Chart Builder to group by client over various time windows to spot persistent, low-volume activity.

2. Living-off-the-Land Techniques:

   # Look for legitimate-looking requests with malicious intent
   method:GET AND path:/api/* AND user_agent:*legitimate_app* AND (query:*debug* OR query:*test*)
   

3. Session Hijacking Attempts: Detecting session anomalies often requires stateful analysis. A good starting point is to use the UI to find sessions (session_id) that are associated with multiple IP addresses (client_ip).

   Filter for event_type:AccessGranted, then use the Chart Builder to group by session_id and find the distinct count of client_ip. A count greater than 1 is suspicious.

Incident Response Workflows

Immediate Response Actions

Critical Threat Response:

1. Block Malicious IPs:

   Action: Create firewall rule
   Expression: ip.src in {203.0.113.45, 198.51.100.23}
   Action: deny
   Reason: Active attack campaign
   

2. Geographic Blocking:

   Action: Temporary country block
   Expression: ip.geoip.country in {"CN", "RU"} and 
              not (ip.src in $trusted_ips)
   Action: challenge
   Duration: 24 hours
   

Evidence Collection

Preserve Investigation Data:

1. Export Relevant Events:

   Timeframe: Attack window ± 2 hours
   Format: JSON export with full event details
   Fields: All available (headers, payloads, fingerprints)
   Retention: Store for forensic analysis
   

2. Document Attack Patterns:

   Evidence Package:
   - Attack timeline and progression
   - Source IP/ASN/Geographic data
   - Payload analysis and IOCs
   - Impact assessment and affected systems
   

Long-Term Threat Intelligence

Build Organizational Knowledge:

1. Create Custom Lists:

   List: attack_campaign_20240808_ips
   Type: IP addresses
   Content: All IPs from confirmed attack campaign
   Usage: Block future attempts from same infrastructure
   

2. Update Detection Rules:

   Rule: Detect Campaign Signature
   Expression: (http.user_agent contains "sqlmap" or 
               query contains "UNION SELECT") and
               ip.geoip.country in {"CN", "RU"}
   Action: deny
   

Incident Analysis and Reporting

Impact Assessment

Quantify Attack Impact:

1. Volume Analysis:

   Total Events: 1,247 attack attempts
   Blocked: 1,245 (99.8%)
   Challenged: 2 (0.2%)
   Bypassed: 0 (0.0%)
   

2. Resource Impact:

   Bandwidth Consumed: 15.3 MB
   Processing Time: 0.23 seconds average
   False Positive Rate: < 0.1%
   

Create Investigation Report

Security Incident Report Template

Structured Incident Documentation:

## Security Incident Report - SQL Injection Campaign

### Executive Summary
- **Incident Type**: Coordinated SQL injection attack campaign
- **Duration**: 2024-08-08 14:25 - 15:30 UTC (65 minutes)
- **Source**: China-based infrastructure (primarily AS4134)
- **Impact**: All attacks successfully blocked by WAF
- **Status**: Contained, monitoring continues

### Attack Details
- **Primary Vector**: SQL injection via WordPress admin endpoints  
- **Secondary Vectors**: XSS, command injection attempts
- **Tools Used**: Custom scripts, sqlmap signatures detected
- **Geographic Origin**: 78% China, 15% Russia, 7% other

### Response Actions
1. Real-time blocking via WAF rules
2. Enhanced monitoring of source networks
3. Temporary geographic rate limiting
4. IOC sharing with threat intelligence feeds

### Recommendations
1. Review WordPress plugin security posture
2. Implement additional admin endpoint protection
3. Consider geographic access restrictions
4. Enhance monitoring for similar campaigns

Optimize Security Posture

Learn from Attacks

Improve Detection Capabilities:

1. Enhance WAF Rules:

   New Rule: Detect Attack Tools
   Expression: http.user_agent matches ".*(sqlmap|nmap|nikto).*"
   Action: deny
   Priority: 5
   

2. Improve Rate Limiting:

   Enhanced Limits:
   - Admin endpoints: 10 requests/minute
   - API endpoints: 100 requests/minute  
   - International IPs: 50% of standard limits
   

Threat Intelligence Integration

Leverage Investigation Insights:

1. Update IP Reputation Lists:
2. Add confirmed attack IPs to blocklists
3. Share IOCs with security community

4. Monitor for infrastructure reuse

5. Behavioral Analysis:

6. Create profiles for common attack patterns
7. Implement machine learning for anomaly detection
8. Develop predictive threat models

Troubleshooting Investigation Issues

Missing Event Data

Problem

Expected security events not appearing

Solutions

• Verify time range settings and timezone
• Check if events were rate limited or sampled
• Review firewall rule priorities and matching
• Confirm logging configuration is complete

Performance During Investigation

Problem

Slow query performance on large datasets

Solutions

• Use more specific time ranges for initial analysis
• Leverage indexed fields (IP, time, event type) in filters
• Use sampling for pattern analysis on very large datasets
• Export data for offline analysis if needed

False Positive Analysis

Problem

Legitimate traffic marked as threats

Solutions

• Review user agent patterns for false positives
• Analyze geographic patterns vs. user base
• Check for overly aggressive WAF rules
• Implement allow lists for known legitimate sources

Best Practices for Security Investigation

Investigation Methodology

• Start with high-level analytics, then drill down to specifics
• Document all findings and actions taken
• Preserve evidence before taking remediation actions
• Collaborate with development teams for context on application behavior

Continuous Improvement

• Regular review of security event patterns
• Update detection rules based on new threat intelligence
• Training on emerging attack techniques and tools
• Integration with broader security operations workflows

You now have comprehensive security investigation capabilities that enable rapid threat detection, detailed analysis, and effective incident response using Peakhour's rich security event data and analytics platform.