Lab 01 — Detection-as-Code Correlation Engine

Difficulty: 4/5 | Runs locally: yes

Pairs with the Phase 09 WARMUP Chapters 1, 7–9 (detection & response as one loop, detection engineering, ATT&CK) and the HITCHHIKERS-GUIDE ("The Detection Development Loop").

Why This Lab Exists (Purpose & Goal)

A single suspicious event is usually noise; an attack is a sequence of events that, individually, look benign — a secret committed, an unusual login, a privileged workload created, a sensitive read. The goal of this lab is to build a correlation engine that detects the multi-stage story rather than paging on each isolated event, and to do it as code — versioned, tested, owned — rather than as a brittle ad-hoc query.

The Concept, In the Weeds

The synthetic incident is a realistic cloud-to-Kubernetes kill chain:

source-control secret alert → unusual identity login → privileged workload creation → sensitive data read

A good correlation detection has properties that are subtle to get right:

Correlate by identity/tenant within a time window — the stages must be tied to the same actor, or you'll join unrelated events into false incidents.
Tolerate out-of-order and late events — real telemetry arrives with variable latency; a detection that assumes perfect ordering misses the attack.
Suppress exact duplicates — the same event re-delivered shouldn't inflate the signal.
Require all stages — paging on any single stage produces the alert fatigue that gets detections muted. The multi-stage requirement is what makes the alert high-confidence and actionable.

The deeper principle (Phase 09 WARMUP, Chapter 7): detections must be behavioral, not brittle — and every detection ships with an owner, runbook, data dependency, false-positive strategy, and positive/ negative fixtures. A detection with none of those is noise; an alert nobody owns is worse than no alert, because it creates the illusion of coverage.

Why This Matters for Protecting the Company

Detection is half of the security mission (prevention is the other half), and it is the half that catches what prevention missed. But detection has a failure mode that is worse than silence: a flood of low-confidence single-event alerts trains analysts to ignore the console, so the real attack scrolls past unread. A correlation engine that fires on the multi-stage story — high signal, low noise, actionable — is how a security operations team actually catches breaches in progress instead of drowning in alerts. Building detections as code (tested, owned, regression-checked) is how a detection program stays reliable as the environment changes.

Build It

Implement the correlation engine for the four-stage sequence: correlate by identity/tenant within a time window, tolerate out-of-order events, suppress exact duplicates, and require all stages.

LAB_MODULE=solution pytest -q

Secure Implementation Patterns

The anti-pattern (alert fatigue). Paging on each single event drowns the real attack:

if event.kind == "unusual-login": page()      # every benign anomaly pages -> console gets muted

The secure pattern — correlate the multi-stage story per actor (mirrors solution.py):

STAGES = ("secret-alert", "unusual-login", "privileged-workload", "sensitive-read")

def correlate(events, *, window_seconds=900) -> tuple[Alert, ...]:
    unique = {e.event_id: e for e in events}                  # suppress exact-duplicate re-deliveries
    grouped = {}
    for e in unique.values():
        grouped.setdefault((e.tenant, e.identity), []).append(e)   # correlate by the SAME actor
    alerts = []
    for (tenant, identity), group in sorted(grouped.items()):
        ordered = sorted(group, key=lambda e: (e.timestamp, e.event_id))  # tolerate out-of-order arrival
        for start in range(len(ordered)):
            expected, chosen, base = 0, [], ordered[start].timestamp
            for e in ordered[start:]:
                if e.timestamp - base > window_seconds: break          # bounded time window
                if e.kind == STAGES[expected]:                          # advance the kill-chain
                    chosen.append(e); expected += 1
                    if expected == len(STAGES):                         # ALL stages -> high-confidence alert
                        alerts.append(Alert(tenant, identity, chosen[0].timestamp,
                                            chosen[-1].timestamp, tuple(c.event_id for c in chosen)))
                        break
            if expected == len(STAGES): break
    return tuple(alerts)

The signal comes from requiring the whole chain for the same actor within a window — and from tolerating real-telemetry messiness (duplicates, out-of-order). One stage is noise; the sequence is an incident.

The portable rule (Sigma), for the SIEM:

title: Unexpected privileged Kubernetes workload by a non-CI principal
logsource: {product: kubernetes, service: audit}
detection:
  sel: {verb: create, objectRef.resource: pods}
  known: {user.username|startswith: ['system:serviceaccount:ci:', 'system:serviceaccount:argocd:']}
  condition: sel and not known
level: high

Production practices to carry forward:

Behavioral, not brittle — target the invariant behavior (encoded command + unusual parent + egress), not an exact string an attacker changes in one byte.
Every detection ships the contract: owner, runbook, data dependency, false-positive strategy, and positive and negative fixtures.
Signal over volume — alert-count is a vanity metric; an analyst's attention is the scarce resource, so each alert must be actionable.
Source-health alarms so a correlation breaks loudly when a required telemetry source goes silent (a dead source masquerading as coverage is worse than a known gap).

Validation — What You Should Be Able to Do Now

Write a multi-stage correlation detection that fires on the story, not on single events.
Handle real-telemetry realities: out-of-order, late, and duplicate events, correlated by actor.
Equip every detection with owner, runbook, data dependency, FP strategy, and positive/negative fixtures — and explain why an unowned/brittle detection is worse than none.

The Broader Perspective

This lab teaches that detection quality is measured by signal, not volume — the opposite of the beginner instinct to "alert on everything." Alert-count is a vanity metric; an analyst's attention is the scarce resource, and a detection that burns it on noise is negative value. The discipline of high-confidence, owned, tested, behavioral detection is what makes the difference between a SOC that catches the breach and one that finds it in the post-incident review. It also closes the detection-and-response feedback loop: every incident teaches a new correlation to add, every false positive tunes an existing one — the program gets sharper over time only if detections are engineered, not improvised.

Interview Angle

"Design detection for unexpected Kubernetes workload creation." — Source: K8s audit logs; behavior: a workload created by an unusual principal / from an unusual image / with risky settings, correlated not single-field; with owner, runbook, data dependency, FP strategy (allowlist known controllers), and positive/negative fixtures; mapped to ATT&CK for coverage.

Extension (Stretch)

Express the rules as Sigma, deploy to a SIEM, and add source-health alerts so a correlation breaks loudly when a required telemetry source goes silent.

References

Phase 09 WARMUP Chapters 1, 7–9; Sigma rule project; MITRE ATT&CK.
David Bianco, "The Pyramid of Pain".

Security Engineer