Hitchhiker's Guide — Building and Testing Detections

The operational companion to the Phase 09 WARMUP. The WARMUP unifies detection and response as one feedback loop (IR lifecycle, evidence/volatility, timelines, detection engineering, ATT&CK, forensics, safe triage); this guide is the how — building the end-to-end range, the detection-development loop, the investigation notebook, the credential and host drills, and the after-action evidence. Generate incident behavior with benign scripts and seeded artifacts only.

Table of Contents

• Build the End-to-End Range
• Lab Telemetry Matrix
• The Detection Development Loop
• Detection-as-Code Requirements
• Investigation Workflow and Notebook
• Credential Incident Drill
• Host / Workload Drill
• Safe Malware Triage
• Executive Update
• After-Action Evidence
• Worked Examples
• Interview Drills
• References

Build the End-to-End Range

Generate synthetic endpoint, identity, cloud, Kubernetes, network, and application events with shared correlation IDs, and route them through the same parser, enrichment, storage, query, alert, and case workflow your detections use. Include source heartbeats and canaries — you must detect when a source goes silent (Phase 09 WARMUP, Chapter 6). The investigation is only possible if this telemetry was flowing before the incident.

Lab Telemetry Matrix

The Detection Development Loop

1. Write a threat hypothesis: actor, precondition, action, target, expected evidence.
2. Identify required fields and source-health assumptions.
3. Create positive, near-miss, benign, duplicate, late, and missing-source fixtures.
4. Implement portable logic (Sigma) before vendor syntax where possible.
5. Measure precision on representative benign data (not just that it fires).
6. Attach severity, owner, triage runbook, containment options, and data dependencies.
7. Exercise the runbook and feed findings back into the rule.

Target behavior, not brittle strings (Phase 09 WARMUP, Chapter 7) — an exact-command-line rule is one byte from evasion.

Detection-as-Code Requirements

• version the query and its schema assumptions;
• include positive and negative fixtures;
• test out-of-order, duplicate, late, and missing events;
• enforce tenant/identity boundaries;
• set the time window from threat behavior and log latency;
• attach owner, severity, response, and data health. (An alert with no owner/runbook is noise.)

Investigation Workflow and Notebook

alert → validate sources → establish identity/session → scope actions
   → preserve evidence → contain credential/workload → assess data
   → rotate/rebuild → monitor → review

Notebook discipline: record query text/version, time bounds, systems searched, result count, event IDs, timezone (normalize to UTC, Phase 09 WARMUP, Chapter 5), and analyst interpretation. Preserve original evidence separately from normalized copies. Link every scope claim to evidence and label negative searches with their source-coverage limitations (facts vs hypotheses, Chapter 3). Don't let a hypothesis harden into a fact, and don't announce attribution early.

Credential Incident Drill

For a leaked token/key (the capstone scenario): identify issuer, subject, audience, privileges, issue/expiry, downstream sessions, impersonation, API actions, data reads, resource changes, and persistence. Then revoke the credential and dependent sessions, rotate related secrets, rebuild modified workloads, and monitor for continued access — and prove the old credential is rejected. Measure time from canary → alert → decision → invalidation → scope confidence.

Host / Workload Drill

Preserve volatile state when proportional (RAM before shutdown — order of volatility, Phase 09 WARMUP, Chapter 4); isolate safely; collect the process tree, executable hashes, network state, persistence, user/session, and relevant memory/disk artifacts. Compare with known-good deployment provenance. Reimage/redeploy from a trusted source rather than assuming process termination restored trust (eradicate all footholds, not just the entry).

Safe Malware Triage

Use inert or purpose-built samples in a disconnected sandbox (Phase 00 containment). Record hashes and static metadata first; observe only bounded behavior; never connect samples to real networks or credentials. The goal is detection and containment evidence, not malware development.

Executive Update

State: known facts + confidence, business exposure, action taken, key unknowns, the decision needed, and the next update time (Phase 09 WARMUP, Chapter 3). Avoid attribution or customer-impact certainty before evidence supports it — bounded honesty under uncertainty maintains trust.

After-Action Evidence

• an incident chronology with confidence and source references;
• a decision/action log with owners and rollback;
• coverage and source-health gaps;
• containment and recovery timings;
• the customer/privacy assessment basis;
• detection/runbook regressions (the loop closes here);
• corrective actions with system owner, due date, and verification — blameless and systemic (Phase 09 WARMUP, Chapter 12).

Worked Examples

A behavioral Sigma rule with fixtures

Brittle: alert on an exact base64 command. Robust: target the behavior (Phase 09 WARMUP, Ch 7):

title: Kubernetes workload created by unexpected principal
logsource: {product: kubernetes, service: audit}
detection:
  selection:
    verb: create
    objectRef.resource: pods
  filter_known:
    user.username|startswith: ['system:serviceaccount:ci:', 'system:serviceaccount:argocd:']
  condition: selection and not filter_known
level: high
# fixtures: positive = pod created by a user token; negative = pod created by the CI SA (must NOT fire)
falsepositives: [new legitimate deploy identity — add to filter_known with owner+date]

Every rule ships with an owner, runbook, data dependency (K8s audit healthy), FP strategy, and positive + negative fixtures. An alert with no owner/runbook is noise.

Normalizing a timeline across clock skew

Three sources, three clocks — naively merged, the "effect" precedes the "cause":

raw:   edge-fw   2026-06-12T14:32:10-04:00   (EDT)
       app       2026-06-12T18:32:09Z        (UTC, +1.4s drift measured)
normalized to UTC, skew-corrected:
       18:32:09.0Z  app: auth failure  (evidence E-12, confidence high)
       18:32:10.0Z  fw : egress denied (evidence E-07, confidence high)

The discipline (WARMUP Chapter 5): convert everything to UTC, apply each source's measured skew, attach an evidence ID and confidence, and note alternatives where ambiguous. "The log says 14:32" is not a fact until you've answered whose clock, which timezone.

First-30-minutes runbook (cloud credential)

0–5m   declare; assign IC + roles; set severity
5–15m  PRESERVE before contain: pull IdP + cloud audit logs of the credential's activity
15–20m scope blast radius via IAM graph (what could it reach transitively)
20–25m contain: revoke/rotate; VERIFY old access is rejected; keep business running
25–30m brief: known / unknown / actions / risk / next-update; start a facts-vs-hypotheses log

Interview Drills

1. "Lead the first 30 minutes of a cloud credential incident." (Runbook above.) Emphasize: preserve evidence before containment, scope via the IAM graph, verify old access fails, no premature attribution, structured executive update.
2. "Investigate suspicious PowerShell without assuming malice." Sysmon process-create (command line, parent, hashes) + script-block logs + correlated egress. Build a mini-timeline; lots of admin PowerShell is benign, so hold "malicious" as a hypothesis and look for corroboration (encoded cmd
   • unusual parent + egress). Separate proven from inferred.
3. "Validate a memory-forensics claim." Ask what artifact supports it — "injected code" needs a specific Volatility finding (an executable private region / hollowed process), not "looked weird." Check whether the image is trustworthy (a compromised kernel can hide processes), seek a second source, state confidence + alternatives.
4. "Design detection for unexpected K8s workload creation." (Sigma above.) Behavior not strings; correlate unusual principal/image/settings; owner + runbook + data dependency + FP strategy + fixtures; map to ATT&CK for coverage.
5. "Communicate an uncertain blast radius to executives." State known (which credential, what it accessed per logs), can't-yet-rule-out (lateral movement still scoped), actions in progress, current business risk, next update time. Don't guess the full scope or go silent — bounded honesty.

References

• NIST SP 800-61r2 (incident handling); NIST SP 800-86 (forensics, order of volatility).
• MITRE ATT&CK and ATT&CK Navigator; Sigma rule project; YARA / Suricata / Zeek / osquery docs.
• The Art of Memory Forensics (Volatility); Brian Carrier, File System Forensic Analysis.
• David Bianco, "The Pyramid of Pain"; Timesketch, Velociraptor docs.
• SANS Incident Handler's Handbook.