Warmup Guide — Incident Response, Forensics, and Detection as One System

Zero-to-expert primer for Phase 09. It builds the defensive discipline from first principles: the incident-response lifecycle and incident command, evidence and the order of volatility, building a high-confidence UTC timeline, detection engineering as code (Sigma/YARA/Suricata/Zeek/osquery), MITRE ATT&CK as a coverage map, safe malware triage, and communicating uncertainty to executives. Assumes Phases 02, 05, 07. The through-line: detection and response are one system — you build the telemetry and detections in peacetime so the investigation is possible in wartime, and every incident feeds back improved controls. By the end you should be able to lead the first 30 minutes of an incident, preserve evidence while containing, and write detections that survive harmless variation.

Table of Contents

Chapter 1: Detection and Response Are One Feedback Loop

Zero background. Detection is noticing that something bad may be happening; incident response (IR) is what you do about it. Beginners treat them as separate teams; in reality they are one feedback loop: you can only respond to what you can see, you can only see what you instrumented in advance, and every incident teaches you a new detection to add. The loop:

instrument telemetry → write detections → detect → respond/investigate
   → learn → improve controls + add detections → (back to telemetry)

Why "peacetime work decides wartime outcome." When an incident hits, you cannot retroactively collect logs you never enabled. The Phase 09 capstone (a leaked dev token used to deploy an unauthorized container touching synthetic records) is only investigable because Git, identity, cloud, Kubernetes, network, Linux, and app logs were already flowing. The first half of this phase is building the detection range and logging contract precisely so the second half (the investigation) is possible.

Detection-as-code. Detections are engineering artifacts: versioned, tested with positive and negative fixtures, owned, and shipped through CI — not ad-hoc queries. "Every detection has an owner, runbook, data dependency, false-positive strategy, and test" is the bar.

Misconception to kill now. "We'll figure out logging when something happens." By then it's too late — the evidence window closed. Detection engineering is preparation, and preparation is the first phase of IR (Chapter 2).

Chapter 2: The Incident Response Lifecycle

The NIST SP 800-61 lifecycle — the canonical model you must internalize:

  1. Preparation — telemetry, detections, runbooks, roles, tabletop drills (Chapter 1). The most important phase, done before anything happens.
  2. Detection & Analysis — recognize an incident, triage severity, scope it.
  3. Containment — stop the bleeding while preserving evidence and business operation.
  4. Eradication — remove the attacker's access/footholds.
  5. Recovery — restore to known-good and monitor for return.
  6. Post-Incident Review (PIR) — learn; feed improvements back to Preparation.

The containment tension (the hardest judgment). Containment often destroys evidence (pulling a machine off the network kills volatile state; rebuilding wipes the disk). The skill — a README evaluation item — is preserving evidence while taking urgent containment action: capture volatile state before you isolate, snapshot before you rebuild, and document what you did so the investigation survives the response.

Containment must preserve business operation. A containment that takes down production to stop a low-severity incident can cause more harm than the attacker. Decisions weigh blast radius vs. operational tradeoff.

Misconception to kill now. "Rebuild the box and move on." Rebuilding before evidence collection (a flagged mistake) destroys your ability to scope the incident — you won't know what else the attacker touched. Collect, then contain/rebuild.

Chapter 3: Incident Command and Communicating Uncertainty

Incident command. A serious incident needs structure: an Incident Commander (IC) who owns decisions (not necessarily the most technical person), defined roles (investigation lead, comms, scribe), a decision log, and a severity rating that sets the response tempo. This prevents the chaos of ten people investigating in parallel and contaminating each other's evidence (Phase 00 incident conversion).

Facts vs. hypotheses vs. disproven hypotheses — the discipline that prevents wrong conclusions. Investigations go wrong when a guess hardens into a fact. Maintain three explicit buckets: what you know (with evidence IDs), what you hypothesize (to be confirmed/refuted), and what you've disproven. "Separate facts, hypotheses, and disproven hypotheses" is a core skill — announcing attribution too early (a flagged mistake) is the canonical failure.

Executive communication under uncertainty. Executives need decisions, not forensic detail. The required update format: known / unknown / actions in progress / current risk / next update time. You will not know the full blast radius early — so you communicate the bounds honestly ("we know X was accessed; we cannot yet rule out Y; we are doing Z; next update in 2 hours") rather than guessing or going silent. Communicating an uncertain blast radius well is a senior IR skill and a direct interview question.

Misconception to kill now. "Wait until we know everything to brief leadership." Silence during an incident is itself a failure; structured updates that state uncertainty maintain trust and enable business decisions.

Chapter 4: Evidence and the Order of Volatility

Order of volatility (NIST SP 800-86) — collect most-perishable first. Evidence disappears at different rates; if you collect in the wrong order, you lose the volatile stuff while grabbing the durable stuff. Roughly, from most to least volatile:

CPU registers/cache → RAM (running processes, network connections, keys)
   → network state → running processes → disk → logs/archives → physical media

So in a live incident you capture memory and live network/process state before you pull the plug or image the disk — because a reboot or shutdown vaporizes RAM, where malware, injected code, and keys often live only in memory.

Chain of custody (from Phase 00). Every artifact: stable ID, source, collector + tool/version, UTC collection time + clock source, SHA-256 hash, storage/access, and every transformation. You analyze copies, preserve originals read-only. The evidence manifest is what makes findings defensible.

Data minimization in collection. Scope identity/endpoint/cloud/data/persistence risk without unsafe collection — don't hoover up personal data you don't need (a flagged mistake and a privacy/ legal exposure). Collect what answers the investigation question.

Misconception to kill now. "Image the disk first, it's the important evidence." Disk is durable; RAM is perishable and often holds the most incriminating, ephemeral evidence (live connections, in-memory payloads, keys). Volatile first.

Chapter 5: Timelines, Clock Drift, and Confidence

The super-timeline — the core forensic artifact. An investigation reconstructs what happened when by merging events from many sources (logs, file timestamps, network) into one ordered timeline. This is how you turn scattered evidence into a narrative of the attack.

Why UTC and clock drift dominate. Events come from machines whose clocks disagree (drift) and which log in different timezones. If you merge them naively, the causal order is wrong — you might conclude the "effect" preceded the "cause." So you normalize every timestamp to UTC and account for clock skew per source. "Ignoring clock/timezone" is a flagged mistake that produces false narratives. The Forensic Timeline Normalizer lab makes you handle UTC, per-source skew, and confidence.

Confidence and alternative hypotheses. Each timeline event carries a confidence and cites an evidence ID. A log can be forged or incomplete; a timestamp can be wrong. A mature timeline notes alternative hypotheses where evidence is ambiguous (Chapter 3's facts-vs-hypotheses applied to time). "Timeline events cite evidence IDs and confidence" is a measure.

Misconception to kill now. "The logs say it happened at 14:32, so it did." Whose clock? Which timezone? Could the log be incomplete or tampered? A timestamp is evidence with a confidence, not a fact — normalize and qualify it.

Chapter 6: Telemetry Sources Across Platforms

You must know which source answers which question (Phase 05 telemetry, unified here):

  • Windows: Event Logs + Sysmon (process creation with command line + hashes, network, image loads) — the richest endpoint signal.
  • Linux: auditd / journal (syscalls, auth, file access); eBPF for richer collection.
  • macOS: Unified Log, Endpoint Security events.
  • Cloud: control-plane/API audit logs, IAM events, network flow logs, data-access logs (Phase 07).
  • Kubernetes: API audit logs (who created/changed what), workload logs.
  • Identity provider: authentication, MFA, token issuance events — often the first place a credential-abuse incident shows up.
  • Network: Zeek (protocol-aware connection/metadata logs), Suricata (signature/IDS), packet captures.

The logging contract. Before detections, you define a contract: source inventory, schemas (normalized fields), retention, and health alerts (you must be told when a source stops logging — a silent source is a blind spot an attacker exploits). The joined "user → workload → resource" timeline (Week 3) is only possible if all these sources share a normalized schema.

Misconception to kill now. "We have logs." Do you have the right logs, normalized, retained, and health-monitored? Unhealthy/missing telemetry (a lab focus) is where investigations stall.

Chapter 7: Detection Engineering from First Principles

What a detection is. A detection is a codified hypothesis about attacker behavior that fires on matching telemetry. The craft is writing detections that catch the behavior while surviving benign variation and not drowning analysts in false positives.

Behavioral, not brittle. The novice writes an exact-string rule ("alert if command line == powershell -enc <exact base64>"); the attacker changes one byte and evades it. Brittle exact-string rules are a flagged mistake. Robust detections target the invariant behavior (e.g. "PowerShell with an encoded command and network egress and spawned from Office") — harder to evade and more meaningful.

The detection contract (every rule needs all of this): an owner, a runbook (what an analyst does when it fires), a data dependency (which source/schema it needs — so a logging change that breaks it is caught), a false-positive strategy, and a test with positive (should fire) and negative (should not fire) fixtures. The Detection-as-Code Correlation Engine lab makes you build correlation (combining multiple signals into one higher-confidence detection) with these properties.

Searching for behavior, not just known indicators. "Searching only for known indicators" — IOCs (Indicators of Compromise: specific hashes, IPs, domains) — is a flagged mistake, because indicators are trivially changed by attackers. You also hunt for behaviors at the TTP level (Tactics, Techniques, and Procedures — what the attacker is trying to achieve and how they do it, independent of the specific tool or hash; Chapter 9), which generalize across campaigns.

Misconception to kill now. "More alerts = better coverage." Alert-count is a vanity metric (a flagged mistake). Coverage is measured against behaviors (ATT&CK), and each alert must be actionable (owned, with a runbook). A noisy detection that's always ignored is negative value.

Chapter 8: The Detection Languages — Sigma, YARA, Suricata, Zeek, osquery

Each tool detects at a different layer; know what each is for:

  • Sigma — a portable detection format for log events (a YAML rule that compiles to Splunk/OpenSearch/etc. queries). Write once, deploy to any SIEM (Security Information and Event Management system — the central platform that ingests, stores, correlates, and queries security logs). The lingua franca of detection-as-code.
  • YARA — pattern matching over files/memory (strings and byte patterns) — for identifying malware families. In this phase, YARA rules are written for benign lab samples only.
  • Suricata — network IDS/IPS (Intrusion Detection / Prevention System — inspects network traffic and alerts on, or inline blocks, matching signatures): signature-based detection on packet streams.
  • Zeek — network analysis: turns traffic into rich, protocol-aware connection logs (not signatures) — great for behavioral network detection and forensics.
  • osquery — query your endpoint fleet like a SQL database ("which hosts have this process/ file/listening port?") — for hunting and live inventory.

The Week-4 deliverable: 12 tested rules with positive/negative fixtures across these tools. The point is fluency in choosing the right layer: a file-hash detection → YARA, a log-pattern → Sigma, a network-protocol anomaly → Zeek, a fleet question → osquery.

Misconception to kill now. "One tool/language covers detection." Each operates on a different data layer (logs vs files vs packets vs endpoint state); real coverage spans them.

Chapter 9: MITRE ATT&CK as a Coverage Map

What ATT&CK is. The MITRE ATT&CK framework is a structured catalog of real-world adversary tactics (the why — e.g. Persistence, Privilege Escalation, Exfiltration) and techniques (the how — e.g. "Valid Accounts," "Create or Modify System Process"). It gives the industry a shared vocabulary for attacker behavior.

Why it's a coverage map, not a checklist. You map your detections to ATT&CK techniques to see where you're blind: "we detect lateral movement and exfiltration but have no coverage for credential access." This turns "do we have enough detections?" (unanswerable by count) into "which behaviors can we detect?" (answerable and prioritizable). The Detection Coverage lab audits ATT&CK-style coverage and data health.

The pyramid of pain (the strategic insight). Detections based on hashes/IPs are easy for attackers to change (low pain); detections based on tools and TTPs (behaviors) are expensive for attackers to change (high pain). Invest detection effort high on the pyramid — at the behavior/technique level — so your detections survive the attacker swapping indicators.

Misconception to kill now. "We map alerts to ATT&CK for the report." The value is the gap analysis it drives — finding the techniques you can't see and prioritizing those detections.

Chapter 10: Disk and Memory Forensics

Disk forensics. Recover what happened from persistent storage: file system metadata (MAC times — modified/accessed/changed), deleted-file recovery, browser/artifact analysis, and registry (Windows). Tools: Sleuth Kit/Autopsy. You work on a forensic image (a hashed copy), never the original.

Memory forensics — why it's powerful. RAM (captured before shutdown, Chapter 4) holds what disk doesn't: running processes and their command lines, network connections, injected code, unpacked malware, and encryption keys. Volatility parses a memory image to reconstruct the process list, network state, loaded modules, and injected regions — often the only place to find fileless/in-memory malware.

Validating a forensics claim (a README interview item). A forensics finding is a claim backed by an artifact, and you must be able to challenge it: does the evidence actually support "process X exfiltrated data," or only "process X had a network connection"? Distinguish proven from inferred (Phase 08 Chapter 6, applied to forensics) and note caveats — a process list from a compromised kernel can itself be tampered with.

Misconception to kill now. "The tool said it, so it's true." Forensic tools produce findings with caveats; an evil-maid kernel or anti-forensics can mislead them. State confidence and alternative explanations.

Chapter 11: Safe Malware Triage

The safety frame (from Phase 00). A real sample is live ordnance — detonate only in an isolated sandbox with no path to your LAN/internet/management plane, with snapshots, no shared folders/ clipboard/host creds. This phase uses inert or purpose-built samples only; you do not build malware, persistence, or destructive payloads.

Static triage (no execution): file metadata, hashes, strings, imports, packing detection, and YARA classification — answers "what is this?" without the risk of running it.

Dynamic triage (controlled execution in the sandbox): observe behavior — files created, registry/ persistence changes, network beaconing, process injection — answers "what does it do?" The output is containment guidance: indicators and behaviors defenders can detect and block.

The goal is triage, not reverse engineering. You're answering "is this malicious, what does it do, how do we detect/contain it?" — fast classification feeding detection (Chapters 8–9), not a full RE teardown (that's Phase 08 territory).

Misconception to kill now. "Run it and see." Run it only in proven isolation, after static triage, with capture started first — and never on anything with a path back to real infrastructure.

Chapter 12: Containment, Recovery, and the Post-Incident Review

Containment stops the attacker's progress while preserving evidence and operations (Chapter 2): isolate affected systems, rotate compromised credentials/keys and prove old access fails (Phase 07), block C2 (command-and-control — the channel a compromised host uses to receive attacker instructions and exfiltrate data; the periodic beaconing to it is a prime hunting signal), disable abused accounts — weighing blast radius against operational impact.

Eradication & recovery. Remove footholds (don't just patch the entry — find all the attacker's access, like variant analysis for intrusions), restore from known-good, rebuild workloads, validate, and recover under monitoring (the attacker may return; watch for it before declaring done).

The Post-Incident Review (PIR) — blameless and systemic. After recovery, a blameless review asks: what were the root cause and contributing factors, what went well/poorly, and what durable engineering and process improvements prevent recurrence. Blaming an individual instead of fixing system conditions is a flagged mistake — a person clicking a phishing link is a symptom of missing controls (MFA, egress filtering, detection), and the fix is the control. The PIR's output is improved controls + new tested detections — closing Chapter 1's loop.

Misconception to kill now. "Find who caused it." Blame stops learning. Ask what system conditions let it happen and fix those; the durable output is controls and detections, not a reprimand.

Lab Walkthrough Guidance

The labs build the detection-and-investigation backbone:

  1. Lab 03 — Detection Coverage and Data-Health Auditor. Start here: inventory sources, schemas, retention, and health (Chapter 6), and map detection coverage to ATT&CK (Chapter 9).
  2. Lab 01 — Detection-as-Code Correlation Engine. Build correlated, tested detections with owner/runbook/data-dependency/FP-strategy/fixtures (Chapters 7–8).
  3. Lab 02 — Forensic Timeline Normalizer. Merge multi-source events into a UTC super-timeline with per-source skew and confidence (Chapter 5).
  4. Lab 04 — Malware Static-Triage. Classify a sample without executing it (Chapter 11) — entropy/ packing, import-combination behavior inference (injection/persistence/network/crypto), and YARA-style rules → a verdict that feeds detections.

Generate incident behavior with benign administrative scripts and seeded artifacts only. Run:

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v

Success Criteria

You are ready for Phase 10 when you can, without notes:

  1. Run the IR lifecycle and explain the preparation→learning loop.
  2. Preserve evidence (volatile-first, chain of custody) while taking urgent containment action.
  3. Separate facts, hypotheses, and disproven hypotheses, and avoid premature attribution.
  4. Build a UTC super-timeline handling clock skew and confidence with evidence IDs.
  5. Write a detection robust to harmless variation, with owner, runbook, data dependency, FP strategy, and positive/negative tests.
  6. Map detection coverage to MITRE ATT&CK and find the gaps; explain the pyramid of pain.
  7. Triage malware safely (static then dynamic in isolation) and produce containment guidance.
  8. Brief executives with known/unknown/actions/risk/next-update under genuine uncertainty.

Common Mistakes

  • Rebuilding before evidence collection; collecting durable disk while losing volatile RAM.
  • Announcing attribution too early; hardening hypotheses into facts.
  • Searching only for known indicators (IOCs), not behaviors.
  • Ignoring clock drift/timezone in timelines.
  • Alert-count vanity metrics; brittle exact-string rules; alerts with no owner/runbook.
  • Over-collecting personal data during investigation.
  • Blaming an individual instead of fixing the system conditions that allowed the incident.

Interview Q&A

Q: Lead the first 30 minutes of a cloud credential incident. A: Declare and assign an IC and roles; set severity. Preserve evidence before containment: capture the identity-provider and cloud audit logs of the credential's activity, and any volatile state. Scope blast radius via the IAM graph (Phase 07) — what could it reach. Contain by revoking/rotating the credential and verifying old access fails, while keeping business operating. Maintain a decision log and a facts/hypotheses split, and send a known/unknown/actions/risk/next-update brief. Avoid attribution; hunt for what the credential did, not just known indicators.

Q: Investigate suspicious PowerShell without assuming maliciousness. A: Pull the Sysmon process-creation event (command line, parent process, hashes), the script-block logs, and correlated network egress. Build a small timeline: what spawned it, what it did, did it reach the network or touch sensitive files? Lots of admin PowerShell is benign — I hold "malicious" as a hypothesis, look for corroborating behavior (encoded command + unusual parent + egress), and distinguish proven from inferred before concluding.

Q: Validate a memory-forensics claim. A: Ask what artifact supports it. "Process injected code" should be backed by a specific Volatility finding (an executable private region, a hollowed process) — not just "looked weird." I'd check whether the memory image is trustworthy (could a compromised kernel hide processes?), seek a second corroborating source, and state confidence plus alternative explanations rather than treating the tool's output as ground truth.

Q: Design detection for unexpected Kubernetes workload creation. A: Source: K8s API audit logs. Behavior: a workload created by an unusual principal, from an unusual image/registry, or with risky settings (privileged, hostPath) — correlated, not a single field. Give it an owner, a runbook (verify with the service owner; if unauthorized, isolate via network policy and preserve), a data dependency on audit logging being healthy, an FP strategy (allowlist known controllers/CI identities), and positive/negative fixtures. Map it to ATT&CK (e.g. deploy container) for coverage tracking.

Q: Communicate an uncertain blast radius to executives. A: I state what we know (which credential, what it accessed per logs), what we can't yet rule out (lateral movement still being scoped), the actions in progress (rotation, hunting, containment), the current risk in business terms, and the next update time. I don't guess the full scope or go silent — bounded honesty maintains trust and lets them make business decisions.

References

Standards

  • NIST SP 800-61 Rev. 2 (Incident Handling); NIST SP 800-86 (forensic techniques, order of volatility).
  • SANS Incident Handler's Handbook; the PICERL lifecycle.

Detection

  • MITRE ATT&CK framework and the ATT&CK Navigator (coverage mapping).
  • Sigma rule specification; YARA documentation; Suricata and Zeek documentation; osquery docs.
  • David Bianco, "The Pyramid of Pain"; "Detection Engineering" maturity writings (Palantir, etc.).

Forensics and malware

  • The Art of Memory Forensics (Ligh, Case, Levy, Walters); Volatility documentation.
  • Brian Carrier, File System Forensic Analysis; Sleuth Kit/Autopsy; Timesketch; Velociraptor.
  • Practical Malware Analysis (Sikorski & Honig) — safe, lab-only triage concepts.