Lab 03 — Detection Coverage and Data-Health Auditor

Difficulty: 3/5 | Runs locally: yes

Pairs with the Phase 09 WARMUP Chapters 6, 9 (telemetry sources, MITRE ATT&CK as a coverage map).

Why This Lab Exists (Purpose & Goal)

"Do we have enough detections?" is unanswerable by counting them — 500 noisy detections may cover less than 20 good ones. The right question is "which attacker behaviors can we detect, and which can we not?" The goal of this lab is to build the auditor that maps detections to MITRE ATT&CK techniques, exposes the gaps in behavioral coverage, and flags detections whose underlying telemetry is unhealthy — so you know where you're blind before an attacker finds out for you.

The Concept, In the Weeds

Two orthogonal failure modes the auditor catches:

Coverage gaps. Mapping detections to ATT&CK techniques turns "do we have enough?" into "we detect lateral movement and exfiltration but have no coverage for credential access." This gap analysis — prioritized by which techniques matter for your environment — is the actual value of ATT&CK; mapping for a report is not.
Data-health failures. A detection is only as good as the telemetry it keys on. If a required source has gone silent (an agent died, a log pipeline broke), the detection is silently disabled — it will never fire, and nobody knows. A detection with unhealthy telemetry is a blind spot wearing the costume of coverage, which is more dangerous than a known gap.

The audit also enforces that every detection has required sources, an owner, a runbook, and positive/ negative fixtures — the detection contract. The strategic lens (Phase 09 WARMUP, Chapter 9 — the pyramid of pain): invest detection effort at the behavior/technique level, where it's expensive for an attacker to evade, not at the hash/IP level, where they swap an indicator and walk past you.

Why This Matters for Protecting the Company

A security operations program can feel well-covered while having enormous blind spots — whole ATT&CK tactics with no detection, or detections quietly dead because their data source broke months ago. Either way, the company learns about the gap when an attacker uses it. A coverage-and-health auditor makes the blind spots visible and prioritized, so the team invests where it matters and knows when a detection has gone dark. This is how a detection program is managed as an engineering system with measurable coverage, rather than a pile of rules nobody can reason about.

Build It

Implement the auditor: map detections to ATT&CK-style techniques, required sources, owners, runbooks, and positive/negative fixtures. Report uncovered prioritized techniques and detections whose telemetry is unhealthy.

LAB_MODULE=solution pytest -q

Secure Implementation Patterns

The anti-pattern. Measuring coverage by counting detections — which hides both gaps and dead telemetry.

The secure pattern — map to ATT&CK, find gaps, check data health (mirrors solution.py):

def audit(detections, priority_techniques, healthy_sources) -> tuple[str, ...]:
    findings, covered = [], set()
    for i, det in enumerate(detections):
        covered.update(det.get("techniques", ()))                # what behaviors do we detect?
        for field in ("owner", "runbook", "positive_fixture", "negative_fixture"):
            if not det.get(field):
                findings.append(f"{i}.{field}:missing")          # enforce the detection contract
        for src in sorted(set(det.get("sources", ())) - healthy_sources):
            findings.append(f"{i}.source-unhealthy:{src}")       # a detection on dead telemetry is BLIND
    for technique in sorted(priority_techniques - covered):
        findings.append(f"uncovered:{technique}")                # the gap analysis = the real value
    return tuple(sorted(findings))

Two orthogonal failures fall out: uncovered:<technique> (a prioritized behavior nobody detects) and source-unhealthy:<src> (a detection silently disabled because its telemetry died — a blind spot wearing the costume of coverage).

Production practices to carry forward:

Map to ATT&CK techniques, not counts — turn "do we have enough?" into "which behaviors can't we see?" and prioritize by what matters for your environment.
Continuously verify telemetry health (heartbeats) so a dead source raises an alert instead of faking coverage.
Invest high on the pyramid of pain — detect tools/TTPs (expensive for an attacker to change), not hashes/IPs (swapped trivially).
Render coverage with the ATT&CK Navigator to make gaps visible to the whole team.

Validation — What You Should Be Able to Do Now

Map detection coverage to MITRE ATT&CK and turn "do we have enough detections?" into a prioritized gap analysis.
Explain why a detection with unhealthy telemetry is worse than a known gap (false sense of coverage).
Apply the pyramid of pain: invest detection at the behavior/TTP level, not at trivially-changed indicators.

The Broader Perspective

This lab teaches you to manage security coverage as a measurable, prioritized system rather than a feeling — and to be ruthlessly honest about blind spots, including the insidious ones (dead telemetry) that masquerade as coverage. The instinct to ask "where am I blind, and how would I know?" is the same one behind the range-containment "prove the negative" (Phase 00), the source-health monitoring of the event normalizer (Phase 05), and the AI-eval "known blind spots" reporting (Phase 11). Mature security isn't about how much you cover; it's about knowing precisely what you don't cover and prioritizing the gaps that matter. False confidence is the enemy.

Interview Angle

"How do you know your detection coverage is adequate?" — Not by count; map detections to ATT&CK techniques, prioritize by what matters for the environment, find the uncovered techniques, and continuously verify telemetry health so a dead source can't masquerade as coverage. Invest high on the pyramid of pain (behaviors), not on swappable indicators.

Extension (Stretch)

Visualize the coverage with the ATT&CK Navigator and wire the data-health check to live source heartbeats so a silent source raises an alert.

References

Phase 09 WARMUP Chapters 6, 9; MITRE ATT&CK and ATT&CK Navigator; "The Pyramid of Pain".

Security Engineer