Lab 02 — Purple Team Coverage Scorer

Operation Cedar Lattice — Phase 12 The purple team replay session is complete. Six FIN-LATTICE techniques were executed against Meridian Freight's production-equivalent staging environment. This lab scores the results.

Objective
Safety Boundary
Background
The Exercise Schema
API to Implement
Metric Definitions
Running the Lab
Expected Behavior
Hints

Objective

Implement seven functions that compute standard detection metrics from purple team exercise result dicts. The scorer produces precision, recall, F1, false positive rate, detection coverage, top gap rankings, and exercise summary strings from the confusion-matrix counts recorded during the session.

Safety Boundary

This lab operates entirely on Python dicts representing synthetic purple team exercise data. No exploitation, no network traffic, no payloads. The scorer is a statistical analyzer — it reads integer TP/FP/TN/FN counts and computes ratios.

Background

A purple team exercise produces a confusion matrix for each ATT&CK technique tested:

                       Reality
                  Attack | No Attack
Alert Fired:  TP (hit)  |  FP (noise)
No Alert:     FN (miss) |  TN (correct silence)

From these four counts, the scorer computes:

Recall: What fraction of real attacks did the detection catch?
Precision: Of all alerts fired, what fraction were real attacks?
F1: Harmonic mean of precision and recall.
FP Rate: What fraction of non-attack windows triggered an alert?
Detection Coverage: Mean recall across all tested techniques.

These metrics drive the remediation roadmap. Techniques with recall of 0.0 are priority-one detection gaps. Techniques with recall under 0.5 are priority-two. Techniques above 0.9 are validated.

The Exercise Schema

Each exercise result is a Python dict:

{
    "attck_id":       str,   # ATT&CK technique ID, e.g. "T1003.006"
    "attck_name":     str,   # Human-readable name, e.g. "DCSync"
    "tp_count":       int,   # True positives: alert fired AND attack was happening
    "fp_count":       int,   # False positives: alert fired BUT no attack
    "tn_count":       int,   # True negatives: no attack, no alert (correct silence)
    "fn_count":       int,   # False negatives: attack happened, alert did NOT fire
    "detection_rule": str,   # Detection rule name that fired, or "" if none
    "notes":          str,   # Freeform session notes
}

API to Implement

All seven functions live in solution.py. Implement them in lab.py.

`precision(exercise: dict) -> float | None`

Return TP / (TP + FP). Return None if TP + FP == 0 (the rule never fired — precision is undefined).

`recall(exercise: dict) -> float | None`

Return TP / (TP + FN). Return None if TP + FN == 0 (the technique was never executed — recall is undefined).

`f1(exercise: dict) -> float | None`

Return 2 * P * R / (P + R). Return None if either precision(exercise) or recall(exercise) is None, or if P + R == 0.

`detection_coverage(exercises: list) -> float`

Return the mean recall across exercises where recall(exercise) is not None. Return 0.0 if the list is empty or no recall values are computable.

`fprate(exercise: dict) -> float | None`

Return FP / (FP + TN). Return None if FP + TN == 0 (the rule never had a chance to fire on non-attack windows).

`top_gaps(exercises: list, n: int = 3) -> list[dict]`

Return the top n exercises with the lowest recall (worst detection gaps). Only include exercises where recall(exercise) is not None. Sort ascending by recall (lowest recall first). For ties, sort ascending by attck_id. Return a list of exercise dicts with a "recall" key added (the computed float recall value).

`exercise_summary(exercise: dict) -> str`

Return a one-line summary string in the format:

{attck_name} ({attck_id}): recall={recall:.0%} precision={precision:.0%} f1={f1:.2f}

If any metric is None, show "N/A" for that metric in the string. Examples:

DCSync (T1003.006): recall=0% precision=N/A f1=N/A
Valid Accounts (T1078): recall=100% precision=100% f1=1.00
Exploit Public-Facing Application (T1190): recall=N/A precision=N/A f1=N/A

Metric Definitions

Metric	Formula	Returns None when
Precision	`TP / (TP + FP)`	`TP + FP == 0` (rule never fired)
Recall	`TP / (TP + FN)`	`TP + FN == 0` (technique never executed)
F1	`2PR / (P+R)`	Either P or R is None, or P+R == 0
FP Rate	`FP / (FP + TN)`	`FP + TN == 0` (no non-attack windows)
Coverage	`mean(recall)` for non-None	0.0 if no computable recall values

Important distinction:

recall = None means "unmeasured" — the technique was not executed enough times to compute recall.
recall = 0.0 means "measured, zero coverage" — the technique was executed and the rule never fired.

Both are problems, but they require different responses: None means the measurement gap must be closed; 0.0 means the detection gap must be closed.

Running the Lab

cd lab-02-purple-team-coverage

# Install dependencies
pip install -r requirements.txt

# Run against stubs (expect NotImplementedError on all tests)
pytest -q

# Run against reference solution (expect all 11 tests to pass)
LAB_MODULE=solution pytest -q

11 passed in Xs

All 11 adversarial tests pass when run against solution.py.

Hints

For precision, recall, and fprate: check the denominator before dividing. Return None if denominator is zero.
For f1: call precision(exercise) and recall(exercise) rather than re-reading the dict — this avoids duplicating the None-check logic.
For detection_coverage: filter out exercises where recall(e) is None before computing the mean. Use the filtered list's length as the denominator.
For top_gaps: add the computed recall to each returned dict so callers do not need to recompute it. Sort by (recall_value, attck_id) to break ties alphabetically.
For exercise_summary: format each metric individually. If the metric is None, the format spec .0% cannot be applied — use "N/A" instead.

Cedar Lattice Phase 12 — Lab 02 of 02.

Red Team Engineer