System Design: Purple Team Exercise Tracking & Detection-Gap Platform
1. Problem Statement
Design a platform that tracks purple team exercises — sessions where red and blue teams work together to test whether detection rules fire on specific ATT&CK techniques — and produces a prioritized detection-gap report with recommended actions.
Functional requirements:
- Track exercise sessions: date, scope, red team operator, blue team analyst, techniques tested
- For each technique execution: record TP/FP/TN/FN counts from the detection rule during the session
- Compute precision, recall, F1 per technique; aggregate across a session
- Track improvement over time: if the same technique is tested in two sessions 90 days apart, show recall improvement
- Identify top detection gaps: techniques with lowest recall, sorted by threat-actor prevalence and business risk
- Generate ATT&CK Navigator layer: color-code by recall score (green = covered, red = gap)
- Recommend next actions per gap: which Sigma rule to deploy, which data source to enable
Non-functional requirements:
- Multi-engagement: one team runs multiple sessions over time
- Export: generate an ATT&CK Navigator JSON layer file for each session
- Audit: all session results are immutable after the blue team analyst signs off
- API-first: CLI tool and Slack bot can post session results without logging into the web UI
2. Constraints
In scope:
- Exercise session data model
- Metric computation (precision/recall/F1)
- Gap prioritization model
- ATT&CK Navigator export
- API design
Out of scope:
- Red team tooling integration (automatic detection of TP/FP events)
- SIEM integration (pulling alert data automatically)
- ATT&CK content management (the ATT&CK matrix itself is an external data source)
3. High-Level Architecture
┌──────────────────────────────────────────────────────────────────────────┐
│ Input channels │
│ Web UI (blue team analyst enters results manually) │
│ REST API (CLI tool, Slack bot, automated SIEM export) │
└───────────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Session Ingestion Service │
│ - Validate ATT&CK technique ID exists │
│ - Validate TP+FP+TN+FN counts are non-negative integers │
│ - Store raw results │
└───────────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Metric Computation Engine │
│ - precision = TP / (TP + FP) │
│ - recall = TP / (TP + FN) │
│ - F1 = 2 × P × R / (P + R) │
│ - Aggregated across techniques per session │
│ - Trend: compare same technique across sessions │
└───────────────────────────────┬──────────────────────────────────────────┘
│
┌──────────┴───────────┐
│ │
▼ ▼
┌──────────────────────┐ ┌─────────────────────────┐
│ Gap Prioritizer │ │ ATT&CK Navigator │
│ - Sort by recall │ │ Export Generator │
│ - Weight by threat │ │ - JSON layer file │
│ actor prevalence │ │ - Color by recall │
│ - Recommend action │ │ score │
└────────────┬─────────┘ └────────────┬────────────┘
│ │
└────────────┬────────────┘
▼
┌─────────────────────────┐
│ Report Service │
│ - Session summary PDF │
│ - Gap report markdown │
│ - Executive dashboard │
└─────────────────────────┘
4. Component Deep-Dives
4.1 Session Data Model
CREATE TABLE exercise_sessions (
id UUID PRIMARY KEY,
engagement_id UUID,
session_date DATE NOT NULL,
red_operator TEXT NOT NULL,
blue_analyst TEXT NOT NULL,
scope_notes TEXT,
signed_off_at TIMESTAMPTZ, -- blue analyst sign-off; results immutable after this
signed_off_by TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE technique_results (
id UUID PRIMARY KEY,
session_id UUID REFERENCES exercise_sessions(id),
attck_id TEXT NOT NULL, -- e.g., "T1055.003"
attck_name TEXT NOT NULL,
tp_count INT NOT NULL CHECK (tp_count >= 0),
fp_count INT NOT NULL CHECK (fp_count >= 0),
tn_count INT NOT NULL CHECK (tn_count >= 0),
fn_count INT NOT NULL CHECK (fn_count >= 0),
detection_rule TEXT, -- Sigma rule ID or SIEM rule name
notes TEXT, -- operator notes on the execution
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE technique_recommendations (
attck_id TEXT PRIMARY KEY,
data_source TEXT NOT NULL, -- e.g., "sysmon_eid_8"
sigma_rule_ref TEXT, -- public Sigma rule link or local ID
effort_days INT, -- estimated deployment effort
detection_notes TEXT
);
4.2 Metric Computation
Metrics are computed on read (not stored), so they always reflect the latest data:
def precision(tp, fp):
if tp + fp == 0:
return None # no alert fired: precision undefined
return tp / (tp + fp)
def recall(tp, fn):
if tp + fn == 0:
return None # technique was not executed: recall undefined
return tp / (tp + fn)
def f1(p, r):
if p is None or r is None or (p + r) == 0:
return None
return 2 * p * r / (p + r)
def detection_coverage(session_results):
recalls = [recall(r["tp"], r["fn"]) for r in session_results
if recall(r["tp"], r["fn"]) is not None]
return sum(recalls) / len(recalls) if recalls else 0.0
Edge cases:
tp_count + fn_count = 0: the technique was not executed during the session. Exclude from recall computation; note as "not tested."tp_count + fp_count = 0: the detection rule never fired. Precision is undefined; this is a pure miss (recall = 0 iffn_count > 0).tp_count > 0andfn_count = 0: perfect recall for this technique in this session.
4.3 Gap Prioritizer
Two-factor priority:
- Recall score (primary): lower recall = more urgent gap. Sort ascending.
- Threat actor prevalence (secondary): how many real-world threat actors use this
technique, according to MITRE ATT&CK's
Actor Mappingdata? More actor usage = higher priority if recall is equal.
def top_gaps(session_results, threat_actor_map, n=5):
gaps = []
for r in session_results:
rc = recall(r["tp"], r["fn"])
if rc is None:
continue # not tested
prevalence = threat_actor_map.get(r["attck_id"], 0)
gaps.append({
"attck_id": r["attck_id"],
"attck_name": r["attck_name"],
"recall": rc,
"prevalence": prevalence,
"priority_score": (1 - rc) * 10 + prevalence,
})
gaps.sort(key=lambda g: -g["priority_score"])
return gaps[:n]
Recommendation generation: for each top gap, the technique_recommendations table provides
the data source to enable, the Sigma rule reference, and the effort estimate. The report
renders this as: "Technique X (recall = 20%): Enable Sysmon EID 8. Deploy Sigma rule Y.
Estimated effort: 2 days."
4.4 ATT&CK Navigator Export
The ATT&CK Navigator accepts a JSON layer file that colors each technique cell. We map recall to a color gradient:
def recall_to_color(recall_value):
if recall_value is None:
return "#cccccc" # grey: not tested
if recall_value >= 0.8:
return "#00b050" # green: well-covered
if recall_value >= 0.5:
return "#ffcc00" # yellow: partial
return "#ff0000" # red: gap
def export_navigator_layer(session_results, session_name):
techniques = []
for r in session_results:
rc = recall(r["tp"], r["fn"])
techniques.append({
"techniqueID": r["attck_id"],
"score": round((rc or 0) * 100),
"color": recall_to_color(rc),
"comment": f"Recall: {rc:.0%}" if rc is not None else "Not tested",
"enabled": True,
})
return {
"name": session_name,
"versions": {"attack": "14", "navigator": "4.9"},
"domain": "enterprise-attack",
"techniques": techniques,
}
4.5 Trend Tracking
For each technique tested in multiple sessions, track recall over time:
SELECT
attck_id,
session_date,
tp_count::float / NULLIF(tp_count + fn_count, 0) AS recall
FROM technique_results
JOIN exercise_sessions ON session_id = exercise_sessions.id
WHERE attck_id = 'T1055.003'
ORDER BY session_date;
This query powers the trend chart in the dashboard: "Recall for T1055.003 went from 20% in January to 80% in April after Sysmon EID 8 was deployed."
5. Tradeoffs
Manual entry vs. automated SIEM integration:
- Automated: the platform pulls alert counts directly from the SIEM API when the red team runs a technique. No analyst manual entry required.
- Automated: requires SIEM API credentials stored in the platform, integration maintenance per SIEM version, and handling of delayed alert delivery.
- Manual: the blue analyst counts alerts during the exercise window. Simple, works with any SIEM.
- Decision: manual entry as the primary mode; API endpoint for automated entry from CLI tools or SIEM webhooks. Manual entry remains correct even when the automation breaks.
Immutability after sign-off:
- Sign-off by the blue analyst marks results as final. After that,
UPDATEis blocked by the application service role. - This prevents retroactive result inflation ("we changed the detection rule and want to re-score the session").
- Tradeoff: if an error is discovered post-sign-off, an admin must manually correct the data (with an audit log entry explaining the change).
ATT&CK Navigator vs. custom visualization:
- Building a custom heatmap means maintaining a frontend visualization component.
- ATT&CK Navigator is a well-known, well-maintained tool that clients and blue teams already use. Exporting a Navigator layer JSON file means zero custom frontend work and maximum compatibility.
- Decision: export ATT&CK Navigator JSON; the platform provides no custom heatmap.
6. Detection-Forward Perspective
This platform is the institutionalization of the detection-forward philosophy:
- Every technique tested in a purple team session has a measured recall score. Low recall is the quantified detection gap.
- The gap prioritizer and recommendation table translate measurement into action: "T1055.003 recall = 20%, gap is Sysmon EID 8, deploy this Sigma rule, 2-day effort."
- The trend chart answers the CISOs question: "Are we getting better?" The answer is now data-driven, not anecdotal.
- The ATT&CK Navigator export turns the abstract detection matrix into a specific, colored heat map of the organization's actual coverage state.
This is the endpoint of the Cedar Lattice curriculum. The red team engagement identifies the gaps. The purple team session validates whether the deployed detections close those gaps. The platform tracks the improvement over time. The full loop is: engage → report → detect → purple-team → measure → improve → re-engage.