Lab 03 — Compliance Control Evidence Validator

Difficulty: 3/5 | Runs locally: yes

Pairs with the Phase 07 WARMUP Chapter 12 (compliance as engineering — controls and evidence).

Why This Lab Exists (Purpose & Goal)

To most engineers, "compliance" means screenshots and spreadsheets — bureaucracy to be endured. To a security engineer, a compliance framework is a requirement specification you translate into tested engineering controls with fresh evidence. The goal of this lab is to build the validator that enforces that translation: it rejects control records that are checkbox theater and demands a real, verifiable chain from requirement to evidence.

The Concept, In the Weeds

A real control record links a chain, and each link is load-bearing:

requirement → implementation → test → evidence → owner → frequency → freshness → exception

The validator detects the ways this chain is faked:

stale evidence — the control was tested once, a year ago; "freshness" is part of the control, because a control that no longer operates is no control.
untested controls — an "implementation" with no test proving it operates; this is the most common gap. "We enforce TLS" with no automated check that rejects cleartext is a claim, not a control.
missing owners — nobody accountable means nobody maintains it.
scanner-result-without-implementation — citing a scan finding as if it were a control. A scan is evidence of a test, not the implementation itself; the implementation is the actual enforcement.

The decisive idea — the same as the threat-model validator (Phase 03) — is that a test/verification method is what separates a control from a wish. "Scan-only compliance" produces a checkbox with no proof the control actually operates, which is exactly the failure that turns a "compliant" company into a breached one.

Why This Matters for Protecting the Company

Companies are required to be compliant (FedRAMP for government cloud, PCI for cards, HIPAA for health, SOC 2 and ISO 27001 for enterprise sales) — compliance is often a precondition to having customers at all. But compliance done as paperwork gives false assurance; compliance done as engineering (tested controls + fresh evidence) is a genuine security backbone. A validator that enforces the evidence chain lets a security team turn audit obligations into real, continuously-verified controls — satisfying the auditor and actually reducing risk, instead of doing theater for one and security for the other.

Build It

Implement the validator: control records must link requirement, implementation, test, evidence, owner, frequency, freshness, and exception. Detect stale evidence, untested controls, missing owners, and controls that cite a scanner result without an implementation.

LAB_MODULE=solution pytest -q

Secure Implementation Patterns

The anti-pattern (scan-only "compliance"). A control whose "implementation" is a scanner result and whose evidence is a year old:

- id: C-12
  requirement: "encrypt data in transit"
  implementation: "scanner: tls-checker passed"   # a scan is EVIDENCE OF A TEST, not the control
  last_verified: 20240101                          # stale

The secure pattern — enforce the full evidence chain + freshness (mirrors solution.py):

REQUIRED = ("id", "requirement", "implementation", "test", "evidence",
            "owner", "frequency_days", "last_verified")

def validate(controls, *, today) -> tuple[str, ...]:
    issues = []
    for i, ctrl in enumerate(controls):
        for field in REQUIRED:
            if ctrl.get(field) in (None, ""):
                issues.append(f"{i}.{field}:missing")                 # the whole chain must be present
        freq, verified = ctrl.get("frequency_days"), ctrl.get("last_verified")
        if isinstance(freq, int) and isinstance(verified, int) and today - verified > freq:
            issues.append(f"{i}.evidence:stale")                      # FRESHNESS is part of the control
        if str(ctrl.get("implementation", "")).lower().startswith("scanner"):
            issues.append(f"{i}.implementation:not-a-control")        # a scan != an implementation
    return tuple(sorted(issues))

A well-formed control:

- id: C-12
  requirement: "encrypt cardholder data in transit (PCI 4.2.1)"
  implementation: "TLS 1.2+ enforced by edge config and an admission policy rejecting cleartext"
  test: "ci/test_no_cleartext_endpoints"        # automatable verification
  evidence: "artifacts/tls-scan-2026-06-12.json"
  owner: "platform-security"
  frequency_days: 7
  last_verified: 20260610

Production practices to carry forward:

Implementation = the enforcement; evidence = the test result. Don't conflate them; a scan proves the test ran, it isn't the control.
Freshness is enforced — a control whose evidence aged past its frequency is failing, not passing.
Wire evidence to live sources (cloud config queries, CI test results) with automated freshness checks, so the same control satisfies the auditor and protects production.
One requirement → control → test → evidence → owner → frequency chain serves both audit and security; never run "paperwork for the auditor, security for ourselves."

Validation — What You Should Be Able to Do Now

Translate a compliance requirement (e.g. "encrypt data in transit") into implementation → test → evidence → owner → frequency.
Distinguish an implementation (the enforcement) from evidence (the test result) and explain why citing a scanner result as a control is a gap.
Explain why freshness is part of the control, and why an untested control is no control.

The Broader Perspective

This lab completes the curriculum's central refrain — first met in the threat-model validator and the provenance verifier: every claimed control must be tied to a test that proves it operates, with evidence that stays fresh. Internalizing requirement → control → test → evidence is what lets you turn the most "checkbox" part of the job into rigorous engineering, and it's the lens that exposes hollow assurances everywhere ("we handle that" — show me the passing test). At the staff level, this is how you make compliance a force multiplier for security rather than a tax: the same controls satisfy the auditor and protect the company, with one evidence chain serving both.

Interview Angle

"Map a PCI/HIPAA/FedRAMP requirement into implementation and evidence." — Take "encrypt data in transit": implement TLS 1.2+ enforced by config + admission; test = an automated check rejecting cleartext endpoints; evidence = the recurring result + config, owned by platform, run continuously with a freshness check. Deliver requirement → control → test → evidence → owner → frequency, not a screenshot.

Extension (Stretch)

Connect the validator to live evidence sources (cloud config queries, CI test results) with automated freshness checks, so a control whose evidence goes stale is flagged automatically.

References

Phase 07 WARMUP Chapter 12; NIST SP 800-53; FedRAMP; PCI DSS v4.0; SOC 2 TSC; ISO 27001.

Security Engineer