P03 Lab 05 — Secret Detection

WARMUP: Chapter 6 (the scanner zoo — secret scanning). The model behind gitleaks / truffleHog.

The idea

Credentials committed to source are a top breach cause. Detect them two ways: high-confidence patterns (AWS keys, GitHub PATs, private-key headers, Slack tokens) and high Shannon entropy strings that look random enough to be a key — while suppressing placeholders and an explicit allowlist to control false positives. The key operational lesson: a hit means rotate the secret (it's in git history forever), not just delete the line.

What you build (lab.py)

  • shannon_entropy(s) — bits/char.
  • scan(lines, entropy_threshold, min_len, allowlist) — pattern matches (critical), hardcoded assignments (high, placeholder-suppressed), and high-entropy tokens (medium); de-duplicated and sorted by (line, severity, rule).
  • has_secrets(lines).

Cases the tests cover

  • Entropy basics (""/"aaaa" = 0; a random 39-char token > 4.0 bits); AWS/GitHub/private-key patterns; hardcoded-secret assignments; placeholders not flagged; high-entropy detection; a long low-entropy string not flagged; allowlist suppression; no double-flag (an AWS key is high-entropy too but reported once as the pattern); line numbers and a clean file.

Run

pip install -r requirements.txt
LAB_MODULE=solution pytest -q
pytest -q

Hardening / extensions

  • Add verification (call the provider to test if a key is live) to cut false positives.
  • Scan git history (not just the working tree) and compute per-rule precision against a labeled set.
  • Add more providers (GCP, Stripe, JWTs) and base64/hex-aware entropy; add inline # pragma: allowlist.

Interview / resume

"Built a secret detector combining high-confidence regex patterns with Shannon-entropy detection and placeholder/allowlist suppression — the gitleaks/truffleHog model — and encoded that a hit means rotate, because the secret is already in git history."

Limitations: working-tree text scan (no git history or live verification); entropy threshold is a tunable heuristic with the usual precision/recall tradeoff.