🛸 Hitchhiker's Guide — Phase 10: Reliability, Debugging & RCA

Read this if: you debug by instinct and want it to be a repeatable method — the JD's #1 skill. Compressed map of the loop, the toolset, the prevention patterns, and the RCA.

0. The 30-second mental model

Debugging is a method: reproduce → observe (don't guess) → "what changed?" → hypothesize → bisect → fix → prevent. Match the tool to the symptom. For deployed systems you can't restart, work evidence-first via OOB + eBPF + correlation. Then prevent failures with timeouts, retry+backoff+jitter, circuit breakers, idempotency, reconciliation, prove it with chaos, and close the loop with a blameless RCA that ships action items.

1. The debugging loop

reproduce → observe the evidence first → ask "what changed?" (deploy/firmware/config/load) → hypothesize (falsifiable) → bisect (code/data/fleet/time) → fix → verify → prevent (test/ alert/guard/runbook). Scope the failure domain first (Phase 01): one node or shared resource?

2. Tool ↔ symptom

Anti-pattern: printf-rebuild loops when strace/ASan/perf answer it in minutes.

3. Deployed-system debugging

Can't attach/restart → evidence-first: logs/metrics/traces (P07), SEL/BERT + RAS (P08), SoL (P04); eBPF for live insight w/o restart; correlate metric→trace→log; reproduce on a canary; bisect across the fleet (all nodes? one rack? one firmware version?).

4. The hard bugs

races/Heisenbugs (→ TSan, deterministic tests, inject the clock) · memory (→ ASan/valgrind; crash is far from cause) · resource exhaustion (unbounded growth → bound it) · hardware nondeterminism (flaky BMC/marginal PCIe/thermal → is it SW or HW? use RAS+telemetry).

5. Prevention patterns

timeouts everywhere · retry + backoff + jitter + cap (no retry storm) · circuit breaker (open→half-open→closed) · idempotency · reconciliation/self-heal · bulkheads · graceful degradation / fail-safe · backpressure/rate-limit · redundancy + failure-domain awareness (P01).

6. Chaos

Inject failures on purpose (kill/slow/partition/corrupt/exhaust/throttle) → verify graceful degrade + recovery. Hypothesis-driven, bounded blast radius, game days. The fake-driver seam (P02) is unit-level chaos. Chaos finds the unhappy paths tests skip.

7. RCA / blameless postmortem

Blameless (if a human could cause it, the system allowed it) · 5 Whys to the systemic cause · root cause vs contributing factors · doc = timeline + impact (SLO/budget) + detection + root cause + fix + owned, dated action items (test/alert/guard/runbook/design). No tracked actions = a diary, not engineering.

8. The "done this before" tells

"What changed?" · "Is it one node or a shared resource?" · "Did you strace it / look at the flame graph?" · "ASan-clean?" · "Where's the timeout / circuit breaker?" · "Is it idempotent under retry?" · "What are the action items?"

9. How this phase pays off later

It IS the JD's #1 skill — front and center in interview-prep/04. Prevention patterns harden the Phase 13 capstone; RCAs become Phase 12 runbooks; chaos feeds Phase 11 testing; evidence comes from Phases 07/08.