Phase 10 — Reliability, Debugging & Root-Cause Analysis
Difficulty: ⭐⭐⭐⭐⭐ | Estimated Time: 2 weeks
Roles supported: Rack Management SWE (Senior/Staff), SRE, Production/Reliability SWE
Hardware needed: none — a Linux box (or VM) for the real gdb/strace/valgrind extensions
Why This Phase Exists
This is the JD's most-repeated theme, stated three ways: "Own debugging and root‑cause analysis of complex issues across development, integration, and production environments, including deployed systems," "thrives in... deep technical debugging," and the entire role being "execution-focused, emphasizing... debugging... of infrastructure software that runs close to hardware." If you take one phase to interview-ready perfection, make it this one — it's what the hiring manager most wants to see.
Debugging at this level is not luck; it's method. This phase teaches the systematic
discipline (observe → hypothesize → test → narrow → fix → verify → prevent), the Linux toolset
(gdb, strace, ltrace, perf, ftrace, eBPF, sanitizers, core dumps), the reliability
patterns that prevent failures (retries, backoff, circuit breakers, idempotency, reconciliation,
bulkheads), chaos/fault injection to find weaknesses before production does, and the RCA/
postmortem writeup that turns an incident into a permanent improvement (Phase 12).
You'll find a planted bug with real tools and write its RCA, and build a chaos harness that hardens a reconcile loop.
Concepts
Systematic debugging method
- The loop: reproduce → observe → hypothesize → bisect/narrow → test the hypothesis → fix → verify → prevent
- Scientific debugging (one variable at a time); reading the evidence before guessing; the "what changed?" question
- Heisenbugs, race conditions, the role of determinism; binary search (
git bisect) in space and time - Knowing your failure domain first (Phase 01) — single node vs shared resource
The Linux debugging toolset
gdb: breakpoints, backtraces, watchpoints, core-dump analysis, attaching to a live processstrace/ltrace: syscall/library-call tracing — the "what is it actually doing/why is it stuck" toolperf: CPU profiling, flame graphs, hardware counters — for "why is it slow"ftrace/eBPF (bcc/bpftrace): kernel/dynamic tracing — production-safe deep observability- Sanitizers (ASan/UBSan/TSan — Phase 02) and valgrind: memory/race bugs
- Core dumps &
kdump(Phase 08): post-mortem of a crash - Logs/metrics/traces (Phase 07) as the first-line evidence, especially for deployed systems
Reliability patterns (prevent the failure)
- Retries + backoff + jitter (Phase 02); idempotency (Phase 05); timeouts everywhere
- Circuit breaker (stop hammering a failing dependency); bulkhead (isolate failures); rate limiting
- Reconciliation / self-healing (Phase 06): converge to desired state despite transient faults
- Graceful degradation (Phase 02); fail-safe vs fail-open; backpressure
- Redundancy & failure-domain-aware design (Phase 01)
Proactive: chaos & fault injection
- Chaos engineering: inject failures (kill, slow, partition, corrupt) to find weaknesses first
- Fault injection in tests; game days; the blast-radius-bounded experiment
Closing the loop: RCA & postmortems
- The 5 Whys; contributing factors vs root cause; blameless postmortems
- The RCA document: timeline, impact, root cause, the fix, action items to prevent recurrence
- Feeding fixes back into tests (Phase 11), runbooks, and design (Phase 12)
Labs
Lab 01 — Debugging Dojo: Find the Planted Bug
| Field | Value |
|---|---|
| Goal | Find a deliberately planted bug in a close-to-hardware C program using real tools (ASan/gdb/valgrind), then write the RCA — practicing the method, not just the fix. |
| Concepts | Systematic debugging, memory bugs (use-after-free/buffer overflow/race), reading a backtrace, sanitizers, the RCA writeup. |
| Steps | 1) make && ./buggy — observe the wrong/crashing behavior. 2) make asan — let AddressSanitizer point at the bug. 3) Confirm with gdb/valgrind. 4) Fix it and verify. 5) Fill in RCA.md. |
| Stack | C (C11), make, ASan; optional gdb/valgrind |
| Output | The identified bug, a verified fix, and a completed RCA document. |
| How to Test | The fixed program passes its asserts and is clean under ASan; your RCA follows the timeline→root-cause→prevention structure. |
| Talking Points | How you localize a bug systematically; why ASan/TSan catch what testing misses; reading a backtrace; root cause vs symptom. |
| Resume Bullet | "Diagnosed memory-safety bugs in close-to-hardware C using AddressSanitizer/gdb/valgrind and documented blameless RCAs with preventive action items." |
| Extensions | Plant your own race and find it with TSan; debug a hung process with strace; analyze a core dump in gdb; add a git bisect exercise. |
Lab 02 — Chaos & Reconciliation Hardening
| Field | Value |
|---|---|
| Goal | Build a chaos/fault-injection harness against a reconcile loop and prove that reliability patterns (retry/backoff, circuit breaker, idempotency) let it converge despite injected failures, partitions, and slow dependencies. |
| Concepts | Reconciliation/self-healing, retries/backoff/jitter, circuit breaker, idempotency, timeouts, chaos engineering, measuring resilience. |
| Steps | 1) python3 solution.py — run the loop under a fault-free baseline, then under chaos. 2) See the naive loop fail and the hardened loop converge. 3) Watch the circuit breaker open/close. 4) Extensions. |
| Stack | Python 3 stdlib |
| Output | A chaos harness + a hardened reconcile loop that converges under injected faults; metrics on attempts/recoveries. |
| How to Test | Asserts verify the hardened loop converges under chaos where the naive one doesn't, the circuit breaker trips and recovers, and operations are idempotent under retry. |
| Talking Points | Why retries need backoff+jitter+a cap; what a circuit breaker prevents (retry storms); idempotency under retry; chaos finds what tests don't. |
| Resume Bullet | "Built a chaos/fault-injection harness and hardened a reconcile loop with retry/backoff, circuit breaking, and idempotency, proving convergence under injected failures and partitions." |
| Extensions | Add a bulkhead/rate limiter; inject a network partition and a slow dependency; add an SLO and measure the error budget burned under chaos. |
Deliverables Checklist
- You can state the systematic debugging loop and apply it under pressure
-
You can pick the right tool:
gdb(state),strace(syscalls/stuck),perf(slow), ASan/TSan (memory/race), eBPF (production) - You found the planted bug with tools (not by reading luck) and verified the fix under ASan
- Your RCA follows timeline → impact → root cause → fix → prevention
- You can explain retries+backoff+jitter, circuit breakers, idempotency, and reconciliation
- Hardened reconcile loop converges under chaos where the naive one fails
- You can explain why chaos/fault injection finds what normal tests don't
- You can run a blameless postmortem and turn it into action items (Phase 12)
Interview Relevance
- "Walk me through how you'd debug a production node that's intermittently rebooting." (method!)
- "A daemon is using 100% CPU. How do you find why?" (
perf/flame graph) - "A process is hung. What do you do?" (
strace, where is it blocked) - "You suspect a memory corruption. What tools?" (ASan/valgrind,
gdb) - "What's a circuit breaker and why do you need one?" (retry storms)
- "How do you make a flaky operation reliable?" (retry+backoff+jitter+idempotency+timeout)
- "Tell me about a hard bug you root-caused." (your RCA — STAR, Phase 12)
- "What is chaos engineering and would you run it in production?"