Warmup Guide — Reliability, Debugging & Root-Cause Analysis

Zero-to-expert primer for Phase 10 — the JD's most-emphasized skill. You may already debug by instinct; this guide makes it a method you can apply under pressure to a production system you can't restart, with the right tool for each symptom, plus the patterns that prevent failures and the RCA that makes them never recur.

Table of Contents

• Chapter 1: Debugging Is a Method, Not a Talent
• Chapter 2: The Toolset and When to Reach for Each
• Chapter 3: Debugging a Deployed System
• Chapter 4: The Hard Bugs — Races, Heisenbugs, Memory
• Chapter 5: Reliability Patterns That Prevent Failures
• Chapter 6: Chaos & Fault Injection
• Chapter 7: RCA and the Blameless Postmortem
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Debugging Is a Method, Not a Talent

The engineers who seem to "just know" where bugs are are running a disciplined loop fast. Make it explicit:

1. Reproduce (or capture): a bug you can reproduce is half-solved. If you can't, capture maximal evidence (logs/traces/metrics/core dump — Phase 07/08) the next time it happens.
2. Observe before theorizing: read the actual evidence — the error, the backtrace, the metric, the syscall — before guessing. Most wrong debugging is theory-first.
3. Ask "what changed?": the vast majority of production incidents follow a change (deploy, firmware, config, traffic). git bisect, the deploy log, the firmware version (Phase 08).
4. Hypothesize, then test one variable: form a falsifiable hypothesis and design the cheapest experiment that confirms or kills it. Change one thing at a time.
5. Bisect/narrow: binary-search the problem space — in code (git bisect), in data (which input), in the fleet (which node/rack — Phase 01 failure domain), in time (when did it start).
6. Fix → verify: confirm the fix actually resolves it (and didn't just move it), ideally with a regression test (Phase 11).
7. Prevent: the RCA's real output — a test, an alert, a guard, a runbook so it can't recur silently (Chapter 7, Phase 12).

This loop is what an interviewer is listening for when they say "debug this" — they want the method, narrated, far more than a lucky guess.

Failure-domain first (Phase 01): before diving into one node, ask "is this one node or a shared resource?" Six nodes down together → a PDU/ToR/CDU, not six bugs. The first cut of any incident is scoping the blast radius.

Chapter 2: The Toolset and When to Reach for Each

Matching the tool to the symptom is half the skill. The map:

A few signature moves:

• strace -p <pid> on a hung process → see it blocked in a futex (lock) or read (waiting on a slow BMC) → that is the clue.
• perf top / a flame graph on a 100%-CPU daemon → the hot function jumps out.
• ASan on a crashing C program → the exact use-after-free with both stack traces (allocation and use). Lab 01.
• bpftrace -e 'tracepoint:...' to count/trace events on a production node without restarting it.

The anti-pattern: adding printfs and rebuilding for hours when strace/ASan/perf would point at it in minutes. Knowing the toolset is the seniority.

Chapter 3: Debugging a Deployed System

The JD stresses "including deployed systems" — the hardest case, because you often can't attach a debugger, can't reproduce locally, and can't restart it (it's serving tenants). The approach shifts:

• Evidence-first, non-invasive: start with what's already collected — structured logs, metrics, traces (Phase 07), the SEL/RAS logs (Phase 08), and the OOB console (SoL, Phase 04). For a node that won't boot, your only window may be Redfish/SoL.
• eBPF for live insight: bpftrace/bcc attach to a running production process/kernel with minimal overhead to answer "what syscalls is it making / how long are BMC calls taking / which request is slow" without a restart. This is the modern production-debugging superpower.
• Correlate (Phase 07): metric anomaly → trace in that window → logs for the failing span → RAS/telemetry for the hardware. The metrics→traces→logs loop.
• Reproduce in a safe copy: pull the inputs/state and reproduce in a lab/canary, not on the live fleet. Chaos (Chapter 6) and HIL (Phase 11) help you reproduce hardware-adjacent bugs.
• Bisect across the fleet (Phase 01): is it all nodes, one rack, one firmware version (Phase 08), one hardware batch? The pattern localizes the cause.

The mindset: you're a detective working from traces left behind, with OOB as your one reliable channel to a sick machine. Build the system so it leaves good traces (Phase 02/07) — future-you debugging at 3 a.m. depends on past-you's instrumentation.

Chapter 4: The Hard Bugs — Races, Heisenbugs, Memory

The bugs that define "complex issues":

• Race conditions: two threads/processes touching shared state without proper synchronization (Phase 02). Symptoms are intermittent and load-dependent. They vanish under a debugger (timing changes) — a classic Heisenbug. The tool is TSan (catches the racing accesses deterministically) plus disciplined lock review (lock_guard, tiny critical sections).
• Memory bugs (C/C++): use-after-free, buffer overflow, uninitialized reads, double-free. Often crash later, far from the cause, making backtraces misleading. ASan/valgrind give you the real cause (allocation + use sites). Lab 01.
• Heisenbugs generally: bugs that change behavior when observed (timing, optimization, added logging). Strategy: capture non-invasively (eBPF, sampling), make timing deterministic in tests (inject the clock — Phase 02), and reason about the invariant being violated.
• Resource exhaustion: fd/memory/handle leaks, thread-pool starvation, unbounded queues — manifest as slow degradation then collapse. Watch for unbounded growth (Phase 07 metrics) and enforce bounds (Phase 02).
• Hardware-adjacent nondeterminism: a flaky BMC, a marginal PCIe lane (Phase 04), a thermal throttle (Phase 04/08) — the "bug" is the hardware, and the skill is distinguishing a software bug from a hardware fault using RAS (Phase 08) and telemetry (Phase 07).

The recurring lesson: make the nondeterministic deterministic — inject the clock/randomness, use sanitizers, bound resources, and capture enough evidence that you don't need to reproduce.

Chapter 5: Reliability Patterns That Prevent Failures

The best debugging is the failure that never happens. The patterns (some met earlier, unified here):

• Timeouts everywhere: no unbounded wait on a BMC/network/lock. A missing timeout turns one slow dependency into a system-wide hang.
• Retries + exponential backoff + jitter + a cap (Phase 02): handle transient faults without a retry storm (synchronized retries that DDoS a recovering dependency — jitter prevents the thundering herd).
• Circuit breaker: after N failures to a dependency, stop trying for a cooldown (open the breaker), then test tentatively (half-open) before resuming (closed). Prevents hammering a dead BMC and cascading failures. Lab 02.
• Idempotency (Phase 05): operations safe to retry/re-run, so retries and reconciliation can't corrupt state.
• Reconciliation / self-healing (Phase 06): continuously converge actual to desired, so transient faults auto-correct rather than requiring a human.
• Bulkheads: isolate resources (separate pools/queues per dependency or tenant) so one failure can't sink everything (like a ship's compartments).
• Graceful degradation & fail-safe (Phase 02): lose one sensor → mark stale, keep going; on a leak → fail safe (Phase 04). Decide fail-open vs fail-closed deliberately per case.
• Backpressure & rate limiting: shed or slow load rather than collapse under it.
• Redundancy & failure-domain awareness (Phase 01): N+1, spread across domains.

Designing these in is reliability engineering; Lab 02 makes a naive loop survive chaos by adding them.

Chapter 6: Chaos & Fault Injection

You don't truly know your system's failure behavior until you've caused failures on purpose. Chaos engineering: inject controlled failures — kill a process, slow a dependency, drop packets (partition), corrupt a response, exhaust a resource, throttle a node (thermal, Phase 04) — and verify the system degrades gracefully and recovers. The discipline:

• Hypothesis-driven: "if the BMC for node-7 stops responding, provisioning should retry and reconcile, not hang the controller." Then inject and check.
• Bounded blast radius: start in a lab/canary, bound the experiment, have an abort.
• Game days: scheduled exercises where the team practices an incident — building the muscle and the runbooks (Phase 12) before the real one.
• Fault injection in tests (Phase 11): the fake driver (Phase 02 seam) that returns errors/ timeouts/garbage is unit-level chaos — cheap and deterministic. Lab 02's harness scales this idea to a reconcile loop.

Chaos finds the bugs normal tests miss because normal tests exercise the happy path; production is mostly unhappy paths. For a rack platform — fragile BMCs, flaky hardware, network blips — assume faults and prove resilience before the fleet does it for you.

Chapter 7: RCA and the Blameless Postmortem

An incident you don't learn from will recur. The RCA (Root Cause Analysis) / postmortem turns a failure into a permanent improvement — and writing a good one is a Staff-level signal (Phase 12).

• Blameless: focus on the system and process that allowed the failure, not the person. People act reasonably given the information and tools they had; if a human could cause an outage, the system let them. Blame kills the honesty that makes postmortems useful.
• The 5 Whys: keep asking "why" past the proximate cause to the systemic one. "The node went down" → why? "OOM" → why? "a leak in the exporter" → why? "an unbounded cache" → why? "no resource limit / no test for it" → there's the root cause and the fix.
• Root cause vs contributing factors: usually it's not one cause but a chain; identify the systemic one(s) you can fix to prevent the class of failure, not just this instance.
• The document: timeline (what happened when), impact (who/what, how long, SLO/error-budget burned — Phase 07), detection (how you found out — and how you should have), root cause, the fix, and — the most important part — action items: concrete, owned, dated changes (a test, an alert, a guard, a runbook, a design change) that prevent recurrence.
• Close the loop: feed the fix into a regression test (Phase 11), an alert/runbook (Phase 07/12), and the design (Phase 12). An RCA without tracked action items is a diary entry, not engineering.

This is where "own root-cause analysis" (the JD) becomes leadership: you don't just fix the bug, you make the organization permanently better at not having it.

Lab Walkthrough Guidance

Order: Lab 01 (debug a real bug) → Lab 02 (prevent failures with chaos). Reactive skill first, then proactive.

Lab 01 (debugging dojo):

1. make && ./buggy — observe the wrong/crashing behavior; resist guessing.
2. make asan — let AddressSanitizer point at the exact bug (allocation + use sites).
3. Confirm your understanding with gdb (backtrace) / valgrind; fix it; verify clean under ASan.
4. Write RCA.md: timeline → root cause → fix → prevention. The writeup is the lab.

Lab 02 (chaos + reconciliation):

1. python3 solution.py — baseline (no faults), then under chaos (random failures, slow deps, partitions).
2. See the naive loop fail to converge and the hardened loop (retry/backoff + circuit breaker + idempotency) converge anyway.
3. Watch the circuit breaker open under sustained failure and recover (half-open → closed).
4. Extensions: add a bulkhead/rate limiter; inject a partition; measure error budget burned.

Phase capstone question (the one to nail): "A production node reboots intermittently under load. Walk me through your debugging — you can't take it offline." (Answer: scope the failure domain first (one node or shared?); evidence-first via OOB — SEL/BERT (Phase 08) for the last-crash cause, RAS parser for AER/MCE, telemetry (Phase 07) correlating reboots with temperature/power/load; "what changed?" — recent firmware (Phase 08) or deploy; reproduce on a canary with chaos/load; if MCE points at a DIMM with rising correctable ECC under thermal load → predictive DIMM fault → cordon/drain (Phase 06), RMA (Phase 08), re-validate; then prevent — add the alert and the test, write the blameless RCA. The method, narrated, is the answer.)

Success Criteria

You're done with this phase when — without notes:

• You can narrate the systematic debugging loop and apply it to a prompt (Ch. 1)
• You can pick the right tool per symptom (gdb/strace/perf/ASan/TSan/eBPF) (Ch. 2)
• You can debug a deployed system evidence-first via OOB + eBPF + correlation (Ch. 3)
• You can explain races/Heisenbugs/memory bugs and the tools that catch them (Ch. 4)
• You can explain retries+backoff+jitter, circuit breakers, idempotency, reconciliation, bulkheads (Ch. 5)
• You found the planted bug with tools and verified the fix under ASan (Lab 01)
• Your hardened reconcile loop converges under chaos; you can explain why (Lab 02)
• You can write a blameless RCA with a real root cause and tracked action items (Ch. 7)

Interview Q&A

Q1: A production node reboots intermittently under load and you can't take it offline. How do you debug it? A: Method first. (1) Scope the failure domain (Phase 01): is it just this node or others on the same PDU/ToR/CDU? (2) Evidence-first via OOB since the host is unreliable: read the SEL and BERT (Phase 08) for the last-crash cause, run the RAS parser on AER/MCE, and correlate the reboots with telemetry (Phase 07) — do they line up with temperature, power draw, or load spikes? (3) Ask "what changed?" — a recent firmware update (Phase 08) or deploy is the prime suspect; check versions. (4) Form a hypothesis and test it cheaply — e.g., if MCE points at a DIMM with rising correctable ECC that climbs with temperature, the hypothesis is a marginal DIMM failing under thermal load. Reproduce on a canary with load/chaos rather than the live node. (5) Fix: cordon/drain (Phase 06), RMA the DIMM (Phase 08), re-validate (Phase 01), return to service. (6) Prevent: add the alert and a test, and write a blameless RCA. The point is the narrated method and using OOB/telemetry because I can't attach a debugger to a tenant-serving node.

Q2: A daemon is pegged at 100% CPU. How do you find why? A: perf top or a perf record + flame graph on the process shows where the cycles actually go — the hot function jumps out, no guessing. If it's a busy-loop, the flame graph shows the loop; if it's a specific call, it's named. I'd cross-check with strace (is it spinning on a syscall like a failing non-blocking read?) and metrics (Phase 07) for what changed. The anti-pattern is adding print statements and rebuilding for hours; perf answers it in minutes. Then I fix the hot path (or add backpressure if it's overload) and add a CPU-usage alert so it's caught next time.

Q3: A process is hung. What's your first move? A: strace -p <pid> to see the exact syscall it's blocked in. If it's stuck in a futex, it's a lock/deadlock — I'd inspect with gdb -p and look at thread backtraces for who holds what. If it's blocked in read/recvfrom/connect, it's waiting on a slow or dead dependency (a BMC, the network) with no timeout — the bug is the missing timeout (Phase 05/10). strace turns "it's hung" into "it's blocked here, waiting on this," which is the whole clue. For a production process I'd prefer eBPF/bpftrace to peek without risk.

Q4: What's a circuit breaker and why do you need one with retries? A: A circuit breaker stops calling a failing dependency after a threshold of failures: it "opens" (fail fast for a cooldown), then goes "half-open" to test tentatively, and "closes" again once the dependency recovers. You need it with retries because retries alone create a retry storm — when a BMC or service is down, every client retrying (especially in sync) hammers it and prevents recovery, cascading the failure. Backoff + jitter spread the load; the breaker stops the bleeding entirely when the dependency is clearly down, fails fast for callers (better than timeouts piling up), and probes for recovery. Together with idempotency and timeouts, it's the core of resilient calls to flaky hardware like BMCs.

Q5: How do you make an intermittent race condition reproducible and fix it? A: Races are timing-dependent and often vanish under a debugger (a Heisenbug), so I make the nondeterminism deterministic. The decisive tool is ThreadSanitizer — it detects the racing accesses without needing the race to actually manifest, pointing at the two unsynchronized accesses and the missing lock. I review the critical sections (Phase 02): is shared state accessed under a lock_guard, are critical sections tiny, is anything held across I/O? In tests I inject the clock/ scheduling to force the interleaving. The fix is proper synchronization (a guard, an atomic, or redesigning to avoid shared mutable state), and I add a TSan run to CI (Phase 11) so the class of bug can't return. The principle: don't chase a Heisenbug by adding logging (which changes timing) — use the sanitizer that finds it deterministically.

Q6 (Staff): Tell me how you run a postmortem after a serious incident. A: Blamelessly and with tracked outcomes. I write a document with a precise timeline, the impact (who/what, duration, SLO/error-budget burned — Phase 07), how we detected it (and how we should have, sooner), and then drive to root cause with the 5 Whys — past the proximate cause ("the node OOMed") to the systemic one ("an unbounded cache with no memory limit and no test"). I separate the root cause from contributing factors and focus fixes on preventing the class of failure. The most important section is the action items: concrete, owned, dated — a regression test (Phase 11), an alert and runbook (Phase 07/12), a guard, maybe a design change — and I track them to completion. I keep it blameless because if a human could trigger the outage, the system allowed it, and blame destroys the honesty that makes the postmortem useful. As a Staff engineer, the deliverable isn't just the fix — it's an organization that won't have that class of incident again, and the shared learning across the (often distributed) team.

References

• "Debugging" methodology — David J. Agans, Debugging: The 9 Indispensable Rules
• Brendan Gregg, Systems Performance & BPF Performance Tools (perf, eBPF, flame graphs) — https://www.brendangregg.com/
• gdb, strace, ltrace, valgrind, perf man pages & tutorials
• AddressSanitizer/ThreadSanitizer (Phase 02) — https://clang.llvm.org/docs/AddressSanitizer.html
• bpftrace / bcc (eBPF tracing) — https://github.com/iovisor/bpftrace
• Release It! (Michael Nygard) — circuit breaker, bulkhead, stability patterns
• Netflix Chaos Engineering / "Principles of Chaos" — https://principlesofchaos.org/
• Google SRE Book/Workbook — postmortem culture, the blameless postmortem template — https://sre.google/books/
• Cross-track: Security Engineer — incident response & detection