Interview Bank 04 — Reliability, Debugging & Security

Phases 09–11. The JD's most-weighted theme (debugging/RCA) plus reliability patterns, secure boot/ tenancy, and CI/CD/release. Expect a "production is broken" scenario — narrate the method.

Q1. A production node reboots intermittently under load — debug it, you can't take it offline.

Method, narrated: (1) scope the failure domain (just this node or peers on the same PDU/ToR/CDU?); (2) evidence-first via OOB since the host is unreliable — SEL + BERT (last-crash cause), the RAS parser on AER/MCE, telemetry correlating reboots with temperature/power/load; (3) ask "what changed?" (recent firmware/deploy); (4) hypothesize and test cheaply — e.g., MCE points at a DIMM with rising correctable ECC that climbs with temperature → marginal DIMM under thermal load; reproduce on a canary; (5) fix — cordon/drain, RMA the DIMM, re-validate; (6) prevent — add an alert + test, write the blameless RCA. The method is the answer. (P10 Ch. 1–3; Q1.)

Q2. A daemon is at 100% CPU / a process is hung — what do you do?

100% CPU → perf top / flame graph shows where the cycles go (no guessing); cross-check strace and metrics. Hung → strace -p <pid> shows the exact blocked syscall (a futex = lock/deadlock → gdb -p thread backtraces; a read/connect = waiting on a dead dependency with no timeout → the bug is the missing timeout). For production, prefer eBPF/bpftrace to peek without restarting. (P10 Ch. 2; Q2–3.)

Q3. Suspect memory corruption or a race — which tools?

Memory: ASan/valgrind give the exact use-after-free/overflow with allocation+use sites (the crash is usually far from the cause). Race: TSan detects the unsynchronized accesses deterministically without the race needing to manifest (races are Heisenbugs — don't chase them with logging, which changes timing). Make nondeterminism deterministic (inject clock/scheduling) and add the sanitizer to CI. (P10 Ch. 4; Q5.)

Q4. What's a circuit breaker and why do you need one with retries?

After N failures to a dependency it "opens" (fail fast for a cooldown), then "half-opens" to probe, then "closes" on recovery. You need it with retries because retries alone cause a retry storm — every client hammering a down BMC prevents recovery and cascades. Backoff+jitter spread load; the breaker stops calling a clearly-dead dependency and probes for recovery. With idempotency and timeouts, it's the core of resilient calls to flaky hardware. (P10 Ch. 5; Q4.)

Q5. How do you run a postmortem after a serious incident?

Blamelessly, with tracked outcomes: timeline, impact (SLO/error-budget burned), detection (and how it should have been sooner), root cause via the 5 Whys to the systemic cause (not "the node OOMed" but "unbounded cache, no memory limit, no test"), and — most important — owned, dated action items (a regression test, an alert/runbook, a guard, a design change), tracked to done. Blameless because if a human could trigger it, the system allowed it. The deliverable is an org that won't have that class of incident again. (P10 Ch. 7; Q6.)

Q6. How do you isolate tenants on a shared rack, and why is the management plane special?

VLAN-per-tenant (802.1Q) for broadcast-domain isolation; on hosts, namespaces+veth+bridges+VLAN sub- interfaces; firewall ACLs deny cross-tenant; QoS (HTB) prevents noisy neighbors with a reserved management class. The management plane (BMC/PDU/CDU) is a hard boundary — its own VLAN, control- plane-only, never tenant/internet-facing — because those devices have total power/firmware control; an exposed BMC is a full compromise. Defense in depth: no single control is load-bearing. (P09 Ch. 2, 8; Q1.)

Q7. Secure boot vs measured boot — difference, and what's attestation for?

Secure Boot enforces: refuses to run unsigned/untrusted boot code (firmware→shim→GRUB→kernel, db/dbx, PK/KEK) — stops bootkits. Measured boot measures: each stage hashes the next into a TPM PCR (PCR = Hash(PCR || measurement), extend-only) — records exactly what booted. Remote attestation: the node returns a signed PCR quote; a verifier checks it against known-good values before trusting the node (cluster join, key release, tenant work). Essential for multi-tenant/sovereign deploys. (P09 Ch. 5–7; Q3–5.)

Q8. How do you test software that talks to hardware you don't have in CI?

The seam + emulators + HIL: program against a device interface so unit tests run against fakes; run emulators (Redfish mock, ipmi_sim, snmpsim) as CI service containers for integration tests; keep a small HIL pool behind a gated stage for what only hardware can prove (vendor conformance, firmware, timing). The pyramid is wide at the base (fakes), thin at the top (HIL). Testability is the seam, decided up front. (P11 Ch. 1–3; Q1.)

Q9. How do you safely roll out a new version to 10,000 nodes?

A gated, canaried, reversible, domain-aware rollout: sign+version the artifact; canary (1 node → 1%), bake, gate promotion on health/SLOs; progress 1%→10%→50%→100%, each gated, with automatic image-pinned rollback on regression; failure-domain-aware waves; drain-first for reboots; backward/forward-compatible APIs (many versions run at once). Same discipline as firmware (P08) and k8s upgrades (P06). A bad change caught at 1% is an annoyance; at 100% an outage. (P11 Ch. 6; Q3.)

Q10. What do you look for in a code review?

Priority: correctness (incl. failure paths) > security (injection, validation, secrets) > tests + a seam > operability (logs/metrics/errors) > style (let the linter win). Review = knowledge-sharing + mentoring. And every production incident should add a test/CI gate (RCA → gate) — the continuous improvement the JD names. (P11 Ch. 7; Q5.)

Quick-fire drills

  • Debugging is a method: reproduce → observe → "what changed?" → hypothesize → bisect → fix → prevent.
  • Tool↔symptom: gdb (state), strace (stuck), perf (slow), ASan/TSan (memory/race), eBPF (prod).
  • timeouts + retry+backoff+jitter + circuit breaker + idempotency + reconciliation + bulkheads.
  • Blameless RCA; 5 Whys; owned, dated action items.
  • Management VLAN = hard boundary; tenants never reach it.
  • Secure boot = enforce; measured boot = record; attestation = prove → gate trust.
  • No hardware in CI → seam + emulators + HIL pyramid.
  • Fleet rollout = canary + health gates + auto-rollback + domain-aware (same as firmware/k8s).