Warmup Guide — Memory, Parsers, Binaries, and the Kernel Boundary

Zero-to-expert primer for Phase 01. It builds, from first principles, the systems knowledge a security engineer needs below framework level: the C abstract machine and undefined behavior, the memory-defect families, integer arithmetic as the source of length bugs, parser state machines and differentials, executable formats (ELF/PE/Mach-O), the loader and the syscall boundary, and what Rust and Go actually remove. Assumes only basic programming. By the end you should be able to write a parser that cannot be tricked, read a crash and reason about its root cause, and explain why a memory-safe service can still be security-critical.

Table of Contents

Chapter 1: Where Security Bugs Live — The One-Line Model

Zero background. A memory-safety vulnerability is not magic; it is a place where the program's assumptions about bytes diverge from the actual bytes an attacker supplied. Almost every low-level security bug fits one mental model:

untrusted bytes → length/offset arithmetic → allocation → object lifetime → privileged action

Each arrow is a place an assumption can break: bytes get interpreted as a structure that doesn't match; arithmetic on a length wraps or underflows; an allocation is the wrong size; an object is used after its lifetime ends; a privileged operation trusts a value it shouldn't. The entire phase is about making each arrow provably correct.

Why this matters for a security engineer specifically. You are not just trying to write correct code — you are trying to write code whose correctness survives an adversary choosing the worst possible input. A function that works for every "reasonable" input and corrupts memory on one crafted input is, to an attacker, a feature. Your job is to think in terms of what input maximizes damage, then prove that input is rejected.

Misconception to kill now. "Memory bugs are a solved problem; just use a safe language." Safe languages remove whole classes of bugs (Chapter 11), but (a) the world runs on enormous C/C++ codebases you will audit, (b) safe languages still have logic bugs, resource-exhaustion bugs, and unsafe/FFI escape hatches, and (c) understanding why the bugs exist is what lets you design the safe replacement. You cannot defend a boundary you don't understand.

Chapter 2: The C Abstract Machine and Undefined Behavior

The key idea. C is not "portable assembly." The C standard defines an abstract machine with rules, and the compiler is only obligated to make your program behave correctly if you obey the rules. Break a rule and you hit undefined behavior (UB): the standard imposes no requirements at all. The program may crash, may produce wrong output, or may appear to work — and crucially, the optimizer is allowed to assume UB never happens and delete the code that would have handled it.

Pointer provenance. A C pointer is valid only for a specific object, with a specific lifetime, alignment, and set of permitted operations. An address that merely happens to be mapped in the process is not automatically a valid C pointer. Indexing past the end of an array, dereferencing a freed pointer, or reading uninitialized memory are all UB even if the underlying memory page is readable.

Why UB is a security problem, not just a correctness problem. Consider:

char buf[8];
if (idx < 0) return ERR;          // signed check
buf[idx] = x;                      // attacker controls idx

If the optimizer can prove that an out-of-range idx would be UB, it may assume idx is in range and remove a redundant-looking bounds check elsewhere. UB doesn't just let the bad thing happen; it can erase the very checks you wrote to prevent it. This is why "it compiled and ran on my machine" proves nothing about safety.

Production significance. Auditors read C for UB constantly: signed-overflow assumptions, strict-aliasing violations, out-of-bounds, use-after-free, and memcpy with a tainted length. Tools that find UB (UBSan, ASan — Chapter 12) are how you turn "looks fine" into "proven on this input."

Misconception to kill now. "If it doesn't crash, it's fine." UB that doesn't crash is the more dangerous case — it's a latent corruption an attacker can steer.

Chapter 3: Stack, Heap, and the Memory-Defect Families

Stack vs heap — what they actually are. The stack is a region the CPU grows and shrinks automatically as functions call and return; each call pushes a stack frame holding local variables, saved registers, and the return address (where to resume after the function returns). The heap is memory you request explicitly (malloc/new) and must release (free/delete); an allocator tracks free and used chunks, often with metadata stored inline between chunks. Stack and heap are allocation strategies, not security levels — both can be corrupted.

Why a stack overflow is dangerous. A write past the end of a local array can overwrite the saved return address or adjacent locals. Historically this enabled redirecting execution; today, mitigations (Chapter 10) make it harder but the corruption is still a bug and often still exploitable.

The memory-defect families — the vocabulary you must own:

  • Out-of-bounds read/write (OOB). An index, pointer, or length escapes the object's bounds. Reads leak adjacent data; writes corrupt it.
  • Use-after-free (UAF) / double-free. A pointer is used after the object's lifetime ended. Dangerous because the allocator may have handed those bytes to a new owner — now two pieces of code disagree about what lives there.
  • Integer overflow / truncation. The validated number differs from the number actually used to allocate or copy (Chapter 4).
  • Uninitialized read. Stale memory contents influence output or control flow — can leak secrets or create nondeterministic logic.
  • Race / TOCTOU (time-of-check-to-time-of-use). The object checked is not the object used, because another thread or process changed it in between.
  • Type confusion. Memory is interpreted as the wrong type (common in C++ with bad casts), so a field meant to be data is treated as a pointer or vtable. (A vtable is the hidden table of function pointers C++ uses to dispatch virtual methods; if an attacker controls what an object's vtable pointer points at, they choose which function the next virtual call invokes.)

Production significance. When you triage a crash (Phase 08), the first job is to classify it into one of these families, because the family determines exploitability reasoning and the right fix.

Misconception to kill now. "A read OOB is harmless." Information-disclosure OOB reads (Heartbleed being the canonical example) leak keys and tokens and are frequently critical.

Chapter 4: Integer Arithmetic Before Memory Arithmetic

The insight that prevents most length bugs. Memory corruption usually starts in integer math, before any memcpy. The classic mistake:

size_t bytes = count * element_size;     // can overflow / wrap
if (bytes > remaining) return ERROR;     // validates the WRAPPED value
memcpy(dst, src, bytes);

If count * element_size overflows size_t, it wraps to a small number; the check passes; but the intended size was huge. The fix checks the operands before multiplying:

if (element_size != 0 && count > SIZE_MAX / element_size) return ERROR;  // no overflow possible
size_t bytes = count * element_size;
if (bytes > remaining || bytes > configured_limit) return ERROR;          // bounded

The other integer traps:

  • Signed→unsigned conversion. Converting a negative int to size_t yields a gigantic value. A check like if (len > max) passes the wrong way if len was negative and got widened.
  • Truncation. Storing a 64-bit length in a 32-bit field silently drops the high bits.
  • Underflow. remaining - field_len underflows to a huge number if you didn't first prove field_len <= remaining.
  • Implicit promotion. C promotes small types to int in arithmetic; the resulting type and signedness can surprise you. Know the exact types in the expression.

Production significance. "Validate lengths before you trust them, and check operands before the operation" is the single highest-leverage habit in C/C++ auditing. A large fraction of CVEs in parsers reduce to one missing operand check.

Misconception to kill now. "I checked the result is within bounds." Checking the result of an overflowing operation validates the wrapped value, not the intended one. Check the inputs.

Chapter 5: Parsers — Parse, Don't Guess

Why parsers are the front line. A parser is where untrusted bytes first become structured data. Every byte that later flows into a length, an offset, an identity, or a privileged action came through a parser. A parser that "guesses" — accepts ambiguous input and does its best — hands that ambiguity to everything downstream.

The eight rules of a secure parser (the flagship TLV — Tag-Length-Value — lab enforces these):

  1. Bound the total input before parsing anything.
  2. Verify a complete fixed header exists before indexing into it.
  3. For each field, check its length against both a per-field limit and the remaining bytes (length <= max AND length <= remaining).
  4. Reject duplicate singleton / security fields (e.g. two "tenant identity" fields).
  5. Reject unknown critical fields (fields the spec says must be understood).
  6. Decode text strictly (reject invalid UTF-8 rather than substituting).
  7. Consume exactly one message — no trailing bytes left unexplained.
  8. Return structured errors that say what failed without echoing the raw input back.

Why rule 3 needs both clauses. length <= max alone allows a field that claims a legal size but extends past the buffer — an OOB read. length <= remaining alone allows a field within the buffer but larger than the protocol permits — a resource or logic abuse. You need both.

Why rules 4 and 7 are about security, not tidiness. A duplicate identity field lets two components pick different identities (Chapter 6). Trailing bytes let one component stop after message 1 while another reads a hidden message 2. Both are parser differentials — the most underappreciated bug class in protocol security.

Production significance. Length-prefixed and TLV formats are everywhere: TLS records, protobuf, ASN.1/X.509 certificates, image and font files, network protocols. The discipline here is exactly what you'll apply auditing any of them.

Chapter 6: Parser Differentials as Vulnerabilities

Zero background. A parser differential occurs when two components that process the same bytes disagree about what those bytes mean. Security breaks at the disagreement.

Anatomy. A parser has a cursor (position), remaining length, grammar state, a resource budget, and semantic context. Four levels of validation:

  • Syntactic: the bytes form a structurally valid message.
  • Semantic: the message is meaningful in this transaction (right type, right identity).
  • Canonicalization: there is one chosen representation (no two encodings of the same thing).
  • Exact consumption: no second interpretation hides after the first.

The classic exploits this prevents:

  • HTTP request smuggling: a front-end proxy and a back-end server disagree about where one request ends (e.g. Content-Length vs Transfer-Encoding), so the attacker smuggles a second request past the proxy's authorization.
  • Certificate/identity confusion: a signature verifier and an application disagree about which field is the subject identity, so a signed-but-attacker-chosen name is trusted.
  • Cache poisoning: a cache keys on one interpretation of the request while the origin serves another.

The defense, stated once: define a single policy for duplicate fields, whitespace, encodings, integer forms, unknown-critical fields, and trailing data — and make every component on the path enforce the same policy. When the flagship lab makes you fuzz a C and a Rust implementation of the same TLV format and compare accept/reject results, any disagreement is a finding — that's a parser differential in miniature.

Misconception to kill now. "It's just strictness for its own sake." No — leniency in a parser is the attack surface. "Be liberal in what you accept" is a robustness maxim that has cost the industry a thousand CVEs.

Chapter 7: Executable Formats and the Loader

Why a security engineer reads binary formats. You will analyze programs you don't have source for, verify hardening flags, and understand what the loader trusts. The three formats:

  • ELF (Linux/Unix) — Executable and Linkable Format.
  • PE (Windows) — Portable Executable.
  • Mach-O (macOS/iOS) — Mach object format.

All three describe the same things in different layouts: segments/sections (code, data), symbols (named functions/variables), imports (external functions to resolve), relocations (addresses to fix up at load time), permissions (read/write/execute per region), and an entry point.

What the loader actually does. When you run a program, the loader: maps the executable's segments into memory, assigns each the right permissions, applies relocations, loads shared libraries it depends on, sets up runtime state, and jumps to the entry point. Each of these is a trust decision — e.g. where it finds libraries (search path) is a classic hijack surface.

The hardening questions you ask of any binary (the binary-hardening lab makes you extract these without shelling out):

  • Is any writable memory also executable (W^X violation)? Writable+executable pages are an attacker's dream.
  • Is the binary position-independent (PIE) so ASLR can randomize it?
  • Are stack canaries, RELRO (read-only relocations), and CFI present?
  • Which libraries and symbols does it import — i.e., how big is the attack surface?

Production significance. "Is this build hardened?" is a real question in supply-chain review (Phase 03) and incident response (Phase 09). Reading the format yourself, rather than trusting a tool's summary, is how you answer it precisely.

Chapter 8: The Syscall Boundary and File Descriptors as Capabilities

Zero background — the user/kernel boundary. Application code runs in user mode with limited privilege. To touch files, processes, memory, devices, or the network, it must ask the kernel via a system call (syscall) — a controlled transition into privileged mode. The syscall interface is the security boundary of an OS: everything dangerous a process can do, it does through a syscall.

Why this matters for sandboxing (foreshadowing Phase 06). If you can restrict which syscalls a process may make (seccomp), you shrink what a compromised process can do — even if its code is fully controlled by an attacker. The syscall tracer and syscall-policy labs make you observe the actual syscalls a program needs, so you can write a least-privilege allowlist.

File descriptors are capabilities — the crucial idea. A file descriptor (an integer handle to an open file, socket, directory, or device) is an unforgeable token of access. A process that inherits an already-open descriptor to a secret file or a socket can use it without ever calling open — so an allowlist that blocks open but ignores inherited descriptors is incomplete. This is why you review:

  • execve and what descriptors survive it (CLOEXEC flag closes them on exec — its absence leaks capabilities to child processes),
  • the environment and current working directory (both influence what a child trusts),
  • signal handlers and child cleanup.

Production significance. "What can this process actually do?" is answered by enumerating its syscalls and its inherited descriptors, environment, and namespaces — not just its source code.

Misconception to kill now. "Dropping privileges means dropping the user ID." Privilege is the union of UID, capabilities, namespaces, open descriptors, and syscall access. Lower the UID but leave a root-owned socket open and you've dropped nothing that matters.

Chapter 9: CPU Architecture — Registers, Calling Conventions, Control Flow

Why a defender reads assembly. To triage a crash, understand a patch diff, or reverse a binary (Phase 08), you must read a little machine code. You don't need to write assembly; you need to reconstruct what a function does from it.

Registers. Small, fast CPU storage. On x86-64, general-purpose registers include rax, rdi, rsi, rdx, etc.; rsp is the stack pointer, rip the instruction pointer. On ARM64 (AArch64), x0–x30 are general-purpose, sp the stack pointer, lr (x30) the link register holding the return address.

Calling conventions (ABIs). The agreed rules for how functions pass arguments and return values. On the System V AMD64 ABI (Linux/macOS), the first integer arguments go in rdi, rsi, rdx, rcx, r8, r9, return value in rax. On AArch64, arguments go in x0–x7, return in x0. Knowing this lets you read "what was passed to this function" straight from a crash dump.

Stack frames and control flow. A call pushes the return address; the function may set up a frame; ret pops the return address into the instruction pointer. The return address living on the stack is exactly why stack buffer overflows historically hijacked control — and why stack canaries and shadow stacks exist (Chapter 10). Reconstructing control flow means following the conditional branches (jne, b.eq) to rebuild the if/else and loop structure.

Production significance. The toy-debugger and syscall-tracer labs make this concrete: you set breakpoints, read registers, and watch arguments flow into syscalls. That is the same skill applied at scale in a decompiler.

Chapter 10: Compiler and Runtime Hardening

These mitigations don't make bugs correct; they make them harder to exploit. Know what each does and what it does not cover:

  • ASLR (Address Space Layout Randomization) — randomizes where code/stack/heap land, so an attacker can't hardcode addresses. Defeated by an information leak that reveals a real address.
  • NX / DEP / W^X — writable pages are non-executable, so injected data can't run as code. Pushed attackers toward reuse of existing code (ROP — return-oriented programming).
  • Stack canaries — a random value placed before the return address; corrupted by an overflow and checked on return. Detects contiguous stack overwrites, not targeted writes.
  • RELRO (RELocation Read-Only) — makes the relocation and GOT tables read-only after startup. (The Global Offset Table is the per-process table of resolved addresses for imported library functions; while it's writable, an attacker who can write memory overwrites one entry to redirect a library call to code of their choosing — RELRO closes that classic function-pointer overwrite.)
  • CFI (Control-Flow Integrity) — constrains indirect branches to valid targets, blunting ROP and vtable hijacks.
  • Pointer Authentication (PAC) on ARM64 — signs selected pointers (e.g. return addresses) so tampering is detected.

The mental model: these raise the cost and reduce the reliability of exploitation; an attacker defeats them by chaining primitives (a leak to beat ASLR, then a write to beat the rest). When you assess a binary, you note which are present because that shapes the realistic risk — but you never downgrade a memory-corruption bug to "won't fix" just because mitigations are on.

Chapter 11: What Rust and Go Remove — and What Remains

Rust. The borrow checker enforces, at compile time, that references obey ownership and lifetime rules: no two mutable aliases, no use-after-free, no data races in safe code. This eliminates the bulk of Chapter 3's families for safe code. But:

  • unsafe shifts the proof burden to you. An unsafe block or FFI boundary can violate every guarantee. (FFI — a Foreign Function Interface — is where safe code calls into C or another language; the moment you cross it you inherit all of C's hazards, with none of the borrow checker's protection, so the boundary is exactly where memory-safety bugs re-enter a "safe" program.) A safe wrapper around unsafe is sound only if no safe caller can break the invariants the unsafe code relies on (pointer provenance, length/capacity, initialization, alignment, ownership, thread-safety). The Rust capability-token lab is partly about stating those invariants precisely.
  • Logic bugs remain. Rust won't stop you authorizing the wrong tenant.

Go. Memory-safe with garbage collection — no manual free, no UAF. But:

  • Slices alias their backing array — mutating one view mutates another.
  • Goroutines can race on shared state (run the race detector).
  • Typed-nil interfaces create surprising non-nil-looking nils.
  • Unbounded goroutines/allocations are denial-of-service paths — concurrency must be bounded.

The honest framing for interviews: safe languages eliminate memory-corruption classes; they do not eliminate authorization bugs, resource exhaustion, parser differentials, or unsafe/FFI hazards. The Go security-tool lab (bounded workers) and the secure-HTTP-service lab (limits, timeouts) exist precisely to practice the remaining hazards.

Misconception to kill now. "Rewrite it in Rust and it's secure." Rust removes a category; it does not remove the need for input validation, authorization, and resource limits.

Chapter 12: Fuzzing, Sanitizers, and Crash Triage

Fuzzing — what and why. A fuzzer feeds large quantities of malformed/random input to a target and watches for crashes. Coverage-guided fuzzers (libFuzzer, AFL++) instrument the binary to measure which code paths an input reaches and evolve inputs toward new coverage — a guided search through the input space far more efficient than random.

Sanitizers — turning silent corruption into a loud crash. Compiler instrumentation that detects UB at runtime:

  • ASan (AddressSanitizer) — OOB and use-after-free, with the exact offending access.
  • UBSan — undefined behavior (signed overflow, bad shifts, misaligned access).
  • MSan (MemorySanitizer) — uninitialized reads.
  • TSan (ThreadSanitizer) — data races.

Fuzzer + sanitizer is the workhorse pair: the fuzzer reaches the bug, the sanitizer proves it.

Crash triage — the chain you must separate (this is Phase 08's core skill, introduced here):

crash  →  reachability  →  controllability  →  boundary crossed  →  impact

A crash means the program died. Reachability means an attacker can reach the code. Controllability means they influence the corrupted value/address. Boundary crossed asks whether a privilege or trust line was actually crossed. Impact is the real consequence. Confusing "it crashed" with "it's exploitable" is the most common triage error — a null-pointer crash and a controllable heap write are worlds apart.

Minimization. Reduce a crashing input to the smallest input that still triggers the same root cause — this clarifies the bug and makes the regression test precise.

Production significance. Every defect you find gets a fix and a regression test built from the minimized input, so the bug can never silently return. This fix-plus-regression discipline is the difference between patching a symptom and closing a bug class.

Lab Walkthrough Guidance

Recommended order (it builds capability incrementally):

  1. Lab 01 — Bounded Binary Parser and Fuzz Target. Implement the eight rules (Chapter 5). Then run the deterministic smoke fuzzer; then port the same TLV format to C and Rust and compare accept/reject results — any disagreement is a parser differential (Chapter 6).
  2. Lab 04 — C Memory Corruption: Break, Diagnose, Fix, Defend. Build the small owned bug corpus (OOB, lifetime, integer, format), reproduce under ASan/UBSan, then fix and add regression tests. This makes Chapters 2–4 tactile.
  3. Lab 02 — Binary Hardening Report Analyzer. Parse ELF headers/sections/symbols and extract the hardening flags (Chapter 7) without shelling out.
  4. Lab 03 — Syscall Policy Synthesizer. Observe the syscalls a program actually makes and synthesize a least-privilege allowlist (Chapter 8) — the on-ramp to Phase 06 seccomp.
  5. Lab 05 — Rust Capability Token and Replay Defense. A memory-safe capability with tenant, expiry, action, and replay invariants — practice stating invariants precisely (Chapter 11).

Run every lab against both your code and the reference:

pytest test_lab.py -v
LAB_MODULE=solution pytest test_lab.py -v   # Python labs

For the C and Rust components, compile with warnings-as-errors and sanitizers, and run the fuzz smoke target.

Success Criteria

You are ready for Phase 02 when you can, without notes:

  1. Explain why checking length <= max without also checking length <= remaining is insufficient.
  2. Show how an integer overflow before a memcpy defeats a size check, and write the correct operand check.
  3. Distinguish crash → reachability → controllability → boundary → impact on a concrete bug.
  4. State the safety invariants an unsafe Rust wrapper must uphold for a safe caller.
  5. Explain stack vs heap, virtual vs physical memory, and the user→kernel syscall transition from a trace.
  6. Read a small x86-64 and ARM64 function and reconstruct its control flow.
  7. Explain why a memory-safe service can still be security-critical (authorization, resources, differentials).
  8. Minimize a crash, identify root cause, patch it, and prove regression coverage.

Common Mistakes

  • Parsing before validating lengths; trusting attacker-supplied file offsets.
  • Checking the result of overflowing arithmetic instead of the operands.
  • Unbounded concurrency or allocation (denial of service in "safe" languages).
  • Building subprocess commands by string concatenation (command injection).
  • Swallowing errors; failing open on malformed input.
  • Confusing a crash with exploitability; reporting sanitizer output without root-cause analysis.
  • Claiming a safe language "eliminates" bugs without scoping the claim to memory-safety classes.

Interview Q&A

Q: Design a safe parser for an untrusted length-prefixed format. A: Bound total input first; verify the full fixed header exists before indexing; for each field check length <= field_max AND length <= remaining; reject duplicate singleton/security fields and unknown critical fields; decode text strictly; consume exactly one message and reject trailing bytes; return structured errors that don't echo the input. I'd implement it as an explicit state machine over a cursor with a remaining-length and a resource budget, and fuzz it with sanitizers.

Q: What makes use-after-free possible, and how do ownership systems change the risk? A: A pointer outlives the object it points to; after free, the allocator may reissue those bytes to a new owner, so the stale pointer and the new owner alias the same memory — reads leak, writes corrupt, and a freed-then-reallocated object can be type-confused. Rust's borrow checker makes this a compile error in safe code by tying a reference's lifetime to the owner; the risk moves to unsafe/FFI, where you must uphold the invariants manually.

Q: Walk from a function call to a syscall and back. A: The caller places arguments per the ABI (e.g. rdi, rsi, … on SysV x86-64), call pushes the return address and jumps. To do anything privileged, the function invokes a syscall: arguments go in the syscall-ABI registers, a trap instruction transitions to kernel mode, the kernel validates and performs the operation, returns a result in a register and switches back to user mode; the function reads the result and eventually ret pops the return address to resume the caller.

Q: How would you fuzz a stateful protocol parser? A: Model the protocol as a sequence of messages with state; fuzz at the message-sequence level (not just single buffers) using a structured/grammar-aware harness so the fuzzer reaches deep states; seed a corpus of valid exchanges; add a dictionary of tokens; run coverage-guided with sanitizers; and compare two independent implementations to surface differentials. Minimize crashes and turn each into a regression test.

Q: Why can a memory-safe service still be security-critical? A: Memory safety removes corruption classes, not authorization flaws, parser differentials, resource-exhaustion DoS, secret handling, or unsafe/FFI hazards. A Go or Rust service that authorizes the wrong tenant, accepts an ambiguous request a downstream peer reads differently, or spawns unbounded goroutines is fully exploitable despite being memory-safe.

References

Books

  • Randal Bryant & David O'Hallaron, Computer Systems: A Programmer's Perspective (memory, linking, exceptions, the machine).
  • Remzi & Andrea Arpaci-Dusseau, Operating Systems: Three Easy Pieces (processes, virtual memory, concurrency — free online).
  • Robert Seacord, Effective C and Secure Coding in C and C++; SEI CERT C/C++ Coding Standard.
  • The Rustonomicon (unsafe Rust) and the Rust Unsafe Code Guidelines.

Specifications

  • System V AMD64 ABI; Arm AArch64 ABI (procedure call standard).
  • ELF and DWARF specifications; Microsoft PE/COFF specification; Apple Mach-O reference.

Fuzzing and sanitizers

  • The Fuzzing Book (Zeller et al., free online).
  • LLVM libFuzzer documentation; AFL++ documentation.
  • AddressSanitizer / UBSan / MSan / TSan documentation (Clang/LLVM).

Foundational reading

  • Aleph One, "Smashing the Stack for Fun and Profit" (historical, for the stack-frame mental model).
  • "The Heartbleed Bug" (CVE-2014-0160) — the canonical OOB-read information-disclosure case study.