Warmup Guide — Software Supply Chain Security at Scale

Zero-to-expert primer for Phase 14 (elite/principal track). The attacker's highest-leverage move is no longer to breach your app — it is to compromise the build or a dependency and have you ship their code to every customer through your trusted update channel (SolarWinds, event-stream, xz/ CVE-2024-3094, dependency-confusion). This phase builds, from first principles, the full supply-chain defense: build provenance and SLSA, in-toto attestations, reproducible and hermetic builds, keyless signing and transparency logs (Sigstore/Fulcio/Rekor), binary authorization, dependency security (confusion, typosquatting, malicious packages, lockfile integrity), SBOM and VEX, and package-registry/build-system hardening. Assumes Phase 03. By the end you should understand how each control is implemented and be able to build the verifiers in the labs.

Table of Contents

Chapter 1: The Supply Chain Is the Attack Surface

Zero background. Modern software is assembled, not written: your application is a thin layer over thousands of third-party packages, built by automated pipelines, distributed through registries and update servers. Each of those — a dependency, a build server, a signing key, a package registry — is a link, and an attacker who compromises any link can inject code that you then ship, signed and trusted, to every customer. This is fundamentally different from app security: the attacker never touches your running service; they poison what goes into it.

Why this is the defining security problem of the decade. The landmark incidents are all supply-chain: SolarWinds (a compromised build server inserted a backdoor into a signed update shipped to 18,000 organizations), event-stream (a maintainer handed off a popular npm package to an attacker who added a wallet stealer), xz/CVE-2024-3094 (a multi-year social-engineering campaign planted a backdoor in a core Linux compression library's release tarballs), and dependency-confusion (publishing a public package with an internal name and a higher version so build systems fetch the attacker's copy). FAANG and critical-OSS teams now treat the build and the dependency graph as first-class attack surface — because a single supply-chain compromise has the blast radius of every downstream user at once.

Misconception to kill now. "We scan our dependencies for CVEs, so our supply chain is covered." CVE scanning finds known-vulnerable versions; it does nothing against a maliciously injected dependency, a compromised build, a typosquat, or a backdoor that has no CVE. Supply-chain security is about integrity and provenance — "did this come from where I think, unmodified, built how I think" — not just known-vuln hygiene.

Chapter 2: A Threat Model for the Build

Map every link. The canonical supply-chain threat model (SLSA's) walks the path from source to consumer and asks what can be compromised at each step:

developer → source repo → dependencies → build system → artifact → registry → consumer
   (A) bad code   (B) source    (C) poisoned   (D) compromised   (E) tampered   (F) malicious
       committed      tampering      dependency     builder         artifact        registry/mirror
  • (A) Source — a malicious commit, or a compromised maintainer account. Defense: two-person review, signed commits, branch protection.
  • (B) Source integrity — the build pulls different source than what was reviewed. Defense: provenance binds the artifact to a specific commit.
  • (C) Dependencies — an injected, confused, typosquatted, or backdoored dependency (Chapter 8).
  • (D) Build system — the builder itself is compromised (SolarWinds) or a build step exfiltrates/ modifies (Chapter 5 — isolation, hermeticity).
  • (E) Artifact — the binary is swapped after build. Defense: signing + verification.
  • (F) Distribution — a malicious or mirrored registry serves a different artifact. Defense: digests, transparency logs.

The unifying defense: verifiable provenance + verify-at-consumption. For every link you want a signed, machine-checkable statement of what happened, and a consumer who verifies it before trusting the artifact. That is what the rest of this phase builds.

Misconception to kill now. "Our build pipeline is internal, so it's trusted." The build pipeline is the most valuable target — it holds the keys to ship trusted code to every user. Compromising it yields a backdoor with perfect provenance. The build is in scope, hardened, and isolated.

Chapter 3: Provenance and in-toto Attestations

What provenance is. Build provenance is verifiable metadata answering: what artifact, built from which source (repo + commit), by which builder, using which inputs (the resolved dependencies, by digest). It is emitted by the build system as a signed attestation.

in-toto — the framework. in-toto is the standard for supply-chain attestations: a signed statement with a subject (the artifact, by digest) and a predicate (the claim — e.g. a SLSA provenance predicate describing the build). The DSSE (Dead Simple Signing Envelope) format wraps and signs it. The power is composition: you build an attestation graph — source-review attestation, build-provenance attestation, test attestation, scan attestation — and a policy engine verifies the chain end to end.

Materials reconciliation — the part most people miss. The single most security-relevant field is materials: the exact dependencies the build consumed, by digest. A real verifier reconciles them against your reviewed lockfile — every material must be in the lockfile, by matching digest, with nothing extra (an injected dependency) and nothing missing (the build used something other than what you locked). This is exactly verify_provenance in Lab 01. Field-matching the builder and source is table stakes; reconciling materials is where dependency-injection attacks are caught.

Misconception to kill now. "We sign our artifacts, so provenance is handled." A signature says who signed; provenance says how it was built and from what. You need both — and you need to verify the materials, not just the signature.

Chapter 4: SLSA — The Levels and What Each Buys

SLSA (Supply-chain Levels for Software Artifacts) is a graded framework — a target maturity for build integrity. The build track levels (v1.0, simplified — this is achieved_slsa_level):

  • L1 — Provenance exists. The build produces provenance identifying the builder and build process. Buys: basic transparency; catches accidental mistakes. Does not stop a determined attacker.
  • L2 — Signed, service-generated provenance. Provenance is signed and produced by a hosted build service (not a developer laptop). Buys: tamper-evidence and a known builder identity.
  • L3 — Hardened, isolated builder; non-falsifiable provenance. The build runs on an isolated, hardened platform where no build step can forge the provenance (the provenance generator is outside the build's control). Buys: defense against a compromised build step — the SolarWinds class. L3 is the practical bar for production-critical software.
  • L4 (stricter internal bar) — Hermetic + reproducible. All inputs declared, no network at build time, and the build is bit-for-bit reproducible by an independent rebuilder (Chapter 5). Buys: the strongest assurance that the artifact corresponds to the source.

How you enforce a level. A policy says "deploy only artifacts with provenance at SLSA ≥ 3 from an allowed builder, for the expected source repo, with materials matching the lockfile." The verifier computes the achieved level from the provenance's properties and refuses anything below the bar — the heart of Lab 01.

Misconception to kill now. "We're SLSA L4 because we sign things." Levels are about build integrity properties (service-generated, isolated, non-falsifiable, hermetic, reproducible), not about how fancy your signing is. Most orgs are L1–L2 and think they're higher.

Chapter 5: Hermetic and Reproducible Builds

Hermetic builds. A hermetic build declares all its inputs up front and runs with no network access — it cannot fetch an undeclared dependency, a "curl | bash" install, or a moving "latest". Every input is pinned by digest. This closes an entire attack class: a build that can reach the network can be made to pull attacker-controlled content at build time (a poisoned mirror, a compromised CDN, a typo'd URL). Bazel and Nix are the canonical hermetic build systems.

Reproducible builds. A build is reproducible if anyone, given the same source and declared inputs, can independently produce a bit-for-bit identical artifact. This is profound: it means the binary can be independently verified to correspond to the source — defeating a compromised builder that injects code, because an independent rebuilder would get a different binary. Reproducibility requires eliminating non-determinism: timestamps (SOURCE_DATE_EPOCH), build paths, file ordering, locale, and embedded randomness. The Reproducible Builds project and Debian's bit-for-bit work are the references.

Why they're the top of the SLSA ladder. Together, hermeticity (no smuggled inputs) and reproducibility (the binary provably matches the source) give the strongest answer to "is this artifact really built from this source, with nothing extra?" — which is the whole question of supply-chain integrity.

Misconception to kill now. "Our builds are deterministic, that's reproducible enough." Determinism on your machine isn't reproducibility — reproducibility means an independent party gets the same bytes. Embedded timestamps and build paths silently break it.

Chapter 6: Keyless Signing and Transparency Logs (Sigstore)

The key-management problem. Traditional artifact signing requires every project to manage a private signing key — which gets lost, leaked, or stolen. Sigstore solves this with keyless signing:

  • Fulcio — a certificate authority that, after you authenticate via OIDC (your GitHub/Google identity), issues a short-lived (≈10-minute) signing certificate bound to that identity. You sign with an ephemeral key; the certificate proves who signed. No long-lived key to steal.
  • Rekor — an append-only transparency log (a Merkle tree — a hash tree where every node is the hash of its two children, so one root hash commits to every leaf and any tampering changes the root) of every signing event. Every signature is publicly recorded, so a maliciously-issued signature cannot be hidden — anyone can audit the log and detect an unexpected signature for their project.

Transparency logs and inclusion proofs — the technical heart (this is Lab 02). A transparency log is a Merkle tree of entries. The log periodically signs its root (a Signed Tree Head). To prove an entry is in the log, the log gives an inclusion proof: an audit path of sibling hashes from the leaf to the root. A verifier recomputes the root from the leaf + path and checks it matches the signed root — proving inclusion without downloading the whole log. RFC 6962 (Certificate Transparency) defines the domain-separated hashing (leaf = H(0x00‖data), node = H(0x01‖left‖right)) that prevents a second-preimage attack where an internal node is presented as a leaf. This is the same mechanism as Certificate Transparency (every TLS certificate is logged so mis-issuance is detectable), applied to software signing.

Misconception to kill now. "Keyless signing means no cryptography / no trust root." It means no long-lived key — the trust root moves to the OIDC identity provider and the transparency log, which are themselves auditable. You still verify a certificate chain and an inclusion proof; you just don't hold a key that can be stolen.

Chapter 7: Binary Authorization — Verify at the Door

Signing is theater without verification. All the provenance and signing in the world changes nothing if no one checks it before running the artifact. Binary authorization is the enforcement point: admission control (Kubernetes — Phase 07) or a deploy gate that verifies, at the moment of deployment, that an image has the required attestations — signed by a trusted identity, with SLSA-≥-N provenance for the expected source, recorded in the transparency log, with materials matching the lockfile — and refuses to run anything that doesn't. Google's Binary Authorization, Sigstore policy-controller, and Kyverno image-verification implement this.

The full picture. Provenance (Chapter 3) is the claim; SLSA (Chapter 4) is the bar; transparency logs (Chapter 6) make it auditable; binary authorization is the enforcement — verify at the door, fail closed. Skip the last step and the rest is decoration.

Misconception to kill now. "We generate provenance and sign images." And does anything verify them before deploy, by digest, against a policy? If not, you have metadata, not a control.

Chapter 8: Dependency Security

The dependency graph is where most real supply-chain compromises happen. The attack classes:

  • Dependency confusion / substitution. You have an internal package acme-utils; an attacker publishes a public acme-utils at a higher version, and your build tool — configured to check the public registry too — fetches the attacker's. Defense: scoped/namespaced packages, explicit registry pinning per package, and refusing public resolution of internal names.
  • Typosquatting. A malicious reqeusts/python-sqlite next to the real package, hoping for a typo or a copy-paste. Defense: allowlists, popularity/age heuristics, and curated internal mirrors.
  • Malicious / hijacked packages. A maintainer account is compromised or sold (event-stream), or a package adds a malicious post-install script. Defense: pinning by digest (a moved version can't silently change), disabling install scripts, behavioral analysis, and the xz lesson — watch for maintainer changes and unusual release artifacts.
  • Lockfile integrity. The lockfile pins exact versions and digests; the build must install exactly the lockfile (the materials reconciliation of Chapter 3). A build that resolves "latest" has no integrity.

The unifying control. Pin by digest, resolve from a controlled source, and reconcile what the build actually consumed against the reviewed lockfile — which is precisely the materials check in Lab 01.

Misconception to kill now. "We pin versions in our lockfile." Versions can be re-published (or a mirror can lie); pin by digest, and verify the digest the build used matches.

Chapter 9: SBOM and VEX

SBOM (Software Bill of Materials). A machine-readable inventory of everything in an artifact — every direct and transitive dependency, by version and ideally digest (CycloneDX / SPDX). Its purpose is incident response velocity: when the next Log4Shell/xz drops, "are we affected, and exactly where?" must be answerable in minutes by querying SBOMs across your fleet, not by a frantic manual audit.

VEX (Vulnerability Exploitability eXchange). An SBOM tells you a vulnerable component is present; VEX tells you whether it is exploitable in this product — not_affected (with a justification, e.g. the vulnerable code path is unreachable), affected, fixed, or under_investigation. VEX is what stops SBOMs from drowning teams in unreachable-CVE noise (the reachability lesson from Phase 03): you ship the SBOM and a VEX statement so consumers know what actually needs action.

Misconception to kill now. "We generate an SBOM, so we're done." An SBOM you can't query across the fleet, kept fresh per build, paired with VEX for exploitability, is a file, not a capability. The value is answering "where are we affected" fast and accurately.

Chapter 10: Package Registries and Build-System Hardening

The registry is critical infrastructure. Your artifact registry / package repository is where the "ship trusted code to everyone" power concentrates. Harden it: require signed/attested artifacts to be pushed and pulled; immutable tags (a published version can't be overwritten); least-privilege publish tokens (short-lived, scoped, OIDC-federated — Phase 07); audit every publish; and serve by digest so a compromised tag can't redirect consumers.

The build system is the crown jewel (Chapter 2 (D)). Treat the CI/build platform as production infrastructure with write access to production: ephemeral isolated runners, no long-lived deploy secrets exposed to untrusted PR code, pinned actions/images by digest, no cross-trust cache reuse, and separation of build identity from deploy/approve identity (Phase 03 Chapter 9). SLSA L3's "non-falsifiable provenance generated outside the build's control" is exactly the hardening that defeats a compromised build step.

Misconception to kill now. "Our registry is internal, so it's safe." The registry and the build system are precisely where an attacker wants to be — one push, and every consumer pulls their code with your name on it. They get the hardest hardening, not the least.

Lab Walkthrough Guidance

The two labs implement the two technical cores of supply-chain verification:

  1. Lab 01 — SLSA Provenance Verifier. Build verify_provenance: subject-digest binding, builder/ source policy, materials reconciliation against the lockfile (the dependency-injection defense), and achieved_slsa_level + minimum-level enforcement (Chapters 3–4, 8).
  2. Lab 02 — Transparency-Log Inclusion-Proof Verifier. Build RFC 6962 domain-separated hashing, root_from_proof (recompute the Merkle root from a leaf + audit path), verify_sth (trust the signed tree head), and verify_inclusion (Chapter 6). This is the mechanism under Sigstore/Rekor and Certificate Transparency.
LAB_MODULE=solution pytest -q
pytest -q

Success Criteria

You are ready for Phase 15 when you can, without notes:

  1. Walk the source→consumer threat model and name the defense at each link.
  2. Explain provenance, in-toto attestations, and why materials reconciliation is the key check.
  3. Explain the SLSA levels and what each buys; compute an achieved level from build properties.
  4. Explain hermetic and reproducible builds and why they sit at the top of the ladder.
  5. Explain keyless signing (Fulcio) and transparency logs (Rekor), and verify a Merkle inclusion proof.
  6. Explain binary authorization and why "verify at the door, fail closed" is the enforcement that matters.
  7. Name the dependency attack classes (confusion, typosquat, hijack) and the pin-by-digest defense.
  8. Explain SBOM + VEX and why both are needed.

Interview Q&A

Q: What does SLSA L3 buy that L2 doesn't? A: L2 gives signed, service-generated provenance (known builder, tamper-evident). L3 adds a hardened, isolated builder whose provenance cannot be falsified by any build step — so even a compromised build step (the SolarWinds class) can't forge a clean provenance. L3 is the practical bar for production-critical software.

Q: How do you defend against dependency confusion? A: Use scoped/namespaced packages, pin per-package to a specific registry, refuse public resolution of internal names, and pin by digest so a higher-versioned public impostor can't be substituted. Reconcile the materials the build actually consumed against the reviewed lockfile.

Q: How does a transparency log let you trust a signature? A: Every signing event is recorded in an append-only Merkle log whose root the operator signs (a Signed Tree Head). An inclusion proof — an audit path of sibling hashes — lets you recompute the root from your leaf and check it against the signed root, proving the entry is logged. Because the log is public and append-only, a maliciously-issued signature can't be hidden; anyone monitoring the log detects it. RFC 6962's 0x00/0x01 domain separation prevents presenting an internal node as a leaf.

Q: Signing without verification — what's wrong? A: It's theater. The control is binary authorization — verifying, at deploy/admission, that the artifact has the required attestations (trusted signer, SLSA-≥-N provenance for the expected source, logged, materials matching the lockfile) and refusing anything that doesn't. The build/signing pipeline is itself the top target, so verification at the consumption point is the backstop.

Q: SBOM vs VEX? A: An SBOM inventories what's present (every dependency, ideally by digest), enabling fast "are we affected and where" when a CVE drops. VEX states whether a present vulnerability is actually exploitable in this product (not_affected/affected/fixed, with justification), cutting unreachable-CVE noise. You ship both.

References

Frameworks and specs

  • SLSA specification (slsa.dev) — build track levels and requirements.
  • in-toto attestation framework and predicate specs; DSSE envelope; SPDX and CycloneDX SBOM; OpenVEX / CSAF VEX.
  • RFC 6962 (Certificate Transparency — Merkle inclusion/consistency proofs).
  • NIST SP 800-218 SSDF; NIST SP 800-161 (supply-chain risk management); EO 14028 / NIST guidance.

Tooling and projects

  • Sigstore: Cosign, Fulcio, Rekor; Sigstore policy-controller; Google Binary Authorization; Kyverno image verification.
  • Reproducible Builds project; Bazel and Nix (hermetic builds); SLSA GitHub generator; OpenSSF Scorecard.

Incidents to study

  • SolarWinds/SUNBURST; event-stream; xz/CVE-2024-3094; dependency-confusion (Alex Birsan); ua-parser-js.
  • Phase 03 (SBOM/SLSA/Sigstore intro), Phase 07 (admission/binary authorization), Phase 13 (firmware supply chain).