Hitchhiker's Guide — Building Security into CI/CD

The operational companion to the Phase 03 WARMUP. The WARMUP frames product security as a control system (threat modeling, scanner limits, supply chain, vuln management); this guide is the how — the pipeline shape, runner hardening, gate semantics, seeded gate tests, the release trust boundary, CVE fleet response, and the program evidence that proves the controls operate. Scanner output by itself is not a program.

Table of Contents

• Pipeline Shape
• The Release Trust Boundary
• Runner Hardening
• Gate Semantics
• Seeded Gate Tests
• Developer Experience
• Emergency Release
• CVE Fleet Response
• Threat-Model Review and Rollout
• Program Evidence
• Worked Examples
• Interview Drills
• References

Pipeline Shape

Put each control at the earliest stage where it has enough signal to be accurate (Phase 03 WARMUP, Chapter 7):

pre-commit : formatting, secret canaries, fast unit security tests
pull request: SAST, SCA, IaC scan, policy unit tests, threat-model delta
build      : isolated runner, pinned deps, SBOM, provenance, signing
pre-deploy : signature/admission verification, DAST against an ephemeral env
runtime    : config-drift detection, alerts, vulnerability intake, rollback

The Release Trust Boundary

The single most important supply-chain control is separating identities along the path:

source contributor → reviewer/merger → trusted builder → deployer
   (writes code)      (approves)         (builds)         (ships)

Pull-request code must never mint a production credential. A trusted build consumes an immutable, reviewed revision and emits an artifact digest, SBOM, provenance, test results, and a signature; deployment verifies those claims before accepting the digest (Phase 03 WARMUP, Chapter 8 — verification is the control, not signing).

Map every writable input to the build: source, lockfiles, generated code, base images, registries, actions/plugins, caches, toolchains, environment, and network downloads. Pin everything possible by immutable digest and isolate caches by trust level.

Runner Hardening

The pipeline is production infrastructure with write access to production (Phase 03 WARMUP, Chapter 9):

• untrusted pull-request code receives no deployment secret;
• ephemeral, isolated runners with short-lived OIDC-federated workload identity (no stored long-lived keys);
• read-only source where possible; bounded network and artifact access;
• protected cache keys; no cross-trust cache reuse (cache poisoning);
• pin actions/images by immutable digest, not a mutable tag;
• scan logs and artifacts for secrets;
• separate build and deploy identities.

Gate Semantics

A gate has three modes, and choosing wrong either blocks delivery on noise or ships real risk:

• block — evidence strong, impact material, remediation established;
• warn / measure — while tuning, or for lower-confidence signals;
• exception — only with owner, scope, compensating controls, expiry, and retest (no permanent suppressions).

Seed vulnerable and safe fixtures and measure: true-positive retention, false blocks, bypass frequency, fix time, and local reproducibility. The release-gate lab must block on seeded criticals and stay green on approved fixtures.

Seeded Gate Tests

Prove the gate fails on each and passes the safe fixtures:

• a reachable critical dependency;
• a low-severity hard-coded secret;
• an expired exception;
• an artifact changed after signing;
• a missing SBOM / provenance;
• a fix without a regression test;
• a scanner false positive with a reviewed, expiring suppression (must pass).

Developer Experience

Every block message must include: finding ID, evidence, why it matters, the exact remediation, a local reproduction, the exception path, and the owner. A noisy, opaque gate gets bypassed — and a bypassed gate is no control. Developer experience is a security control here.

Emergency Release

Keep a documented, time-bounded, approved emergency path with compensating monitoring and mandatory post-release correction. Security that prevents a safe emergency response is not resilient security — but the emergency path must be auditable, and "rollback" must never become an undocumented permanent bypass.

CVE Fleet Response

When a critical CVE drops, the SBOM (Phase 03 WARMUP, Chapter 8) makes "are we affected, and where?" answerable in minutes:

1. establish affected versions/configuration from primary sources;
2. find direct, transitive, shaded, vendored, image, and dormant copies (SBOM query);
3. determine runtime reachability and authority (not every match is exploitable);
4. contain — without confusing mitigation with repair;
5. upgrade, rebuild, redeploy, and rotate exposed secrets when justified;
6. prove the old artifact disappeared from the fleet;
7. preserve the inventory queries and add regressions.

Threat-Model Review and Rollout

Bring architecture, operations, security, and data/identity owners. Walk concrete abuse paths (Phase 03 WARMUP, Chapters 2–4) and record decisions, assumptions, rejected options, owners, and tests as ADRs.

Roll new controls report-only → scoped enforcement → broad enforcement with health metrics at each step. Rehearse signer loss, verifier outage, expired policy, key rotation, and emergency deployment before broad enforcement.

Program Evidence

• deployed-asset and repository coverage;
• threat → requirement → test traceability;
• runner identity and isolation diagram;
• attestation verification logs;
• exception age and recurrence metrics;
• a sampled finding-to-fix developer journey;
• failure-injection results proving the emergency and recovery paths work.

Worked Examples

Keyless CI identity (GitHub Actions OIDC → cloud), no stored secret

The fix for "leaked CI credentials" is to have no long-lived credential (Phase 03 WARMUP, Chapter 9). The workflow presents a short-lived OIDC token the cloud trusts:

permissions:
  id-token: write          # allow the runner to request an OIDC token
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy   # scoped, no stored key
          aws-region: us-east-1

On the cloud side, the role's trust policy is bound to the exact repo and branch so another repo can't assume it:

"Condition": {"StringEquals": {
  "token.actions.githubusercontent.com:sub": "repo:acme/orders:ref:refs/heads/main"}}

The interviewer is checking whether you reach for "rotate the key every 90 days" (still leakable for 90 days) or "eliminate the stored secret entirely" (strictly better).

Verifying provenance at admission, not just signing

Signing without verification is theater (Phase 03 WARMUP, Chapter 8). The deploy gate verifies:

cosign verify --certificate-identity-regexp '^https://github.com/acme/orders' \
              --certificate-oidc-issuer https://token.actions.githubusercontent.com \
              ghcr.io/acme/orders@sha256:abcd...   # by DIGEST, not a mutable tag

cosign verify-attestation --type slsaprovenance ...   # confirm builder + source commit

The provenance-verifier lab makes you detect three tampering classes: a modified artifact (digest mismatch), a forged source reference, and a wrong builder identity. If any check fails, the gate blocks.

Interview Drills

1. "Threat-model a multi-tenant export feature in 5 minutes." DFD: user → export svc → datastore → object storage → signed link, with trust boundaries at internet↔edge, tenant↔tenant, app↔storage. STRIDE the crossings; top threats are cross-tenant data in a shared export job and a long-lived public download link. Mitigate: per-row authZ in the job, short-lived signed URLs, per-tenant isolation, size/rate quotas. End in mitigations, not a filled checklist.
2. "You have 10,000 dependency findings. Prioritize." Not by count. Dedup, then rank by reachability (is the vulnerable function actually called with attacker input across a boundary?) × exploitability × asset criticality. Fix the reachable-critical few now; batch the rest into upgrade hygiene; record exceptions with owner + expiry. SBOM answers "are we affected and where."
3. "Which checks go pre-commit vs CI vs deploy vs runtime?" Pre-commit: fast lint + secret scan. CI: SAST/SCA/secret/IaC/container + security unit tests. Deploy gate: DAST, SBOM, signing + verification, policy. Runtime: detection, WAF, drift. Rule: earliest stage with enough signal; slow/noisy checks async.
4. "SAST is green — are we secure?" No. SAST finds patterns; it can't judge whether your authZ model is correct, whether a trust boundary is missing, or business-logic abuse. Name each scanner's blind spot (DAST can't reach unreached code; SCA findings are mostly unreachable; secret scanning means rotate, not delete).
5. "Persuade a VP to fund a platform authZ library." Frame as leverage: a paved-road fix removes a bug class across every team forever vs. paying for the same finding repeatedly in pen-tests and incidents. Tie to a metric they own (velocity, incident cost, audit findings); present a one-page decision with options + recommendation + investment + success metric.

References

• OWASP SAMM and ASVS; NIST SP 800-218 SSDF.
• SLSA specification (slsa.dev); in-toto attestation; Sigstore / Cosign documentation.
• CycloneDX and SPDX SBOM specs; OpenSSF Scorecard.
• Tooling docs: Semgrep, CodeQL, OWASP ZAP, Trivy, Syft, Checkov, OPA/Conftest.
• Supply-chain postmortems: SolarWinds, event-stream, xz (CVE-2024-3094); "Dependency Confusion".