Phase 08 — CI/CD Pipelines & Secure Supply Chain

Difficulty: ⭐⭐⭐⭐☆ (the mechanism is medium; the security reasoning is principal) Estimated Time: 1–1.5 weeks (14–20 hours) Prerequisites: P03 (OAuth2 / OIDC / JWT — the token you federate is a JWT), P04 (RBAC — the deploy identity's role assignments), P07 (containers/ACR — what you build and push). P01 (ARM deployment) is the thing your pipeline actually triggers.

Why This Phase Exists

Every other phase in this track describes a thing that exists in Azure. This phase describes the thing that puts every other thing into production — and the place where security most often goes to die. The JD asks for it in one line — "Develop and maintain CI/CD pipelines to automate application builds, testing, and deployments" — but the principal-level content is hidden under that sentence:

1. A pipeline is a dependency graph, not a script. Stages depend on stages, jobs on jobs, and the engine runs them in topological order with conditions that decide what gets skipped when something upstream fails. If you think of a pipeline as a linear list of commands, you cannot explain why your deploy stage "didn't run" (it was skipped because a dependency failed its condition) or why a rollback stage fired on a green build (someone set condition: always() instead of failed()). The execution model is the knowledge.

2. The pipeline is your biggest credential-leak surface, and OIDC federation is the fix. For a decade, "CI deploys to cloud" meant storing a long-lived service- principal secret in the CI system — a credential that authenticates from anywhere, forever, and leaks in logs, forks, and breaches. Workload-identity federation replaces it: the pipeline presents a short-lived OIDC token, and Entra exchanges it for an access token only if a configured federated credential matches the token's issuer, subject, and audience. No secret is stored. The subject pins the branch or environment, so a push to a feature branch — or a pull request from a fork — cannot mint a production credential. Understanding why that one string is the security boundary is the principal signal.

3. A deploy is a control problem, not a copy. Blue-green and canary are not buzzwords; they are feedback loops: shift a slice of traffic, measure health, and automatically roll back before a bad release reaches everyone. The principal builds the health gate and the rollback path before the incident, because at 2 a.m. the difference between "we auto-rolled-back at 5% traffic" and "we took prod down for 40 minutes" is whether someone wrote the gate.

4. The supply chain is now an attack surface with a name (SLSA). SolarWinds, Codecov, and the xz backdoor all share a shape: the build system was compromised, so the signed artifact was malicious-but-trusted. The modern answer is artifacts + SBOM + provenance + signing, verified at deploy time. This phase frames it so you can reason about it in a design review even before you wire up Sigstore.

If you skip this phase you can still write a .yml that turns green — and you will be unable to answer "why did the deploy skip," "how does keyless auth work," or "how do you roll back automatically," which are exactly the questions that separate the person who owns the delivery platform from the person who copies a template.

Concepts

• Pipeline structure: stages → jobs → steps. A stage is a deployment boundary (often = an environment); a job runs on one agent; a step is one task/script. The unit of dependency is the stage (and the job within a stage).
• The DAG and dependsOn: edges between stages form a directed acyclic graph; the engine topologically orders it. A cycle is a configuration error (the engine refuses to run); an edge to a non-existent stage is too.
• Conditions (succeeded / always / failed): the gate that decides whether a stage runs given its dependencies' results. succeeded (default) needs all deps green; always runs regardless; failed runs only when a dependency failed (the rollback / notify-on-failure stage). This is where "why didn't my stage run" lives.
• Failure propagation: a failed stage isn't succeeded, so its succeeded-condition dependents skip, and so on transitively. Failure flows down the DAG.
• Matrix fan-out: one job definition expands into the Cartesian product of a matrix (OS × version × …) — N parallel jobs from one spec.
• Environments + approval gates: a protected environment pauses the pipeline until a human approves (or a check passes). The stage is blocked, not failed, until then.
• OIDC workload-identity federation: keyless authentication. The CI provider is an OIDC issuer; a federated credential on an Entra identity trusts a specific issuer + subject + audience; the pipeline's OIDC token is exchanged for an access token only on an exact match. Deny by default.
• Deployment strategies: blue-green (two slots, atomic swap, instant rollback), canary (ramp traffic %, gate on health, auto-rollback), rolling (replace instances in batches) — each with a health gate that triggers automatic rollback.
• Supply-chain integrity: artifact, SBOM (software bill of materials), provenance (who built what, how), signing/attestation (Sigstore/cosign, SLSA levels), verified at the deploy gate.

Labs

Lab 01 — Pipeline Engine (flagship, implemented)

→ Lab folder: lab-01-pipeline-engine/

Integrated-Scenario Suggestions (carried through the whole track)

These tie Phase 08 to the platform the rest of the track builds:

1. Keyless multi-environment promotion. A monorepo builds once, then promotes the same artifact through dev → staging → prod, each a protected environment with its own federated credential (different subject per environment) and approval gate. No secret anywhere. Prove that a non-main branch cannot acquire the prod credential.
2. Container build-and-deploy with provenance. Build an image (P07), push to ACR, generate an SBOM and a signed provenance attestation, then a deploy gate verifies the signature and the builder identity before rolling it out to AKS — refusing any image not produced by the trusted pipeline.
3. Progressive delivery with auto-rollback. A canary on Container Apps shifts 5% → 25% → 50% → 100%, gating on the error-rate/latency SLO at each step, with an automatic rollback to 0% on the first breach — and a failed-condition stage that opens an incident only when the rollback fires.
4. Infra pipeline. The pipeline runs terraform plan (P02), posts the plan as a PR comment, requires an approval, then apply on merge — the IaC change goes through the exact same DAG + gate + keyless-auth machinery as app code.
5. Blast-radius-bounded deploy. A change to a shared platform library fans out (matrix) to rebuild every dependent service, but a circuit-breaker stage halts the fan-out after the first N canary failures so one bad library version can't roll out everywhere.

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Key Takeaways

• A pipeline is a DAG with conditions and gates, not a script. The execution model — a stage runs iff its dependencies satisfied its condition and its approval was granted — explains every "why didn't this run" you will ever debug.
• OIDC workload-identity federation is the modern answer to the worst CI security smell: a long-lived stored cloud secret. The pipeline presents a short-lived, subject-scoped OIDC token; Entra exchanges it on an exact iss/sub/aud match; the subject is the branch/environment boundary, and it denies by default.
• Deployments are control loops: blue-green (instant swap/rollback) and canary (ramp + health gate + auto-rollback) bound the blast radius of a bad release. Build the health gate and the rollback path before you need them.
• The supply chain is an attack surface: artifacts → SBOM → provenance → signing, verified at the deploy gate. SLSA is the framework that names the levels.

Deliverables Checklist

• Lab 01 implemented; all 30 tests pass against solution.py and your lab.py
• You can draw a 5-stage pipeline DAG and, for a given failure, say which stages run, skip, or block — and why
• You can explain the OIDC token-exchange match (iss/sub/aud) and why a wrong-branch subject is rejected — without a stored secret anywhere
• You can contrast blue-green vs canary vs rolling and say which rollback is instant
• You can name the four supply-chain integrity artifacts (artifact, SBOM, provenance, signature) and where each is verified