Warmup Guide — CI/CD Pipelines & Secure Supply Chain

Zero-to-principal primer for Phase 08: what a pipeline actually is (a DAG, not a script), the stage/job/step model and the dependsOn graph, the condition engine that decides what runs and what skips, approval gates and environments, matrix fan-out, then the part that gets people owned — OIDC workload-identity federation (keyless auth to Azure with no stored secret, and why the token's subject is the security boundary), then the deployment strategies (blue-green, canary, rolling) as control loops with automatic rollback, and finally the secure supply chain (artifacts, SBOM, provenance, signing, SLSA). Every concept goes from what it is to why it exists to the mechanism under the hood (diagrams, tables, code) to production significance to the misconceptions that get people paged or breached.

Table of Contents


Chapter 1: Why CI/CD Is the Control Plane for Everything Else

From zero. Continuous Integration (CI) means: every change is automatically built and tested the moment it's pushed, so integration problems surface in minutes instead of at a quarterly "merge week." Continuous Delivery / Deployment (CD) means: a change that passes CI is automatically packaged and rolled out (delivery = ready to deploy at a button; deployment = deployed without one). Together they are the automated path from a commit to production.

Why it exists. Before CI/CD, releases were manual, batched, and terrifying — a human ran build scripts on their laptop, copied artifacts to servers, and prayed. The failure mode was large, infrequent, high-risk deploys. CI/CD inverts that: small, frequent, low-risk deploys, each automatically verified, each independently reversible. The whole DevOps thesis (Accelerate, the DORA metrics) is that deploy frequency and lead time correlate with — not trade against — stability, because small changes are easy to verify and easy to roll back.

Why a principal cares more than anyone. Every artifact this curriculum produces — an ARM deployment (P01), a Terraform apply (P02), a container image (P07), a Function (P11) — reaches production through a pipeline. That makes the pipeline the control plane for the control plane: it holds the credentials that can change your whole cloud, it's the gate that lets (or stops) a bad change, and it's the single most attractive target an attacker has (compromise the build, and every signed artifact is trusted). So "the pipeline" is not DevOps plumbing you delegate; it is a security-critical distributed system you must understand at the level of which stage runs when, what credential it holds, and what it's allowed to deploy.

The three things this phase makes you own:

   the GRAPH            the IDENTITY              the ROLLOUT
   (what runs when)     (how it auths, keyless)   (how a bad release is contained)
   Ch. 2–5              Ch. 6–8                    Ch. 9

…plus the supply-chain framing (Ch. 10) that wraps all three.

Chapter 2: A Pipeline Is a DAG — Stages, Jobs, Steps

The hierarchy. Both Azure Pipelines and GitHub Actions use the same three-level model (different nouns):

LevelAzure PipelinesGitHub ActionsWhat it is
topstage(job groups via needs)a deployment boundary, usually one per environment
middlejobjoba unit that runs on one agent/runner; the unit of parallelism
bottomstepstepone task or script line; runs in order within a job

A stage contains jobs; a job contains steps. Steps in a job run sequentially on the same agent (so they share a filesystem); jobs run in parallel on different agents (so they do not share a filesystem — you pass data via artifacts).

Why it's a graph. Stages declare dependencies with dependsOn (Azure) / needs (GitHub). Those dependencies are edges; the stages are nodes; together they form a directed acyclic graph. The engine computes a topological order — every stage runs after all its dependencies — and runs independent stages in parallel.

        ┌──────┐
        │ build│
        └──┬───┘
     ┌─────┴─────┐
     ▼           ▼
 ┌───────┐   ┌────────┐        build → {test, lint} → deploy
 │ test  │   │  lint  │        deploy waits for BOTH test and lint
 └───┬───┘   └───┬────┘        test and lint run in parallel
     └─────┬─────┘
           ▼
       ┌────────┐
       │ deploy │
       └────────┘
# Azure Pipelines
stages:
  - stage: build
  - stage: test
    dependsOn: build
  - stage: lint
    dependsOn: build
  - stage: deploy
    dependsOn: [test, lint]   # fan-in: waits for both

Why acyclic matters. A cycle (a depends on b depends on a) has no valid run order — the engine cannot start either — so it's a configuration error, refused before anything runs. So is an edge to a stage that doesn't exist. In the lab, _topological_order raises on both, exactly like the real engine rejecting your YAML. (Kahn's algorithm: repeatedly emit a node with no remaining incoming edges; if you can't empty the graph, what's left is a cycle.)

Determinism. CI must be reproducible: the same graph must always run the same way. When several stages are simultaneously ready, the lab breaks ties by name, so {stage: status} is byte-identical run to run. Non-determinism in a build system is a bug you debug for days ("it only fails on Tuesdays").

Chapter 3: The Execution Model — Conditions and Failure Propagation

This is the chapter that explains every "why didn't my stage run?" you will ever debug.

The rule, stated once:

A stage runs iff (a) every dependency satisfied this stage's condition, and (b) if the stage requires approval, that approval was granted. Otherwise it is skipped (condition unmet) or blocked (approval withheld). When it runs, its result is succeeded or failed.

The conditions. A condition decides whether to run given the results of the dependencies:

ConditionRuns whenReal use
succeeded (default)all dependencies succeededthe normal case — deploy only if tests passed
alwaysregardless of dependency resultsnotify / cleanup / publish-logs — must run even on failure
failedsome dependency failedrollback / open-incident — run only when something broke

Failure propagation is the whole point of a DAG. A failed stage is not succeeded. So any downstream stage with the default succeeded condition finds an unmet condition and skips. That skip cascades transitively:

build(ok) → test(FAIL) → deploy(?)        deploy.condition = succeeded
                                          test is not succeeded ⇒ deploy SKIPS
                          → notify(?)      notify.condition = always ⇒ notify RUNS
            ── and a rollback stage with condition=failed runs IFF a dep FAILED ──

Two classic bugs this model explains:

  • "My deploy stage silently didn't run." — A dependency failed (or was itself skipped), so the default succeeded condition wasn't met. The stage skipped; it didn't fail. Skipped ≠ failed ≠ blocked — three different statuses, three different causes.
  • "Our rollback ran on a green build." — Someone wrote condition: always() where they meant failed(). always runs no matter what; failed runs only on a failure.

A subtlety the lab gets right: a dependency that was skipped or blocked also isn't succeeded and isn't failed. So a succeeded-dependent of a skipped stage skips, and a failed-dependent of a skipped (not failed) stage also skips — failed needs a real failure, not merely an absence of success. (This is why, in the worked example, a rollback that depends on a skipped deploy is itself skipped — nothing actually failed at deploy; it never ran.)

Chapter 4: Approval Gates and Environments

What it is. An environment (Azure Pipelines / GitHub Actions both call it that) is a named deployment target — dev, staging, production — that you can protect with checks: required reviewers (a human clicks Approve), a wait timer, a business-hours window, an exclusive lock (one deploy at a time), or an automated gate (a query against a monitor must be healthy).

Why it exists. Not every deploy should be fully automatic. Production often wants a human in the loop, or a "soak" period, or a manual smoke test. The approval gate pauses the pipeline rather than failing it — the stage sits in a blocked / waiting state until the approval arrives (or expires, after which it's auto-rejected).

Mechanism. The engine reaches the protected stage, sees an unmet check, and suspends the run there — no agent is consumed, no job starts — until the check passes. In the lab:

if stage.approval and not approvals.get(stage.name, False):
    status[stage.name] = BLOCKED      # waiting for a human; downstream sees "not succeeded"
    continue

Production significance. A blocked stage is not a failure and not a success — so a downstream succeeded-dependent skips (the gate hasn't been approved, so nothing past it should run). This is correct and important: an un-approved prod deploy must not let a post-deploy smoke-test stage run against a version that was never deployed. The lab's test_blocked_dependency_skips_downstream pins exactly this.

Misconception. "Approvals are just a UI nicety." No — an approval gate is a control. It's the difference between a continuous-delivery pipeline (auto to staging, gated to prod) and continuous-deployment (auto all the way). Which one you choose is an explicit risk decision, and a principal states it in the ADR, not in a Slack thread.

Chapter 5: Matrix Fan-Out

What it is. A matrix expands one job definition into the Cartesian product of a set of variables — one job per combination, run in parallel.

# GitHub Actions
strategy:
  matrix:
    os: [ubuntu, windows, macos]
    python: ["3.11", "3.12", "3.13"]
# ⇒ 3 × 3 = 9 parallel jobs, one per (os, python) pair

Why it exists. You want to test (or build) across a grid of environments without copy-pasting nine near-identical jobs. The matrix is the DRY answer: one spec, N runs. The downstream stage (e.g. publish) depends on the whole matrix — fan-out then fan-in.

Mechanism & cost. The engine multiplies out the matrix dimensions and schedules each cell as an independent job (subject to a concurrency cap — max-parallel). The principal note: the matrix is also a cost and time lever. Nine parallel jobs finish in the time of the slowest, but consume nine agents; a careless matrix (every OS × every version × every flag) can be hundreds of jobs and a surprising bill. Prune to the combinations that actually catch bugs.

Chapter 6: The Credential Problem CI Was Born With

The original sin. To deploy to Azure, a pipeline must authenticate — present an identity Azure trusts. The first-generation answer: create a service principal (an Entra app identity), generate a client secret (a password) or a certificate, and store it in the CI system as a secret variable. The pipeline reads it and signs in.

Why that's dangerous — count the ways:

  • It's long-lived. A secret valid for a year (or, lazily, forever) authenticates from anywhere, not just your pipeline. Steal it once and you're in until someone rotates it.
  • It's exfiltratable. Secrets leak into logs (echo $SECRET), into forked pull requests, into compromised actions/tasks, into screenshots, into the breach of the CI vendor itself.
  • It's rotation hell. Many secrets across many pipelines, each with its own expiry, is a perpetual maintenance tax — and the one nobody rotated is the one that leaks.
  • It's a standing target. The secret exists at rest 24/7 whether or not a build is running. The attack surface is "always."
   ┌───────────┐  stores  ┌──────────────────┐  presents  ┌────────┐
   │  Entra SP │ ───────► │ CI secret store  │ ─────────► │ Azure  │
   │  + secret │  (leaks) │  (logs, forks,   │  forever   │  ARM   │
   └───────────┘          │   vendor breach) │            └────────┘
        the credential exists at rest, valid from anywhere, long-lived

Every major CI breach (Codecov 2021, CircleCI 2023) had the same blast radius multiplier: stored long-lived cloud credentials. The fix isn't "store it more carefully." It's don't store it at all — Chapter 7.

Chapter 7: OIDC Workload-Identity Federation — Keyless Auth

The idea in one sentence. Instead of storing a secret, the pipeline proves who it is with a short-lived OIDC token that the CI provider mints per-run, and Entra exchanges that token for an Azure access token — only if a pre-configured federated credential trusts the token's issuer, subject, and audience. No secret is stored anywhere.

The cast:

  • OIDC issuer — the CI provider runs an OpenID Connect identity provider. GitHub: https://token.actions.githubusercontent.com. Azure DevOps: https://vstoken.dev.azure.com/{org-id}. It publishes signing keys (JWKS) at a well-known URL and mints a signed JWT (the OIDC token) for each pipeline run.
  • The OIDC token's claims — a JWT with iss (the issuer above), aud (the audience, e.g. api://AzureADTokenExchange), and crucially sub (the subject, identifying which repo, branch, or environment this run is), plus exp (it lives minutes).
  • Federated credential — a configuration object on an Entra app registration or a user-assigned managed identity that says "I will trust a token with this issuer, this subject, and this audience." It stores no secret — just those three strings.

The exchange, step by step:

1. pipeline run starts
2. CI provider mints an OIDC token (a JWT):
      { iss: token.actions.githubusercontent.com,
        sub: repo:contoso/app:ref:refs/heads/main,
        aud: api://AzureADTokenExchange, exp: now+5m, ... }   # signed by issuer
3. pipeline POSTs it to Entra's token endpoint as a client assertion
      grant_type=client_credentials
      client_assertion_type=...jwt-bearer
      client_assertion=<the OIDC token>
4. Entra:
      a. fetches the issuer's JWKS, VERIFIES the token's signature + exp
      b. looks up the app's federated credentials
      c. checks: does some credential match iss AND sub AND aud  EXACTLY?
      d. yes ⇒ mint an Azure access token (scoped by the identity's RBAC)
         no  ⇒ reject (AADSTS700213 / 700212)
5. pipeline uses the access token to call ARM (deploy)

The lab models step 4c — the match decision — which is where the security lives (step 4a, the signature check, is P03's JWT-validation lab):

def exchange_oidc_token(token_claims, creds):
    for required in ("iss", "sub", "aud"):
        if required not in token_claims:
            raise ValueError(...)                  # a malformed token is denied loudly
    for cred in creds:
        if (cred.issuer   == token_claims["iss"]
            and cred.audience == token_claims["aud"]
            and subject_matches(cred.subject, token_claims["sub"])):
            return True                            # exact match ⇒ exchange allowed
    return False                                   # DENY BY DEFAULT

Why this beats a stored secret — directly:

PropertyStored SP secretOIDC federation
Credential at restyes, long-lived, always presentnone — nothing to steal
Validity windowmonths/yearsminutes (the token's exp)
Usable from anywhereyesno — only a run that the issuer vouches for
Scoped to a branch/envnoyes — via the subject
Rotationmanual, perpetualn/a — there's nothing to rotate
Leak in logs/forkscatastrophiclow value — expires in minutes, subject-pinned

Misconception. "Keyless means there's no identity." Wrong — there's still an Entra identity with RBAC role assignments (P04); what's gone is the stored secret. The identity is the same; the proof changed from "I know a password" to "an issuer Entra trusts vouches that I'm this exact pipeline run."

Chapter 8: The Subject Is the Security Boundary

The single most important field in the whole flow is the OIDC token's sub and the federated credential's subject that it must match. The subject encodes which exact pipeline context is asking:

GitHub Actions subjects:
  repo:contoso/app:ref:refs/heads/main          ← a push to main
  repo:contoso/app:ref:refs/heads/release        ← the release branch
  repo:contoso/app:environment:production        ← a deploy to the 'production' environment
  repo:contoso/app:pull_request                  ← ANY pull request (incl. forks!)

Why this is the boundary. Configure the production identity's federated credential with subject = repo:contoso/app:ref:refs/heads/main, and only a run triggered by main can mint the prod token. A push to a feature branch produces sub = ...:ref:refs/heads/feature/xno match, denied. A pull request from a forked repo produces sub = ...:pull_requestno match, denied. The wrong subject is rejected, and that rejection is the security control. The lab's test_oidc_reject_wrong_subject_branch is the entire point of the OIDC half of the lab:

good = {"iss": ISS, "sub": "repo:contoso/app:ref:refs/heads/main", "aud": AUD}
bad  = {**good, "sub": "repo:contoso/app:ref:refs/heads/attacker"}
exchange_oidc_token(good, [cred])  # True  — right branch
exchange_oidc_token(bad,  [cred])  # False — wrong branch, DENIED

The classic misconfiguration. Making the subject too broad — e.g. a credential that matches any branch (...:ref:refs/heads/*) on a repo where pull requests run privileged deploys — lets an attacker open a PR and acquire your cloud credential. The lab supports a trailing-* pattern because the real systems do, with a giant caveat: a wide pattern is a wide blast radius. The principal default is the narrowest exact subject per environment, and one federated credential per (identity, branch/environment) pair.

Audience matters too. The aud (api://AzureADTokenExchange for Entra) prevents a token minted for another relying party from being replayed at Entra. Wrong audience → denied (test_oidc_reject_wrong_audience). And the issuer must be exact — a token from https://evil.example.com is denied even with the right subject (test_oidc_reject_wrong_issuer). All three must match; deny by default if no configured credential does.

Chapter 9: Deployment Strategies as Control Loops

A deploy is not copy new version over old. It's a control loop: introduce the new version gradually or reversibly, measure health, and roll back automatically if health degrades — so a bad release is contained, not broadcast.

Blue-green. Two identical production environments ("slots"), one live (blue), one idle (green). Deploy the new version to the idle slot, warm it, then swap — an atomic cutover at the load balancer. If post-swap health is bad, swap back: rollback is instant because the old version is still warm in the other slot.

before:  [BLUE v1] ◄── 100% traffic        [GREEN] (idle, deploy v2 here, warm it)
swap:    [BLUE v1] (idle)                   [GREEN v2] ◄── 100% traffic
healthy? yes ⇒ done.   no ⇒ swap back ⇒ [BLUE v1] ◄── 100%  (instant rollback)
def blue_green_swap(active, candidate, *, healthy):
    return candidate if healthy else active   # the slot that ends up live

In Azure this is an App Service slot swap. Cost: you pay for two slots; benefit: zero- downtime cutover and instant rollback.

Canary. Don't cut over 100% — shift a small slice (5%) to the new version, watch its health, then ramp (25% → 50% → 100%) only if it stays healthy. On the first failing step, roll back to 0%. The blast radius of a bad release is capped at the canary slice, and rollback is automatic — no human paged.

 5% ─ healthy? ─► 25% ─ healthy? ─► 50% ─ healthy? ─► 100%  (done)
                    │ NO
                    ▼
                rollback to 0%   { final_pct: 0, rolled_back: True, failed_at: 25 }
def canary_rollout(steps, health_check):
    for pct in steps:
        if not health_check(pct):
            return {"final_pct": 0, "rolled_back": True, "failed_at": pct}
    return {"final_pct": 100, "rolled_back": False}

Two design notes the lab pins: on rollback final_pct is 0 (you pulled all the new- version traffic, not just the failing step's), and on success it's 100 regardless of the last step's literal value ("all steps passed" = fully rolled out). In Azure this is AKS/Container Apps progressive traffic-splitting (Argo Rollouts, Flagger, or native revision weights).

Rolling. Replace instances in batches (e.g. 25% at a time), health-checking each batch before the next. No second environment (cheaper than blue-green), but rollback is slower (you must roll the batches back), and during the roll you run mixed versions — which only works if the change is backward-compatible.

StrategyRollback speedExtra costMixed versions live?
Blue-greeninstant (swap back)2× environmentno (atomic swap)
Canaryfast (drop the slice)small (the canary %)yes (canary + stable)
Rollingslow (re-roll batches)noneyes (during the roll)

The principal rule: the health gate and the rollback path must exist before the deploy, not be improvised during the incident. "We'll watch it manually" is not a strategy; an automatic gate is.

Chapter 10: The Secure Supply Chain — Artifacts, SBOM, Provenance, Signing

Why this is now a named discipline. SolarWinds (2020), Codecov (2021), and the xz backdoor (2024) share one shape: the build/distribution system was compromised, so the artifact everyone downloaded was malicious but signed-and-trusted. You can't defend against that by trusting the artifact — you have to be able to ask "who built this, from what source, with what builder, and can I verify that cryptographically?"

The four artifacts, and where each is checked:

ArtifactWhat it isProducedVerified
Artifactthe built thing (image, zip, package) — content-addressed by digestbuild stageby digest, on every pull
SBOMsoftware bill of materials — every dependency + version (CycloneDX / SPDX)build stagescanned for known CVEs at the gate
Provenancea signed statement: this digest was built by builder X from source commit Y (in-toto / SLSA)build stageat the deploy gate
Signaturea cryptographic signature over the artifact + provenance (Sigstore/cosign)build/sign stageat the deploy gate (verify signer identity)

SLSA (Supply-chain Levels for Software Artifacts) is the framework that names the maturity levels: roughly L1 = you have provenance, L2 = it's signed by the build service, L3 = the build runs in a hardened, isolated builder that can't be tampered with, L4 = two-party review + hermetic builds. You don't need to memorise the levels; you need to be able to say in a design review "our deploy gate verifies a signed provenance attestation that the artifact came from our main pipeline, so a hand-pushed image is refused."

The connection to the rest of this lab: the same OIDC identity (Ch. 7) that deploys is the identity that signs (keyless signing with Sigstore uses an OIDC token too — the signer's identity is the pipeline's subject), and the deploy gate (Ch. 3–4) is where the verify step lives as a condition. The supply chain isn't a separate system; it's the pipeline, with attestation added at build and verification added at the gate.

Lab Walkthrough Guidance

Lab 01 — Pipeline Engine, suggested order:

  1. Stage + _topological_order — the Stage dataclass validates its name, condition (succeeded/always/failed), and rejects self-dependency. Then Kahn's algorithm with a sorted ready-set for determinism; raise on a duplicate name, an edge to an unknown stage, or a cycle (Ch. 2). Test with test_cycle_raises, test_unknown_dependency_raises, test_topo_order_is_deterministic.
  2. run_pipeline — for each stage in topo order: SKIP if its condition isn't satisfied by its dependencies' statuses; BLOCK if it needs approval and none was granted; else run (stage.run(context) or outcomes[name], default success) → SUCCEEDED/FAILED (Ch. 3–4). The key cases: test_skip_on_failed_dependency, test_always_runs_even_after_failure, test_failed_condition_runs_only_on_failure, test_approval_blocks_then_proceeds.
  3. exchange_oidc_token + FederatedCredential + subject_matches — exact issuer and audience, subject via subject_matches (exact, plus a trailing-* pattern); deny by default; raise on a missing claim (Ch. 7–8). The four negatives are the whole point: test_oidc_reject_wrong_issuer / ..._subject_branch / ..._audience, and test_oidc_deny_by_default_no_creds.
  4. blue_green_swap + canary_rollout — blue-green returns the live slot; canary ramps and rolls back to 0% on the first failing step (Ch. 9). Pin test_canary_rolls_back_at_failing_step (returns failed_at and final_pct: 0) and test_canary_completes_when_all_healthy (final_pct: 100).

Run it red, then green:

pytest test_lab.py -v                      # your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # the reference — all 30 green
python solution.py                         # the worked example

Success Criteria

You are ready for Phase 09 when you can, from memory:

  1. Draw a 5-stage pipeline DAG and, for a given mid-graph failure, say which stages run, skip, or block — and why each (condition unmet vs approval withheld vs dep failed).
  2. State the execution rule in one sentence ("a stage runs iff every dependency satisfied its condition and any required approval was granted").
  3. Distinguish succeeded / always / failed conditions and name the real stage that uses each (deploy / notify / rollback).
  4. Draw the OIDC token-exchange flow end to end and name the three claims that must match (iss, sub, aud) and which one is the branch/environment boundary (sub).
  5. Explain — without a stored secret anywhere — why OIDC federation beats a long-lived service-principal secret, using at least three of the table rows in Ch. 7.
  6. Explain why a wrong-branch subject is denied and why that denial is the security control.
  7. Contrast blue-green, canary, and rolling on rollback speed, extra cost, and mixed- version exposure — and say which rollback is instant and why.
  8. Name the four supply-chain integrity artifacts and where each is verified.

Interview Q&A

Q: A teammate says "the deploy stage just didn't run, no error." Walk me through how you debug it. First I clarify the status: skipped, blocked, or not-reached — they're different bugs. Skipped means a dependency didn't satisfy the stage's condition: most often a succeeded condition where an upstream stage failed or was itself skipped (failure propagates down the DAG). Blocked means a required approval/check on its environment hasn't been granted — it's waiting, not broken. Not-reached means an upstream stage is still running or the DAG edge isn't what they think. I'd look at the upstream stages' results and the deploy stage's condition, because 90% of "silently didn't run" is "a dependency wasn't succeeded and the default condition needs all deps green."

Q: How does a pipeline deploy to Azure without storing a secret, and what stops a feature branch from deploying to production? OIDC workload-identity federation. The CI provider is an OIDC issuer; per run it mints a short-lived signed JWT whose sub claim encodes the exact context (repo + branch or environment). The pipeline presents that token to Entra as a client assertion; Entra verifies the signature against the issuer's JWKS, then checks whether any federated credential on the target identity matches the token's iss, sub, and aud exactly — and if so mints an Azure access token. Nothing is stored: no secret to leak, the token lives minutes, and it's scoped to one subject. What stops a feature branch from deploying to prod is the subject: the prod identity's federated credential is pinned to ...:ref:refs/heads/main (or ...:environment:production), so a feature-branch run produces a different sub, matches no credential, and is denied. The wrong subject is the boundary.

Q: Why is OIDC federation actually more secure than a service principal with a client secret? Be specific. Four reasons. (1) No credential at rest — there's nothing in the CI vault to steal; a stored secret is a standing target valid 24/7. (2) Short-lived — the OIDC token expires in minutes, so even if it leaks in a log its window is tiny, versus a secret valid for a year. (3) Context-bound — the token is only minted by a real run the issuer vouches for, and only for the specific subject; a stolen secret authenticates from anywhere. (4) No rotation — there's nothing to rotate, which removes the "the one secret nobody rotated" failure mode entirely. The trade is that you must get the federated-credential subject right; too broad a subject (matching any branch, or pull requests on a privileged repo) is the one way to make it less safe than a tightly-scoped secret.

Q: Blue-green vs canary — when do you pick each, and how does rollback differ? Blue-green when I want an atomic, zero-downtime cutover with instant rollback and can afford a second full environment — the old version stays warm in the idle slot, so rollback is one swap. Canary when I want to limit blast radius and let real traffic and metrics gate the rollout: I shift 5%, watch the SLO, ramp only if healthy, and auto-roll- back to 0% on the first breach — so a bad release only ever hit the canary slice. Rolling when I can't afford a second environment and the change is backward-compatible, accepting slower rollback and mixed-version exposure during the roll. The non-negotiable across all three: the health gate and rollback path exist before the deploy, automated — "we'll watch it" is not a strategy.

Q: An attacker opens a pull request to your repo. Your pipeline runs on PRs and deploys to prod with OIDC. What's the risk and the fix? The risk is that a PR — including one from a fork the attacker controls — triggers a run that can mint your prod cloud credential. The defense is the subject: PRs get sub = ...:pull_request, which must not match any privileged federated credential. Concretely: pin the deploy identity's credential to ...:ref:refs/heads/main (or the production environment), require the deploy to run only on main/after merge, never on pull_request, and use GitHub's environment protection so fork PRs can't even request the id-token for prod. Also: never run privileged steps with untrusted PR code in the same job that holds the token. The whole point of the narrow subject is to make "PR-from-a-fork" a non-match.

Q: What's the difference between continuous delivery and continuous deployment, and how does the pipeline encode it? Continuous delivery keeps the pipeline green and deployable to prod at the click of a button — there's an approval gate before prod. Continuous deployment removes that gate: every change that passes CI goes all the way to prod automatically. In the pipeline it's literally one thing: whether the prod stage's environment has a required-reviewer check. That's a risk decision — how costly is a bad prod deploy vs how mature is your automated verification and rollback — and a principal makes it explicitly in an ADR, not by default.

Q: How do you stop a compromised build from shipping a trusted-but-malicious artifact? You make the artifact verifiable, not just signed. The build emits a provenance attestation (in-toto/SLSA: this digest, from this source commit, by this builder) and signs it (keyless via Sigstore, where the signer identity is the pipeline's OIDC subject). The deploy gate then verifies — refusing any artifact whose provenance isn't signed by the expected builder identity, so a hand-pushed or side-loaded image is rejected even if it's "signed" by something else. Plus an SBOM scanned for known CVEs at the gate. SLSA names the maturity levels; the principal point is that trust moves from "the artifact looks fine" to "I can cryptographically attribute it to my hardened pipeline."

References

  • Azure Pipelines — Stages, dependencies & conditions: https://learn.microsoft.com/azure/devops/pipelines/process/stages
  • Azure Pipelines — Specify conditions (succeeded(), failed(), always()): https://learn.microsoft.com/azure/devops/pipelines/process/conditions
  • Azure Pipelines — Environments and approvals/checks: https://learn.microsoft.com/azure/devops/pipelines/process/environments
  • GitHub Actions — Using jobs / needs (the job DAG): https://docs.github.com/actions/using-jobs/using-jobs-in-a-workflow
  • GitHub Actions — OIDC hardening / OIDC token: https://docs.github.com/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect
  • Microsoft Entra — Workload identity federation (concept + supported scenarios): https://learn.microsoft.com/entra/workload-id/workload-identity-federation
  • Microsoft Entra — Configure a federated credential on an app: https://learn.microsoft.com/entra/workload-id/workload-identity-federation-create-trust
  • Azure — azure/login GitHub Action with OIDC (no secret): https://github.com/Azure/login
  • Azure App Service — Deployment slots and swap: https://learn.microsoft.com/azure/app-service/deploy-staging-slots
  • OpenID Connect Core 1.0 — the iss/sub/aud semantics: https://openid.net/specs/openid-connect-core-1_0.html
  • SLSA — Supply-chain Levels for Software Artifacts: https://slsa.dev
  • in-toto — Software supply chain attestation: https://in-toto.io
  • Sigstore / cosign — keyless signing with OIDC identities: https://www.sigstore.dev
  • DORA / Accelerate (Forsgren, Humble, Kim) — deploy frequency vs stability
  • The track's own CHEATSHEET.md and GLOSSARY.md