Lab 01 — Pipeline Engine

Phase: 08 — CI/CD Pipelines & Secure Supply Chain | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

A CI/CD pipeline is not a YAML file — it is a DAG executor with a security boundary bolted to the side. This lab builds the three mechanisms a principal must own under the hood: the stage/job dependency graph that decides what runs and what gets skipped when something fails, the OIDC token-exchange that lets a pipeline authenticate to Azure with no stored secret (and rejects the wrong branch), and the rollout controller that does blue-green and canary deploys with an automatic health-gated rollback. Build these and "the pipeline is stuck" / "why did prod not deploy" / "how does keyless auth actually work" stop being mysteries.

What you build

  • Stage + run_pipeline — a deterministic topological executor for a pipeline DAG. Stages have depends_on edges, a condition (succeeded / always / failed), and an optional approval gate. A stage is skipped when a dependency didn't satisfy its condition, blocked when approval is required and not granted, else it runs to succeeded / failed. Cycles and unknown dependencies raise.
  • exchange_oidc_token + FederatedCredential + subject_matches — the workload-identity-federation decision. A token is exchanged for an Azure access token iff a configured federated credential matches its iss + sub + aud. Issuer and audience match exactly; subject matches exactly (with one documented trailing-* pattern). Deny by default. A wrong-branch sub is rejected — the security boundary.
  • blue_green_swap + canary_rollout — the deployment-strategy logic. Blue-green returns the slot that ends up live (instant swap, instant rollback). Canary ramps through traffic percentages, calls a health check at each, and rolls back to 0% on the first failing step — no human paged.

Key concepts

ConceptWhat to understand
The DAG is the pipelinestages → jobs → steps; dependsOn defines edges; topo order is the run order
A stage runs iff...every dependency satisfied its condition AND any required approval was granted
Failure propagatesa failed/skipped/blocked dependency is not succeeded, so succeeded-condition dependents skip
always vs failedalways runs regardless (notify/cleanup); failed runs only when a dep failed (rollback)
Determinismsorted topo order → same input, same {stage: status} every run; CI must be reproducible
OIDC federationpresent a short-lived OIDC token; Entra exchanges it by matching iss/sub/aud — no stored secret
The subject is the knobsub pins the branch/environment; the wrong sub is denied — that's why it beats a long-lived secret
Blue-greentwo slots, instant atomic swap, instant rollback (the idle slot stays warm)
Canaryramp %, gate on health, auto-rollback on the first failing step; limits blast radius

Files

FilePurpose
lab.pyskeleton with # TODO markers — implement these
solution.pycomplete reference; python solution.py runs a worked example
test_lab.pythe proof — run it red, make it green
requirements.txtpytest only (pure stdlib otherwise)

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # the worked example

Success criteria

  • All 30 tests pass against your implementation.
  • You can explain why test_skip_on_failed_dependency returns skipped (not failed) for deploy: a succeeded condition needs all deps green; a failed dep means the condition is unmet, so the stage never runs.
  • You can explain the difference between always and failed conditions, and which one a rollback stage uses vs a notify stage.
  • You can explain why test_oidc_reject_wrong_subject_branch is the whole point of OIDC federation: same repo, same issuer, same audience — but a different sub (branch) is denied, so a PR from a fork or a push to a feature branch cannot mint a prod credential.
  • You can state, in one sentence, why OIDC federation beats a stored service-principal secret (no long-lived credential to leak; the token is per-run and subject-scoped).
  • You can explain why canary_rollout returns final_pct: 0 (not the last step) on rollback, and final_pct: 100 on success regardless of the last step value.

How this maps to real Azure

Lab pieceReal Azure / GitHub mechanismNotes & limits
Stage + depends_onAzure Pipelines stages: with dependsOn:; GitHub Actions jobs.<id>.needs:both build a DAG and run it in topological order
conditionAzure Pipelines condition: succeeded()/always()/failed(); GitHub if: success()/always()/failure()the default is "run iff all dependencies succeeded"
approvalAzure Pipelines Environments with approval checks; GitHub Environments with required reviewersa real gate also supports timeouts, business hours, and exclusive locks
run_pipeline skip/blockthe agent never even allocates a job for a skipped stage; a blocked stage shows "waiting" until approvedreal gates can expire and auto-reject after N days
FederatedCredentialan Entra app registration / user-assigned managed identity federated credential (issuer/subject/audience)max ~20 federated credentials per identity
exchange_oidc_tokenthe client_credentials grant with client_assertion_type=...jwt-bearer presenting the CI OIDC tokenEntra validates the OIDC token's signature via the issuer's JWKS before this match — the lab models the match, not the crypto
subject pinrepo:org/name:ref:refs/heads/main (GitHub) / sc://org/project/pipeline (Azure DevOps)a too-broad subject claim (e.g. matching any branch) is the classic misconfiguration
blue_green_swapAzure App Service deployment slot swapthe swap is atomic at the load-balancer; the old version stays warm in the idle slot
canary_rolloutAKS/Container Apps progressive traffic split (Argo Rollouts / Flagger / built-in revisions)a real canary also bakes (waits) and watches metrics, not a single boolean

Extensions

  • SLSA provenance: add attest_provenance(artifact_digest, builder_id) that emits a signed in-toto-style statement, and verify_provenance that a deploy step calls before promoting — model the "only deploy artifacts our builder produced" gate.
  • Matrix fan-out: extend Stage to expand a matrix={"os": [...], "ver": [...]} into one job per combination, all depending on the same upstream — and assert the Cartesian-product count.
  • Concurrency / fan-in: return not just statuses but a schedule (which stages could run in parallel at each topological "level"), the thing that decides pipeline wall-clock time.
  • Real az: configure a GitHub OIDC federated credential against an Entra app and deploy with azure/login@v2 using permissions: id-token: write — no secret in the repo. Then break it on purpose by deploying from a non-main branch and watch the token exchange fail with AADSTS700213.

Resume / interview bullets

  • Built a deterministic CI/CD DAG executor (stages, dependsOn, conditions, approval gates) that models Azure Pipelines / GitHub Actions execution and failure propagation.
  • Implemented an OIDC workload-identity-federation token-exchange evaluator that authenticates pipelines to Entra ID with no stored secret and denies tokens whose subject (branch/environment) is not the configured one.
  • Built blue-green and health-gated canary rollout controllers with automatic rollback.