Lab 02 — CI/CD Pipeline Engine with Quality & Security Gates

Phase 28 · Lab 02 · Phase README · Warmup

The problem

Every serious platform team runs the same pipeline shape, whether it's spelled Jenkins, GitLab CI, GitHub Actions, or Argo Workflows: build → unit-test → image-scan → integration-test → sign, followed by staged promotion of the resulting artifact through dev → staging → prod. The parts that make it production-grade — and the parts interviewers actually probe — are the gates:

1. A quality gate: all unit tests pass AND coverage meets a threshold, or the pipeline stops.
2. A security gate: the container image is scanned (Trivy is the canonical tool) and any CRITICAL/HIGH finding above policy blocks the pipeline — with an explicit, auditable accepted-risk waiver list, never a silent skip.
3. Promote the artifact, never rebuild it: the digest that passed the gates is the digest that ships. A rebuild "for prod" is a different artifact that passed nothing.
4. Sign after the gates, verify before every deploy: the signature over the digest is the enforcement mechanism for rule 3 — an unsigned, tampered, or wrongly-keyed artifact never deploys.
5. Prod requires a human: a manual-approval gate on the last environment.
6. Rollback is a first-class path: last-good artifact, one operation, signature re-verified.

What you build

File map

Run it

pytest test_lab.py -v                       # your lab.py (red until you implement)
LAB_MODULE=solution pytest test_lab.py -v   # the reference (green)
python solution.py                          # worked example

Success criteria

• A fully green context passes all five stages and exits with a signed artifact.
• A failed gate stops the pipeline immediately — later stages never execute, and the artifact is never signed.
• A CRITICAL or HIGH vulnerability blocks at image-scan; MEDIUM/LOW pass the default policy; an ID on the ignored list is waived even at CRITICAL.
• Coverage below the threshold blocks; coverage exactly at the threshold passes (boundary).
• verify_signature rejects unsigned artifacts, tampered digests, and wrong keys.
• Promotion enforces the ladder: staging requires dev, prod requires staging and approved=True.
• A failed run's artifact cannot be promoted anywhere.
• rollback restores the previous artifact (and re-verifies its signature); rolling back with no prior deployment is a structured error.
• All 29 tests pass under both lab and solution.

How this maps to the real stack

• The stage sequence is what a Jenkinsfile (stages { stage('Build') ... }), a .gitlab-ci.yml (stages: [build, test, scan, deploy]), or a GitHub Actions workflow (jobs.<id>.needs) declares. Our "stage = pure function over a context dict, pipeline = ordered fail-fast execution" is the engine all three implement, minus the distributed runners, caching, and log streaming.
• The severity gate is exactly trivy image --severity HIGH,CRITICAL --exit-code 1 in a CI step: a non-zero exit blocks the job. Our ScanPolicy.ignored is .trivyignore — a versioned, reviewable list of accepted CVE IDs, which is the correct way to waive a finding (versus lowering the severity gate for everyone). Grype/Snyk/Clair slot into the same seam. The injected vulnerabilities list is the scanner's JSON report, which in real life comes from matching the image's packages against vulnerability databases (the NVD and distro feeds).
• The coverage gate is SonarQube's quality gate / coverage report --fail-under=80 / GitLab's coverage keyword — the mechanism is identical: a threshold check that fails the job. Real static analysis (SonarQube, Semgrep, ruff/golangci-lint) is one more gate of the same shape, which is why this lab's StageResult pattern extends to it directly.
• Content-addressed digests are OCI image digests: sha256: over the image manifest. "Promote the artifact, never rebuild" is the real practice of promoting by digest (not by mutable tag — :latest is how staging and prod silently diverge).
• Signing maps to cosign (Sigstore) or Notary/Notation: sign the image digest after the pipeline's gates, store the signature alongside the image, and enforce verify-before-deploy in the cluster with an admission controller (Kyverno's verifyImages, sigstore's policy-controller). Our symmetric keyed hash stands in for cosign's keypair — the mechanism (sign the digest, re-derive on deploy, fail closed) is the same; real cosign adds asymmetric keys, keyless OIDC signing, and a transparency log (Rekor). The SBOM tuple is the miniature of an SPDX/CycloneDX SBOM attached as an attestation.
• Staged promotion + manual approval map to GitLab protected environments and when: manual jobs, GitHub Actions environment protection rules (required reviewers), and Argo CD's sync gates. In a GitOps setup (Phase docs, §GitOps) promotion is a Git commit bumping the image digest in the env's manifest repo, and Argo CD reconciles it — the ladder is the same, the mechanism is a pull-based controller instead of a push-based deploy job.
• Rollback to last-good is helm rollback, kubectl rollout undo, or reverting the GitOps commit — all three are "restore a previously recorded, previously verified artifact/manifest," which is why the lab stores history at deploy time instead of trying to reconstruct it during the incident.

Limits of the miniature. Stages run synchronously in-process — no runners, no artifact registry, no caching, no parallel fan-out (real pipelines run unit tests and scans concurrently and join at the gate). The signature is a symmetric keyed hash, not cosign's asymmetric signature with certificate transparency. Approval is a boolean parameter, not an audit-logged review with identity. And promotion writes a dict, not a Git commit — the GitOps variant is an extension below.

Extensions (your own machine)

• Add a static-analysis stage (lint findings above a severity block, same StageResult shape) between build and unit-test, and watch how little new machinery it needs.
• Make promotion GitOps-shaped: promotion writes {"env": ..., "digest": ...} records to an append-only "manifest repo" list, and a tiny reconciler diffs desired-vs-deployed and applies — then implement drift: hand-edit deployed and watch the reconciler put it back.
• Add attestations: attach the SBOM and the scan report to the artifact as signed metadata, and make the prod deploy require both attestations to be present and verified.
• Run real Trivy against a real image (trivy image python:3.12-slim --severity HIGH,CRITICAL) and diff its JSON report shape against this lab's Vulnerability rows.
• Wire a real cosign keypair (cosign generate-key-pair, cosign sign, cosign verify) around a local registry (registry:2) and compare with sign_digest/verify_signature.

Interview / resume signal

"Built a deterministic CI/CD gate engine: fail-fast staged pipeline (build → unit-test → image-scan → integration-test → sign) with a Trivy-style CRITICAL/HIGH severity gate and auditable accepted-risk waivers, a coverage quality gate, content-addressed artifacts signed after the gates and verified before every deploy (tamper = fail closed), staged promotion dev→staging→prod with a manual prod approval, and one-operation rollback to the last-good signed artifact."