Phase 02 — Terraform, CDKTF & IaC State Internals

Difficulty: ⭐⭐⭐☆☆ (the algorithm) → ⭐⭐⭐⭐⭐ (the judgment about state, replace, and locking) Estimated Time: 1 week (12–18 hours) Prerequisites: Phase 01 (the ARM deployment engine — the dependency graph, idempotent PUT, and what-if). This phase is the stateful sibling of that stateless engine, and the contrast is the whole lesson. No live Azure subscription required for the lab.

Why This Phase Exists

Phase 01 taught you the single most important sentence about ARM: every write is an idempotent PUT of desired state, and ARM derives the diff by asking the resource provider what's actually there. ARM is stateless — there is no file recording what it believes exists, so there is nothing to drift, nothing to lock, nothing to lose.

Terraform makes the opposite bet, and that one architectural decision is the reason it behaves the way it does — including every way it surprises people. Terraform keeps a state file: a recorded map from your configuration's addresses (azurerm_storage_account.main) to the real resource IDs and attributes it created. It trusts that file as its model of reality. The entire tool is the diff between the world you wrote (config) and the world it remembers (state), turned into a plan and executed by apply.

That design buys Terraform things ARM doesn't have — a fast plan that doesn't have to read every resource, a portable engine that works across 4,000+ providers, a single graph that spans Azure and DNS and GitHub and Datadog. And it costs Terraform things ARM never worries about:

State can lie. Someone edits a resource in the portal; Terraform's state still says the old value. That gap is drift, and it's why terraform plan refreshes before it diffs — and why a plan can show changes you didn't write.
State can be lost or corrupted, and losing it means Terraform forgets it owns your production estate — the most feared tfstate incident there is.
State must be shared and locked. Two engineers running apply against the same state at once corrupt it, so remote state takes a lock (an Azure Storage blob lease) for the duration of an apply.

And one decision dwarfs all the others in blast radius: the difference between update in place and replace (destroy-then-create). A change to a mutable attribute is a quiet ~. A change to an immutable (ForceNew) attribute — a storage account's location, a VM's certain SKUs, a database's name — is a -/+: Terraform will destroy the real resource and build a new one, with whatever downtime and data loss that implies. The principal skill is reading a plan and catching the -/+ before you type yes.

So this phase makes you build the plan/apply engine: the per-resource diff, the ForceNew replace decision, the dependency-graph ordering (and its reversal for deletes), and the drift detector. Once you've implemented it, Terraform stops being magic, CDKTF reveals itself as "the same engine with a real language on the front," and you can debug a confusing plan by reasoning about which of the three artifacts is wrong.

What "Principal-Level" Means Here

A senior engineer writes a working Terraform module and runs apply. A principal understands the engine well enough to:

Read a plan as a code review and flag every -/+ (replace) as a potential outage, every unexpected ~ as drift or a provider default, and every - as a deletion to justify — before approving.
Predict replace vs update from the provider schema: know which attributes are ForceNew and never let a "small change" silently recreate a database.
Reason about state as a separate, fragile artifact — where it lives (remote backend), how it's locked (blob lease), how it drifts, how it's recovered, and why terraform import and state mv/rm exist.
Explain Terraform vs ARM/Bicep at the model level: state-file diff vs stateless desired-state PUT, and when each model's failure mode bites.
Operate CDKTF knowing it changes the authoring layer (a real language, loops, abstractions) and not the engine — it synthesizes to Terraform JSON and the same plan/apply runs.

Concepts

The three artifacts. Configuration = the desired world (.tf / .tf.json / synthesized CDKTF). State = the recorded reality (terraform.tfstate: address → real ID + attributes). Plan = the diff of config against state, per resource, in dependency order. apply executes the plan and writes a new state. Hold these three apart and almost every Terraform mystery dissolves.
The plan algorithm. For each resource, compare config to state and emit exactly one of: create (in config, not in state, +), update-in-place (differs, only mutable attrs, ~), replace (differs, an immutable attr moved, -/+), delete (in state, not in config, -), no-op (identical). This five-way classification is Terraform.
ForceNew / immutability → replace. Provider schemas mark some attributes ForceNew: true because the underlying API cannot change them in place (you can't move a storage account to another region). Changing one forces destroy-then-create. The hardest, most valuable skill in this phase is recognising immutable attributes and the blast radius of recreating the resource.
Replace ordering. Default is destroy-before-create (downtime while the new one builds). The create_before_destroy lifecycle flag inverts it (new one up first, then old torn down) — essential for zero-downtime replaces, but it requires non-colliding names/addresses.
The dependency DAG. Terraform builds a graph from references (${azurerm_storage_account.main.id} is an edge) plus explicit depends_on. It applies in topological order (a resource after the things it needs) and destroys in reverse topological order (a dependent before the thing it depends on).
Drift detection. Real ≠ state because of out-of-band changes (the portal, another tool, a human at 2 a.m.). terraform plan refreshes — reads each resource through the provider — then diffs the refreshed reality against config, so drift shows up as changes you didn't write. State is a claim, continually re-checked, not ground truth.
Remote state + locking. State on a developer's laptop doesn't scale to a team. The Azure Storage backend keeps tfstate in a blob and takes a blob lease lock for each apply, so concurrent applies can't corrupt it. This is the backbone of team Terraform.
CDKTF. Author infrastructure in TypeScript/Python/Go/Java/C#, cdktf synth to Terraform JSON (cdk.tf.json), then run the same plan/apply engine. CDKTF buys you a real language (loops, conditionals, classes, tests) on top of an unchanged core.
Terraform vs ARM/Bicep (the P01 contrast). ARM/Bicep: stateless, idempotent desired-state PUT, Azure-only, no state file, drift is recomputed each deploy. Terraform: a state file it diffs against, multi-cloud/multi-provider, a plan you review, but a fragile artifact you must store, lock, and protect.

Labs

Lab 01 — Terraform State Engine (flagship, implemented)

Field	Value
Goal	Build a miniature Terraform plan/apply engine: a per-attribute diff, a four-way per-resource decision (create / no-op / update / replace) that turns a `ForceNew` change into a destroy-then-create, a `plan` that orders creates/updates topologically and deletes in reverse and marks orphaned state for deletion, an `apply` that returns a new state without mutating the input, and a `detect_drift` that finds out-of-band changes
Concepts	Config vs state vs plan; the five-way diff; `ForceNew` → replace and its blast radius; dependency-graph topological ordering and its reversal for deletes; idempotent re-plan; non-mutating apply; drift as real ≠ state
Steps	1. `diff_attributes` (the leaf comparison); 2. `plan_resource` (create/no-op/update/replace, the `ForceNew` boundary); 3. `plan` (classify all, mark orphans `delete`, topo-order creates/reverse-order deletes, summary counts); 4. `apply` (execute, return new state, never mutate input); 5. `detect_drift` (real vs recorded)
How to Test	`pytest test_lab.py -v` — 28 tests covering diff add/remove/change, the four-way decision and the `ForceNew` boundary, topo-order creates vs reverse-order deletes, orphan deletion, replace-on-immutable-change, apply correctness + input-immutability + determinism, and drift found vs none-when-synced
Talking Points	"What are Terraform's three artifacts and what does `apply` do?" / "When is a change an `update` and when a `replace`, and why does that matter at 2 a.m.?" / "Why does Terraform destroy in reverse dependency order?" / "What is drift and how does `plan` surface it?" / "Terraform vs Bicep — state file vs stateless `PUT`."
Resume bullet	Built a Terraform-style plan/apply state engine (per-attribute diff, `ForceNew` replace detection, dependency-graph topological ordering with reverse-order teardown, non-mutating apply, and out-of-band drift detection) to model IaC state internals

→ Lab folder: lab-01-terraform-state-engine/

Integrated-Scenario Suggestions (carried through the whole track)

The phases compound. Keep these in mind as you build — each plugs the state engine into a larger platform:

Team remote-state bootstrap — a storage account + container holding tfstate with blob-lease locking, the first thing any real Terraform org provisions (often itself created out-of-band or by a bootstrap module). The lock is the difference between a team that can run apply safely and one that corrupts state weekly. (→ P05, P12)
The dangerous replace — a config change that looks trivial (rename a storage account, move a region, change a SKU) but is a ForceNew → the plan shows -/+ and the apply destroys a production resource. Catching this in PR review is the whole skill. (→ P06, P11)
Drift reconciliation — a portal edit (someone bumped a tier in an incident) creates drift; the next plan shows changes nobody wrote. Decide: re-apply (config wins, revert the edit), import/update config (reality wins), or ignore_changes (accept the edit). This is the daily reality of shared estates. (→ P04, P05)
CDKTF for a landing zone — express a repeated pattern (N spokes, each a VNet + subnet + NSG) as a CDKTF loop in Python, synth to JSON, and watch the same engine plan it — the case where a real language beats hand-written HCL. (→ P05, P06)
Terraform vs Bicep on the same estate — model one resource group both ways and compare the failure modes: Bicep's stateless PUT (no state to lose, Azure-only) vs Terraform's state diff (multi-cloud, reviewable plan, fragile state). The ADR that chooses between them is a Round-5 architecture question. (→ P01, P15)

Guides in This Phase

HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
WARMUP.md — the full primer; read slowly
BROTHER-TALK.md — the candid version

Key Takeaways

Terraform is the diff between config and state. Three artifacts — config (desired), state (recorded reality), plan (the diff) — and apply executes the plan. Hold them apart and the tool stops surprising you.
ForceNew is the load-bearing concept. Update is a ~; replace is a -/+ that destroys real infrastructure. Reading a plan is reading for the -/+.
Order matters and reverses for deletes. Apply in topological order; destroy in reverse topological order. The graph drives both.
State is a fragile, separate artifact. It drifts (real ≠ state), it must be stored remotely, and it must be locked (Azure blob lease) so a team can share it. Most Terraform horror stories are state-management failures, not config bugs.
CDKTF changes the authoring layer, not the engine. Synthesize to Terraform JSON; the same plan/apply runs. A real language buys abstraction, not a different model.
Stateful (Terraform) vs stateless (ARM/Bicep) is a design tradeoff, not a quality ranking. State buys a portable, reviewable, multi-cloud diff; it costs you a fragile artifact to store, lock, and protect.

Deliverables Checklist

Lab 01 implemented; all 28 tests pass against solution.py and your lab.py
You can name Terraform's three artifacts and say exactly what apply does
Given a diff and a provider schema, you can call update vs replace correctly and explain the blast radius of a replace
You can explain why deletes run in reverse dependency order
You can define drift, explain how plan's refresh surfaces it, and list the three ways to reconcile it
You can explain remote state + blob-lease locking and why a team needs it
You can contrast Terraform's state model with ARM/Bicep's stateless PUT (P01)

Principal Azure Cloud Engineer & Architect