👨🏻 Brother Talk — Phase 02, Off the Record
No slides, no certification objectives. This is me, your brother, telling you what I wish someone had told me before I learned it the expensive way. Read it once now, and again the day you're about to run an
applyagainst production.
Listen. Almost everyone who "knows Terraform" knows it the way you know how to drive without
knowing what's under the hood: you can get from A to B, and you're completely lost the moment
something goes wrong. They can write a module, run apply, and feel productive. And then one
day the plan says -/+ next to a database, they don't notice, they type yes, and the
database is gone. Not down. Gone. Recreated empty.
I'm not telling you that to scare you. I'm telling you because the entire difference between the person who causes that outage and the person who catches it in review is one habit: they read the plan. That's it. That's the whole separation between senior and principal in IaC. Not memorizing more resource types. Reading the plan like it's a loaded document, because it is.
So let me give you the things that actually matter, brother to brother.
The state file is not a config file. It's a load-bearing wall. Here's the mental shift
that took me too long: config is just text you can rewrite anytime. State is Terraform's
memory of what it owns in the real world. Lose it and Terraform doesn't lose your config
— it loses the knowledge that your production estate belongs to it. The next plan looks at
an empty memory and a full config and cheerfully proposes to create everything you already
have. I've watched a team spend two days terraform import-ing a production environment back
into a state file someone deleted "to start clean." Treat state like you'd treat the master
key to the building: remote, locked, versioned, access-controlled, and never on a laptop.
Fall in love with the -/+. Everyone's eyes glaze over reading a plan — it's long, it's
green, it scrolls. Train yourself to do the opposite of glaze: scan for -/+ and - first,
every single time, before you read anything else. Those two symbols are the only ones that
can hurt you. + creates, ~ tweaks — fine. -/+ destroys and recreates; -
destroys. The moment you make "grep the plan for -/+" a reflex, you've internalized the
one thing that makes you safe to give production access to. I literally pipe my plans through
a grep for it. It's not paranoia; it's the job.
Immutability is a fact about the world, not a Terraform quirk. You can't move a house to
a different city; you build a new one and tear the old one down. A storage account's region,
a resource's name, certain VM SKUs — same thing. ForceNew isn't Terraform being annoying;
it's Terraform being honest that the underlying cloud API physically cannot change that
attribute in place. So when a "small rename" turns into a -/+, don't fight it — understand
it. The question is never "why is Terraform recreating this?" It's "is there data or identity
behind this resource that recreation destroys?" If yes, you have a real engineering problem
(move the data, use create-before-destroy, or don't make the change) — not a Terraform
problem.
Drift is not Terraform's failure; it's the world being the world. People get weirdly
moralistic about drift — "someone touched the portal, that's against the rules." Sure, reduce
the surface, restrict portal access, alert on drift. But also: sometimes the person who
edited the resource in the portal at 3 a.m. was right — they were stopping an incident and
didn't have time for a PR. The principal move isn't to reflexively re-apply and revert their
fix. It's to ask why the drift exists and choose: revert it, adopt it into config, or
ignore it. A blind terraform apply that undoes someone's emergency fix is its own incident.
Drift is a conversation, not a violation.
Locking is the cheapest insurance you'll ever buy, and force-unlock is a loaded gun.
The whole reason remote state takes a blob lease is so two people can't apply at once and
shred the state. It works silently and you never think about it — until someone's apply
crashes and leaves an orphaned lock, and the "fix" everyone reaches for is force-unlock.
Here's the trap: if you force-unlock while someone's apply is actually still running, you
just re-created the exact corruption the lock was preventing. Use it only when you're certain
the lock is orphaned. When in doubt, find the human, don't unlock.
CDKTF won't save you from understanding the engine — and that's good news. Some people
reach for CDKTF hoping a real language will let them skip learning how Terraform actually
works. It won't. CDKTF is just a fancier way to write the config; it synthesizes to
Terraform JSON and runs the identical plan/apply, the identical state file, the identical
-/+ rules. Which means the thing you're learning in this lab — the engine — is exactly what
makes you good at CDKTF for free. Learn the engine once, and HCL, CDKTF, and honestly the
shape of ARM and Pulumi and CloudFormation all become front-ends you already understand.
That's the leverage of going under the hood: you stop learning tools and start recognizing
the same machine wearing different clothes.
The honest truth about this phase's difficulty: the algorithm is not hard. A five-way
diff and a topological sort — you'll write it in an afternoon. What's hard is the judgment:
knowing that a -/+ on a stateless thing is fine and on a database is a catastrophe, knowing
when drift should be reverted versus adopted, knowing that state is the fragile thing to
protect and not the config. That judgment is what the interview is actually testing when they
ask "walk me through what happens when you run apply." They're not checking if you can
recite the steps. They're checking if you respect the state file and fear the -/+ — because
that respect is the difference between someone you can trust with production and someone whose
first bad day takes the company down with it.
One more thing, and it's the one that compounds. Write down which resources can never be
replaced, and protect them. Put prevent_destroy on your stateful crown jewels — the
production database, the state storage account itself, the Key Vault. It's a lifecycle flag
that makes the plan fail if anything would destroy them. It feels like overhead until the
day it turns a 2 a.m. catastrophe into a clean error message that says "no." That's the whole
game in this role: turning the irreversible into the merely annoying, on purpose, before you
need it.
Go build the state engine. Read every plan for the -/+ from now on. Then come find me in
Phase 03, where we make identity click — OAuth, OIDC, JWTs — the stuff everyone name-drops and
quietly can't draw on a whiteboard.
— your brother 👨🏻