👨🏻 Brother Talk — Phase 11, Off the Record

No slides, no STAR method. This is me, your brother, telling you the things people only say after the second coffee. Read it once now, and again the week before your interview.

Listen. Almost everyone who's "done serverless" has written an Azure Function, watched it fire when a message landed on a queue, and moved on feeling clever. And it is clever — right up until the day it scales in a way you didn't expect, or someone asks you "so how does it actually decide to add instances?" and you realize you've been saying the word "autoscale" like it's an answer. It isn't. "Autoscale" is the noise you make when you don't know the mechanism. The mechanism is a scale controller reading a backlog and doing ceil(backlog / target). Once you can say that, out loud, with the clamp and the gradual ramp and the scale-to-zero, you've stepped over a line most people never cross. That line is this whole phase.

Here's the thing that took me too long to internalize, and it's the most important sentence in this phase: a Durable orchestrator is not running. It's being replayed. Sit with that, because it's so counterintuitive that even people who use Durable Functions every day don't truly hold it. You write what looks like normal sequential code — call this, await that, then this — and it reads like a process sitting there waiting. It is not. Between every step, your orchestrator isn't running at all; its entire state lives in an append-only history in a storage account. When the next thing happens, the platform re-runs your function from the very first line, fast-forwarding through everything already done by replaying recorded results, until it hits the one new thing to do. Every line of your orchestrator runs dozens of times over the life of the workflow. When that finally clicks — when you can feel the replay — Durable stops being magic and the determinism rule stops being a scary warning and becomes obvious.

And the determinism rule is the whole ballgame, so let me hammer it. Your orchestrator must be a pure function of its history. No DateTime.Now. No Guid.NewGuid(). No random(). No calling an API, a database, a file, anything with a side effect — directly. Why? Because all of those return a different value the next time the line runs — and the line runs again on every replay. The first time, random() returns 0.7 and your code calls activity A and the history records "A was called." The next replay, random() returns 0.3, your code wants to call B, and now the running code disagrees with the recorded history. That's not a warning; that's corruption, and the platform throws a non-determinism error to stop it. The fix is beautiful in its simplicity: anything nondeterministic goes into an activity (which runs once and has its result recorded) or through the ctx API (current_utc_datetime, new_guid, durable timers — each one exists precisely because the raw primitive is unsafe under replay). When you can explain in an interview why current_utc_datetime exists — "so the clock read is recorded once and replayed, instead of returning a new time every replay" — you've shown them you understand event sourcing, not just an API.

Now let me give you the things that actually bite people.

Cold start is a tradeoff, not a bug — and the level-up is treating it like one. Juniors panic about cold start and reach for Premium everywhere, and now they're paying for warm instances 24/7 to dodge a 600 ms hiccup nobody on a nightly batch job will ever notice. The principal move is to compute it: pull out the latency budget from Phase 00 and ask "does this path's SLA care about an occasional cold start?" If it's a background job with a 5-minute budget — Consumption, scale to zero, save the money, cold start is noise. If it's a user-facing API with a 200 ms p99 — yes, pay for Flex always-ready or Premium pre-warmed on that path. Same system, two answers, decided by a number. That's the whole game again: price the tradeoff, don't fear it.

The scale limit is a blast-radius control, and forgetting it is a 2 a.m. story. Here's the one that gets people: the function scales beautifully, a burst comes in, 200 instances spin up — and all 200 open connections to the same database, which falls over. The function did its job too well. The fix isn't to make the function slower; it's to cap the scale limit (functionAppScaleLimit) so the platform can't hammer a shared downstream harder than it can take. This is Phase 00's blast-radius thinking applied to compute: a thing that scales without a cap is a thing that can take down everything it talks to. A principal sets the cap before the incident and can explain why the number is what it is.

Idempotency is still your friend, but for the right reason. People conflate two things here, so get this straight: plain replay does not re-run your completed activities — it replays their recorded results. So replay isn't why activities need to be idempotent. They need to be idempotent because of retries and at-least-once delivery (the Phase 10 stuff): an activity can fail partway, get retried, and run twice; the trigger source can deliver the same message twice. So "make activities idempotent" is true — just know it's the retry/delivery path that demands it, not replay. Saying that precisely in an interview is a quiet flex; it shows you actually understand both mechanisms instead of mushing them together.

The honest truth about this phase: the code is easy. A function is a few lines; Durable patterns are a handful of yields. What makes this principal-level is that the two core mechanisms — scale-from-backlog and replay-forces-determinism — are invisible until they bite, and when they bite they bite in production, not in your local test. Nobody's local test catches the cold start, the burst stampede, or the DateTime.Now that corrupts a replay after the third event. So the work isn't writing the function; it's holding the mechanism in your head clearly enough to predict those failures before they happen. That's why we build the runtime in this lab — write the scale controller and the replay engine by hand, and the mechanisms stop being invisible. You'll see the backlog become an instance count, and you'll watch a divergent orchestrator throw DeterminismError. After that, you can't un-see it, and you'll spot the bug in a code review before it ships.

And here's the career angle, brother to brother. Serverless is where the platform meets the product — it's the tier where the business logic actually runs, triggered by your events (P10), authenticated by your identities (P12), fronted by your gateway (P09). The people who own this tier are trusted with the part of the system that does the work, and they're the ones in the room when something scales wrong or a workflow gets stuck. If you can be the person who says "that's a determinism violation on line 14, here's why replay breaks on it, here's the fix" — calmly, in a code review or an incident — you're not a function-writer anymore. You're the person who understands the runtime. That person gets handed the gnarly distributed workflows, and gnarly distributed workflows are where principals are made.

Go build the runtime. Watch the backlog turn into instances. Put a random() in an orchestrator and watch your engine reject it on replay. Then come find me in Phase 12 — that's where the function you just scaled gets a Managed Identity and stops carrying secrets entirely.

— your brother 👨🏻