Lab 01 — Functions Runtime: Scale Controller + Durable Replay

Phase: 11 — Serverless: Functions, Triggers/Bindings & Durable | Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

Azure Functions looks like magic: you write a function, an event shows up, and somehow the right number of instances appear to process it. Pull the cover off and there are two mechanisms doing the magic — the scale controller that turns an event-source backlog into a desired instance count (and scales to zero when the work drains, which is where cold start comes from), and, for stateful workflows, the Durable replay engine that runs your orchestrator by re-executing it from its recorded history on every event — which is exactly why an orchestrator must be deterministic. This lab builds both: a target-based scale controller with a per-step burst clamp, a trigger/binding dispatcher, and — the centerpiece — a Durable orchestration replay engine that reproduces the same sequence of activity calls from a partial vs full history and raises DeterminismError the moment your orchestrator diverges.

What you build

desired_instances(...) / scale_decision(...) — the scale controller. Target-based: ideal = ceil(backlog / target_per_instance), clamped to [0, max_instances], and rate-limited to move at most max_scale_out_step instances from current per decision (gradual scale-out, no stampede). Backlog 0 → 0 instances (scale-to-zero). The scale_decision returns "scale_out" / "scale_in" / "hold".
Trigger / Binding / FunctionDef / dispatch(...) — the trigger/binding model. A Trigger (type + optional filter) decides which function an event invokes; the function's declared output bindings collect its return value. dispatch routes one event to the one matching function and returns the result plus its output bindings.
The Durable replay engine (run_orchestrator, OrchestrationContext, CallActivity, CurrentUtc, HistoryEvent, DeterminismError) — an orchestrator is a Python generator that yields CallActivity instructions and receives results back. The engine replays the generator from the start against a recorded history: for each yielded call it either replays the recorded result (matched by sequence index) or — for the next un-recorded step — executes the activity, appends the result to history, and continues. Fan-out/fan-in (yield a list of CallActivity → a list of results) and a deterministic current_utc() (served from history, never the wall clock) are built in. A divergent replay raises DeterminismError.

Key concepts

Concept	What to understand
Target-based scaling	`desired = ceil(backlog / target_per_instance)`; the controller chases a per-instance work target, not CPU
Scale to zero	backlog `0` → `0` instances; the next event pays a cold start to spin one up (the Consumption tradeoff)
Per-step burst clamp	the runtime adds a bounded number of instances per decision — protects the source and downstream from a stampede
Max instances	the plan's hard ceiling (Consumption ~200, others configurable); `ideal` is clamped to it
Trigger vs binding	the trigger invokes the function (one per function); bindings declaratively wire inputs/outputs
Orchestrator = generator	it `yield`s instructions and is replayed; it must be a pure function of its history
Event-sourced replay	the engine re-runs the orchestrator from history every turn; recorded steps replay, the next step executes
Determinism	no `datetime.now`, no `random`, no I/O outside activities — or the replay diverges and Durable throws
`DeterminismError`	the yielded call no longer matches history (name/input/kind) → the orchestrator is non-deterministic
Deterministic time	`ctx.current_utc()` reads a recorded timestamp from history, so every replay sees the same "now"

Files

File	Purpose
`lab.py`	skeleton with `# TODO` markers
`solution.py`	complete reference; `python solution.py` runs a worked example (scale table, chaining, fan-out/fan-in, a caught divergence)
`test_lab.py`	the proof — run it red, make it green
`requirements.txt`	pytest only (pure stdlib otherwise)

Run

pip install -r requirements.txt
pytest test_lab.py -v                      # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v  # against the reference
python solution.py                         # worked example

Success criteria

All 22 tests pass against your implementation.
You can explain why test_desired_scales_to_zero_on_empty_backlog is the line that creates cold start: scale-to-zero means there is no warm instance for the next event.
You can explain why test_desired_burst_limited_per_step matters — a 1000-message burst doesn't instantly become 1000 instances; the controller climbs +max_scale_out_step per decision so it doesn't stampede the queue or your database.
You can explain why test_replay_partial_vs_full_same_call_order is the whole point of the Durable engine: the orchestrator is a pure function of its history, so replaying it from a partial history reproduces the identical ordered call sequence as a full run.
You can explain why test_replay_does_not_re_execute_recorded_activities is the efficiency and correctness guarantee: a recorded activity replays its result; only the next un-recorded activity actually runs.
You can explain why test_determinism_error_on_divergent_replay exists — a random() branch or a datetime.now() in the orchestrator makes a later replay yield a different call than history recorded, and Durable detects exactly that mismatch.

How this maps to real Azure (Azure Functions + Durable Functions)

The lab	Real Azure
`desired_instances`	the scale controller, which polls the trigger's event source and computes a target instance count; modern triggers use target-based scaling (a per-instance target)
`backlog`	queue message count (Storage/Service Bus), Event Hub lag (unprocessed events per partition), the depth the controller reads
scale to `0`	Consumption and Flex Consumption scale to zero when idle; the next trigger pays a cold start to spin an instance up
`max_instances`	the plan's instance cap — Consumption defaults to ~200, Premium/Flex/Dedicated are configurable
`max_scale_out_step`	the controller adds instances gradually (a bounded number per ~second), not all at once
`Trigger`	the function's single trigger — `queueTrigger`, `httpTrigger`, `timerTrigger`, `eventGridTrigger`, `serviceBusTrigger`, `eventHubTrigger`
`Binding` (out)	an output binding (`queue`, `blob`, `cosmosDB`, `serviceBus`) — the host writes the return value without you writing client code
`run_orchestrator`	the Durable Task Framework replay loop inside the Durable extension
`CallActivity`	`context.df.callActivity(name, input)` (JS) / `context.call_activity(name, input)` (Python) / `context.CallActivityAsync` (.NET)
`history`	the orchestration's history table in the Durable storage provider (Azure Storage / Netherite / MSSQL) — the event-sourced log replayed on each event
`CurrentUtc`	`context.current_utc_datetime` — the deterministic replay-safe clock
`DeterminismError`	Durable's non-determinism detection — it logs/raises when replayed code produces a different action than history recorded
fan-out / fan-in	the canonical Durable pattern: `yield` a list of activity tasks, then aggregate (`when_all` / `Task.WhenAll`)

What the miniature leaves out (and why it's fine): real Functions runs over HTTP/gRPC to a language worker, persists history to a durable store with leases and partitioning, checkpoints long orchestrations, supports durable timers, external events (human interaction), sub-orchestrations, and eternal orchestrations (continue_as_new), and the scale controller integrates with KEDA on Kubernetes. None of that changes the two mechanisms this lab isolates — (1) a backlog becomes a clamped, step-limited instance count, and (2) an orchestrator is replayed from history and therefore must be deterministic. Those are exactly the parts interviewers probe and on-call incidents turn on.

Extensions (build these in your own subscription)

Durable timers: add CreateTimer(fire_at) — yield a timer instruction the engine resolves from history (a recorded fire time), so a "wait 1 hour then check" monitor replays deterministically. This is how the monitor pattern works.
External events / human interaction: add WaitForExternalEvent(name) plus a RaiseEvent API and a timeout (a CreateTimer raced against the event with when_any) — the human-interaction-with-timeout pattern (approve within 72h or escalate).
Sub-orchestrations: let an orchestrator yield CallSubOrchestrator(fn, input) and have the engine recurse, recording the child's completion in the parent's history.
continue_as_new: support an eternal orchestrator that resets its history to avoid unbounded growth (how a perpetual monitor keeps its history small).
Wire it to real Azure: func init / func new an orchestrator + activity + HTTP starter; deploy with az functionapp create; trigger it and watch the history table in the storage account fill with TaskScheduled / TaskCompleted events as it replays. Then put a random.random() in the orchestrator and watch Durable's non-determinism detection fire.

Interview / resume

Talking points: "How does the Functions scale controller decide instance count, and where does cold start come from?" / "Why must a Durable orchestrator be deterministic — what exactly breaks if it calls DateTime.Now?" / "Walk me through fan-out/fan-in and how replay reconstructs state without re-running completed activities." / "Consumption vs Premium vs Flex — when do you pay for pre-warmed instances?"
Resume bullet: Built a miniature Azure Functions runtime — a target-based scale controller (backlog→instances with scale-to-zero, max-instance clamp, and per-step burst limiting) and an event-sourced Durable Functions replay engine (function chaining, fan-out/fan-in, deterministic replay-safe time, and non-determinism detection) — modeling the Functions host and the Durable Task Framework.

Principal Azure Cloud Engineer & Architect