Phase 10 — Event-Driven: Event Grid, Service Bus & Logic Apps

Difficulty: ⭐⭐⭐⭐☆ (the mechanism) → ⭐⭐⭐⭐⭐ (the delivery-semantics judgment) Estimated Time: 1.5 weeks (16–22 hours) Prerequisites: Phase 03 (Entra ID — the identity a publisher/consumer authenticates with), Phase 09 (REST/JSON + AuthZ — the shape of a webhook handler), and a working mental model of idempotency from Phase 01 (ARM PUT) and Phase 02 (Terraform apply). This phase is where those threads converge into distributed-systems delivery guarantees.

Why This Phase Exists

Up to here the curriculum has been about the control plane (ARM, Terraform, RBAC, landing zones, networking) and synchronous request/response (the API gateway in P09). But the platforms the JD describes are event-driven: a service emits an event, and one or more other services react — asynchronously, decoupled, at their own pace. The moment you decouple producer from consumer with a network and a buffer in between, you have signed up for the hardest problems in distributed systems: messages get lost, duplicated, reordered, and redelivered, and "it worked in the demo" tells you nothing about what happens when a consumer crashes mid-process at 50,000 messages/second.

A senior engineer wires up a Service Bus queue and an Event Grid subscription. A principal understands the delivery state machine underneath well enough to answer the only questions that matter:

• When a consumer pulls a message and then crashes, what happens to that message? (the peek-lock + lock-expiry redelivery answer)
• When the same event is delivered twice, does the system double-charge a customer? (the idempotency + exactly-once-effect answer)
• When a message can never be processed, where does it go and who gets paged? (the max-delivery → dead-letter-queue answer)
• When you need strict order, what does that cost you in throughput? (the sessions answer)
• Should this be a queue, a topic, a stream, or a reactive event? (the Service Bus vs Event Hubs vs Event Grid answer — the most common architecture mistake in the whole space)

The JD names it directly: "Design and implement event-driven architectures using Azure Event Grid, Azure Service Bus, and Azure Logic Apps." This phase makes you build the two delivery engines — the Service Bus broker and the Event Grid retry-delivery path — so the guarantees become arithmetic and a state machine you can draw, not a feature you hope works.

Concepts

• Broker vs router vs log — the three event-plane shapes, and the single decision that picks one:
  • Azure Service Bus — an enterprise message broker. A consumer pulls a message, locks it (peek-lock), processes, and settles it (complete/abandon/dead-letter). Queues (point-to-point) and topics/subscriptions (pub/sub with filters). The home of ordering (sessions), duplicate detection, scheduled/deferred messages, transactions, and dead-letter queues. Use it for commands and workflow steps that must not be lost and may need order or transactions.
  • Azure Event Grid — a reactive event router. The publisher fires a small CloudEvents-shaped notification ("a blob was created", "a resource changed") and Event Grid pushes it to subscribers' webhooks/handlers, at-least-once, with exponential-backoff retry and dead-lettering to storage. Use it for "X happened, react" fan-out at massive scale with no consumer to manage.
  • Azure Event Hubs — a high-throughput partitioned log (Kafka-shaped): millions of events/second, consumers track offsets, events are replayable within a retention window. Use it for telemetry/streaming/analytics ingestion, not discrete commands. (Named here for the contrast; built fully in the streaming track.)
• Peek-lock delivery — the receive-then-settle state machine: receive locks a message invisible for LockDuration; the consumer must complete (remove), abandon (release now), or dead_letter it before the lock expires. Lock expiry → redelivery + DeliveryCount++; once a message has been delivered MaxDeliveryCount times it moves to the dead-letter queue (DLQ) instead of being redelivered. This is the crash-safety mechanism: a dead consumer's message comes back automatically.
• Delivery semantics — the spectrum every async system lands on:
  • at-most-once — fire-and-forget; no retry; may lose (Service Bus ReceiveAndDelete, UDP-style).
  • at-least-once — retry until acknowledged; may duplicate (peek-lock without settle-on-success-only, Event Grid push). The practical default.
  • exactly-once-effect — at-least-once plus an idempotent or de-duplicating consumer so a duplicate is a no-op. The only "exactly-once" that exists across a network boundary, and a per-data-product dial, not a free platform setting.
• Idempotency keys & duplicate detection — a producer stamps a MessageId; Service Bus drops a repeat within the dedup window; a consumer keys side effects on it so a redelivery is harmless. The same idea as ARM's idempotent PUT, now on the data plane.
• Sessions / FIFO — ordering is not free: a session locks all messages of one SessionId to a single consumer, in order, trading parallelism for guaranteed sequence.
• CloudEvents — the CNCF interoperability envelope (id, source, type, subject, time, data) Event Grid speaks, so events are portable across clouds and tools.
• Logic Apps — serverless workflow orchestration (a trigger + a graph of connector actions) — the low-code integration complement to code-first Functions (P11). It is how "when an event arrives, call these five SaaS systems in order with retries" gets built without a service.

Labs

Lab 01 — Messaging Broker (flagship, implemented)

→ Lab folder: lab-01-messaging-broker/

Integrated-Scenario Suggestions

These carry the phase into the rest of the track — each is a real event-platform problem the delivery engines underpin:

1. Order-processing pipeline — an OrderPlaced command flows Service Bus queue → inventory consumer → payment consumer, each step idempotent on OrderId, poison orders to the DLQ with an alert, and a Logic App fanning notifications. Which steps need sessions (per-customer ordering)? Where do you re-earn exactly-once at each boundary? (Uses the whole lab; builds on P09 auth, P12 Key Vault for the payment secret.)
2. Reactive blob-processing fan-out — Microsoft.Storage.BlobCreated → Event Grid → N subscribers (thumbnailer, virus-scan, indexer), each filtered by subject prefix, with dead-letter-to-storage and retry budgets. When does at-least-once double-process a blob, and how does an idempotent handler fix it? (Builds on P07 containers, P11 Functions.)
3. Telemetry vs commands split — the same product emits both high-volume device telemetry and discrete control commands. Route telemetry to Event Hubs (partitioned log, replay) and commands to Service Bus (ordered, transactional, DLQ). Defend the split with the read:write ratio and the loss/duplicate cost per stream.
4. Dead-letter triage runbook — given a DLQ filling at 200 msg/min, write the playbook: classify the failure (poison payload vs downstream outage vs lock-too-short), decide replay vs discard, and fix the root cause (raise MaxDeliveryCount? lengthen LockDuration? add idempotency?). (Builds on P13 observability, P14 reliability.)
5. Saga / workflow orchestration — a multi-system business process (book flight → charge card → email) as a Logic App (or Durable Function, P11) with compensation steps, retries, and a timeout, sitting on top of Service Bus for the durable steps. Where does the orchestration state live, and how does a replay stay deterministic?

Guides in This Phase

• HITCHHIKERS-GUIDE.md — the 30-minute orientation; read first
• WARMUP.md — the full primer; read slowly
• BROTHER-TALK.md — the candid version

Where This Phase Sits in the Track

This phase is the asynchronous backbone: it turns the synchronous request/response world of P09 into the decoupled, buffered, retry-driven platform the business actually runs on.

The mental model to carry forward: decoupling is a buffer plus a delivery guarantee, and the guarantee is at-least-once unless you engineer the duplicate away. Every later phase that reacts to something is standing on one of the two engines you build here.

Key Takeaways

• Service Bus is a broker (pull + lock + settle), Event Grid is a router (push + retry), Event Hubs is a log (stream + offsets). Picking the wrong one is the most common — and most expensive — event-architecture mistake. Match the shape to the workload.
• Peek-lock is the crash-safety mechanism: receive locks a message invisible; settle it, or the lock expires and it's redelivered with DeliveryCount++; after MaxDeliveryCount deliveries it dead-letters. A dead consumer never silently loses a message — that's the whole point.
• The default guarantee is at-least-once, which means duplicates happen. "Exactly-once" end-to-end is a myth across a network; what you build is at-least-once + an idempotent / de-duplicating consumer = exactly-once-effect, chosen per data product by what a duplicate (or a loss) actually costs.
• Ordering is not free: a session trades parallelism for FIFO. Don't pay for ordering a stream doesn't need.
• Duplicate detection (MessageId) and idempotency keys are the same idea as ARM's idempotent PUT, now on the data plane — stamp the producer, key the consumer, and a redelivery becomes a no-op.
• The DLQ is a bulkhead, not a graveyard: a poison message that can't be processed must leave the hot path (so it stops blocking healthy traffic) and raise an alert (so a human triages it) — that is what MaxDeliveryCount buys you.

Deliverables Checklist

• Lab 01 implemented; all tests pass against solution.py and your lab.py
• You can draw the peek-lock state machine (receive → lock → complete/abandon/ dead-letter; lock-expiry → redeliver + ++; max-delivery → DLQ) from memory
• You can state, for a given workload, whether it wants Service Bus, Event Grid, or Event Hubs — and defend it with the read:write ratio and the loss/duplicate cost
• You can explain why now == lock_until counts as expired (the off-by-one)
• You can turn a "we got charged twice" bug into the exactly-once-effect fix (idempotency key on the consumer), and say where the key comes from
• You can name what sessions cost (throughput) and when ordering is actually required
• You can sketch an Event Grid retry policy (backoff schedule, TTL, dead-letter target) and a Logic App workflow that consumes from it