Warmup Guide — Event-Driven: Event Grid, Service Bus & Logic Apps

Zero-to-principal primer for Phase 10: what an event-driven architecture actually is, the three event-plane shapes (broker, router, log) and the one decision that picks between them, the peek-lock delivery state machine under the hood of Service Bus, the at-least-once + exponential-backoff push model under Event Grid, the delivery-semantics spectrum (and why "exactly-once" is a precise lie), sessions and ordering, duplicate detection, dead-lettering, CloudEvents, and Logic Apps — every term from first principles, the mechanism, the math, and the failure signature.

Table of Contents

• Chapter 1: Why Event-Driven — Decoupling Is a Buffer Plus a Guarantee
• Chapter 2: The Three Shapes — Broker vs Router vs Log
• Chapter 3: Service Bus — The Broker and Its Entities
• Chapter 4: Peek-Lock — The Delivery State Machine Under the Hood
• Chapter 5: Dead-Lettering and the Poison-Message Guard
• Chapter 6: Duplicate Detection, Scheduled & Deferred, Transactions
• Chapter 7: Sessions — Ordering Has a Price
• Chapter 8: Delivery Semantics — The Precise Lie of "Exactly-Once"
• Chapter 9: Event Grid — Reactive Routing, CloudEvents, and Retry
• Chapter 10: Event Hubs — The Streaming Sibling (for Contrast)
• Chapter 11: Logic Apps — Serverless Workflow Orchestration
• Lab Walkthrough Guidance
• Success Criteria
• Interview Q&A
• References

Chapter 1: Why Event-Driven — Decoupling Is a Buffer Plus a Guarantee

From zero. A synchronous call — the API gateway of Phase 09 — is a phone call: the caller waits, holds the line, and fails if the callee is down. An event-driven architecture is the postal system: the producer drops a message in a box and walks away; the consumer picks it up when it's ready. The producer does not know who consumes, how many consumers there are, or whether they're up right now. That is decoupling, and it buys three things you cannot get synchronously:

• Temporal decoupling — the consumer can be down when the producer sends, and still process the message later. The buffer absorbs the outage.
• Load leveling — a 10× traffic spike fills the buffer instead of melting the consumer; the consumer drains at its own steady rate (this is the basis of the P11 scale controller).
• Fan-out — one event, many independent reactions, added and removed without touching the producer.

But here is the principal-level truth that the postal metaphor hides: the moment you put a network and a buffer between producer and consumer, you inherit every hard problem in distributed systems. A message can be:

• lost (the producer's ack never came back; the consumer crashed after pulling it),
• duplicated (the producer retried a send that actually succeeded; the consumer processed then crashed before acknowledging, so it's redelivered),
• reordered (two messages take different network paths or land on different partitions),
• delayed (stuck behind a slow consumer, a poison message, or a long lock).

So "event-driven" is not "easier than synchronous." It is different — you trade synchronous failure (an error returned to the caller, now) for asynchronous delivery guarantees that you must reason about explicitly. The rest of this phase is exactly those guarantees. The headline: the default guarantee is at-least-once, which means duplicates are not a bug — they are the contract, and your consumer must be built for them.

Chapter 2: The Three Shapes — Broker vs Router vs Log

Azure gives you three event services, and 80% of event-architecture mistakes are choosing the wrong one. They are three shapes, not three brands:

The one-line decision rule, the thing to say out loud in an interview:

Is it a command I must not lose and maybe must order/transact? → Service Bus. Is it a notification I want many things to react to, push-style, at scale? → Event Grid. Is it a firehose of telemetry I need to stream, partition, and replay? → Event Hubs.

The trap: people reach for Event Hubs because "high throughput" sounds impressive, then need per-message dead-lettering and ordering and transactions — which is Service Bus's job. Or they build a polling loop on a Service Bus queue to do reactive fan-out — which is Event Grid's job. Match the shape to the workload, not the buzzword.

Chapter 3: Service Bus — The Broker and Its Entities

Service Bus is an enterprise message broker: a durable, ordered, transactional middleman. Its namespace contains two entity types:

• Queue — point-to-point. Producers send; competing consumers each pull and the broker hands each message to exactly one of them. This is work distribution / load leveling.
• Topic + Subscriptions — publish/subscribe. A producer sends to a topic; the broker copies the message into every subscription whose filter (a SQL/correlation expression on properties) matches. Each subscription is then its own queue with its own consumers. This is fan-out with routing.

Queue (point-to-point):           Topic + Subscriptions (pub/sub):

producer → [ q ] → consumer A                ┌─ sub:Orders   [filter type='order'] → consumers
                 → consumer B   producer →[topic]┤
              (one of A/B per msg)            └─ sub:Audit    [filter 1=1]          → consumers

A message carries a body (≤ 256 KB Standard, ≤ 100 MB Premium), system properties (MessageId, SessionId, ScheduledEnqueueTimeUtc, TimeToLive, DeliveryCount), and arbitrary application properties (used by subscription filters). The lab's Message models body, message_id, session_id, scheduled_for, plus the broker-owned delivery_count and dead_letter_reason.

Two receive modes sit at opposite ends of the delivery-semantics spectrum:

• ReceiveAndDelete — the broker removes the message the instant it's handed over. Fast, simple, at-most-once: if the consumer crashes before processing, the message is gone.
• PeekLock — the broker locks the message but keeps it; the consumer must explicitly settle it. This is at-least-once and the subject of the next chapter. It is what you use for anything that matters.

Chapter 4: Peek-Lock — The Delivery State Machine Under the Hood

This is the chapter the whole lab exists to make muscle memory. Peek-lock is the mechanism that makes a broker crash-safe. Here is the state machine:

                       receive(now)
                   ┌──────────────────────┐
        ┌──────────▼─────────┐            │ DeliveryCount++ each delivery
        │   READY (visible)  │            │
        └──────────┬─────────┘            │
                   │ receive: lock for    │
                   │ LockDuration         │
        ┌──────────▼─────────┐            │
        │  LOCKED (invisible │            │
        │  until lock_until) │            │
        └─┬────────┬───────┬─┘            │
 complete │ abandon│ dead- │ lock expires │
          │        │ letter│ (now>=until) │
   ┌──────▼──┐  ┌──▼───────┴──────────────┘
   │ REMOVED │  │  back to READY  ──── if DeliveryCount >= MaxDeliveryCount ──▶ DLQ
   └─────────┘  └─────────────────┘

Walk it slowly, because every transition is an interview question:

1. receive(now) pulls the next visible message and locks it: it becomes invisible to every other consumer until lock_until = now + LockDuration. The broker increments DeliveryCount (this delivery counts, whether or not it succeeds). The consumer gets the message and a lock token.
2. The consumer now has until lock_until to do one of three things:
   • complete(token) — "I processed it successfully." The broker removes the message. Done, gone, settled.
   • abandon(token) — "I can't process it right now; give it back." The lock releases immediately and the message is receivable again now (by this or another consumer). DeliveryCount already went up, so abandon-loops are bounded.
   • dead_letter(token, reason) — "this message is poison; don't retry it." It moves to the DLQ with the reason attached.
3. If the consumer does none of these before lock_until — it crashed, GC-paused, or the handler ran long — the lock expires. The broker reclaims the message, makes it receivable again, and on the next delivery DeliveryCount climbs again. This is the crash-safety guarantee: a dead consumer cannot silently swallow a message; it comes back automatically.

The off-by-one that interviewers love. When exactly is a lock expired? The lab's rule — and the right one — is now >= lock_until (inclusive). At the exact instant the lock duration has elapsed, the message is back. In the lab:

# lab/solution Broker.tick:
expired = [t for t, e in self._locked.items() if e.lock_until <= now]

If a message was received at t=0 with LockDuration=10, it is locked through t=9 and receivable again at exactly t=10 (test: test_lock_expiry_at_exact_boundary_is_expired).

Abandon vs lock-expiry are the same effect (redelivery, DeliveryCount++) reached two ways: abandon is the consumer voluntarily giving the message back now; lock-expiry is the broker involuntarily reclaiming it after the lock. The lab routes both through one function, _requeue_or_dead_letter, which is exactly how to think about it.

In real code the receiver also calls RenewMessageLockAsync to extend a lock for a long-running handler (so a slow-but-alive consumer isn't treated as crashed). The lab deliberately omits renewal and models only its consequence — if you don't renew and you're slow, you get redelivered, which is the whole point.

Chapter 5: Dead-Lettering and the Poison-Message Guard

A poison message is one that can never be processed: a malformed payload, a reference to a deleted resource, a bug that throws every time. Without a guard, such a message is abandoned, redelivered, abandoned, redelivered — forever — burning the consumer and blocking the queue behind it (head-of-line blocking). The guard is MaxDeliveryCount.

The rule (and the lab's contract): a message is delivered at most MaxDeliveryCount times. Once it has been delivered that many times, the next disposition that would redeliver it instead routes it to the dead-letter queue (DLQ):

# solution Broker._requeue_or_dead_letter:
if msg.delivery_count >= self.max_delivery_count:
    msg.dead_letter_reason = reason
    self._dlq.append(msg)      # poison → DLQ, NOT redelivered
    return
self._ready.insert(0, msg)     # otherwise back in line (FIFO preserved)

With MaxDeliveryCount=2: deliver (count 1) → abandon → deliver (count 2) → abandon → DLQ at count 2. The message was delivered exactly twice, never a third time (test_max_delivery_count_moves_to_dlq_via_abandon). Service Bus's default is 10.

A message also dead-letters on TTL expiry (it sat unprocessed past TimeToLive) and on explicit dead_letter(). The DLQ is a real sub-entity (<queue>/$DeadLetterQueue) you receive from like any queue — to triage, fix, and replay.

The principal framing: the DLQ is a bulkhead, not a graveyard. Its job is to get the poison message off the hot path (so healthy traffic flows) and raise an alarm (so a human triages it). A DLQ filling up is a signal, and "active vs dead-letter message count" is one of the first alerts you wire in P13. The worst failure mode is a DLQ nobody watches.

Chapter 6: Duplicate Detection, Scheduled & Deferred, Transactions

Duplicate detection. Because delivery is at-least-once, producers also retry — and a producer that retries a send which actually succeeded creates a duplicate at the source. Service Bus can drop these: with RequiresDuplicateDetection, the broker remembers every MessageId for a DuplicateDetectionHistoryTimeWindow (up to 7 days) and silently drops a repeat within that window. The lab models it precisely:

# solution Broker.send:
if self.dedup_window is not None and msg.message_id is not None:
    self._expire_dedup(now)
    if msg.message_id in self._seen:
        return False                 # dropped (same outcome as the original)
    self._seen[msg.message_id] = now

A repeat MessageId at t=5 with a 60s window is dropped; the same id at t=70 (window elapsed) is accepted again (test_dedup_drops_repeat_id_in_window, test_dedup_allows_after_window). This is the producer-side half of exactly-once-effect — the consumer-side half is idempotency (Chapter 8).

Scheduled messages. A message with ScheduledEnqueueTimeUtc in the future is enqueued but not receivable until that time — delayed jobs, "send this reminder in 24h", retry backoff implemented as a reschedule. The lab's scheduled_for does exactly this: receive skips a message whose scheduled_for > now (test_scheduled_message_not_receivable_early), and crucially does not let a future-scheduled message block a ready one behind it (test_scheduled_does_not_block_ready_messages).

Deferred messages (described, not implemented). A consumer can Defer a message it's not ready to handle in order — the broker sets it aside and the consumer fetches it later by sequence number. Used for out-of-order workflow steps ("I got step 3 before step 2; defer 3"). This is an extension in the lab README.

Transactions (framing). Service Bus supports atomic operations within a transaction scope: e.g. complete an input message and send an output message atomically — both happen or neither does. This is how you build "process-and-forward" without a window where the input is consumed but the output never sent. It does not make end-to-end exactly-once across different systems — that still needs idempotency at each boundary.

Chapter 7: Sessions — Ordering Has a Price

By default a queue's competing consumers process messages in parallel and out of order — which is great for throughput and wrong when order matters (apply these bank transactions in sequence; process this customer's events as they happened). Sessions are how Service Bus gives you FIFO:

• Every message carries a SessionId (the partition/ordering key — a customer id, an order id, a device id).
• A session-enabled queue hands an entire session to one consumer at a time: the consumer accepts a session, gets its messages in order, and no other consumer can touch that session until the first releases it.
• Different sessions are processed in parallel by different consumers — so you get FIFO within a key and parallelism across keys.

session A: a1 → a2 → a3   ─locked to─▶ consumer 1   (strict order within A)
session B: b1 → b2        ─locked to─▶ consumer 2   (strict order within B)
                                       (A and B run in parallel)

The lab models this with sessions_enabled=True: a receive that locks session A makes all of A's messages unavailable to other consumers; the next receive skips to a different session rather than handing out A's later messages (test_session_single_active_consumer), and within a session strict FIFO holds (test_session_fifo_within_session).

The price — and the principal point — is in the parallelism: your maximum useful consumer count is bounded by your number of active sessions, and one slow session is a head-of-line block for itself. Ordering is a real throughput tax. Don't pay it for a stream that doesn't need order (most analytics/notification streams don't; most financial/stateful ones do). The interview signal is asking "do we actually need order here, and at what granularity?" before reaching for sessions.

Chapter 8: Delivery Semantics — The Precise Lie of "Exactly-Once"

This is the most important chapter for an interview, because it separates people who say "exactly-once" from people who understand delivery. There are three guarantees:

The crucial truth: true exactly-once delivery is impossible across a network boundary. The reason is the two-generals problem — after the consumer processes a message it must acknowledge; if that ack is lost, the broker must redeliver (it can't tell "processed but ack lost" from "never processed"), so a duplicate is unavoidable. Anyone who promises exactly-once delivery either has a closed system (Kafka transactions within Kafka) or doesn't understand the problem.

What you can build is exactly-once-effect: accept that delivery is at-least-once, and make the consumer idempotent so processing the same message twice has the same effect as processing it once. The pattern:

# idempotent consumer (the consumer-side half of exactly-once-effect)
def handle(msg):
    key = msg.message_id                       # the idempotency key the producer stamped
    if already_processed(key):                 # a dedup store: Redis/Cosmos/a unique index
        ack()                                  # duplicate → no-op, just ack
        return
    with transaction():
        do_the_side_effect(msg)
        mark_processed(key)                    # atomic with the effect
    ack()

classify_delivery(retries, idempotent_consumer) in the lab is exactly this truth table:

def classify_delivery(retries, idempotent_consumer):
    if not retries:            return "at-most-once"
    if not idempotent_consumer: return "at-least-once"
    return "exactly-once-effect"

And the principal framing: exactly-once-effect is a per-data-product dial, not a platform default. It costs throughput (the dedup check) and operability (the dedup store). You pay it where a duplicate is expensive — a duplicate payment is a refund and a furious customer; a duplicate ad impression is a rounding error. Ask "what does one duplicate cost, and what does one loss cost?" per stream before you reach for it.

Chapter 9: Event Grid — Reactive Routing, CloudEvents, and Retry

Event Grid is the push sibling of Service Bus's pull. It is a fully-managed event router: a publisher emits a small event (a notification, not a payload — "blob X was created", "resource Y changed"), and Event Grid pushes it to every subscriber whose filter matches, over HTTP, with no consumer infrastructure to run. It is built for massive fan-out at low operational cost ("react to X").

CloudEvents. Event Grid speaks CloudEvents 1.0, the CNCF standard envelope, so events are portable across clouds and tooling. The required and common attributes:

{
  "specversion": "1.0",
  "id": "a1b2c3",                                  // unique; the idempotency key
  "source": "/subscriptions/.../storageAccounts/x",// who emitted it
  "type": "Microsoft.Storage.BlobCreated",         // what happened (filter on this)
  "subject": "/blobServices/default/containers/in/img.png", // the resource (filter on this)
  "time": "2026-06-25T12:00:00Z",
  "data": { "url": "https://x.blob.core.windows.net/in/img.png" }
}

Subscribers filter by type and subject (prefix/suffix), so a thumbnailer subscribes only to BlobCreated under containers/images/ — the routing is declarative.

Delivery and retry — the heart of the lab's deliver_with_retry. Event Grid is at-least-once: it pushes the event and waits for a 2xx from the subscriber's handler. On a failure (a non-2xx, a timeout, or no response) it retries with exponential backoff — a fixed, growing schedule of intervals (seconds → minutes), continuing until the event is accepted or its TTL (up to ~24h) / max attempts are exhausted, at which point it dead-letters to a configured blob container. The lab models this exactly and deterministically:

# solution deliver_with_retry — first attempt at `now`, then back off per `schedule`
for attempt in range(max_retries + 1):
    interval = schedule[attempt] if attempt < len(schedule) else schedule[-1]
    clock += interval
    record.attempts.append(clock)
    try:
        ok = handler(event)
    except Exception:
        ok = False              # a raised handler is a failed delivery, like a 500
    if ok:
        record.delivered = True; return record
record.dead_lettered = True     # retries exhausted → dead-letter to storage
return record

A handler that fails on attempts 1 and 2 then succeeds on attempt 3 is delivered at times [0, 10, 40] (with schedule (0,10,30,…)); a handler that always fails is dead-lettered after max_retries+1 attempts (test_event_grid_retries_on_fixed_schedule_then_succeeds, test_event_grid_dead_letters_after_max_retries). The whole thing is deterministic — now is injected and advanced by the schedule, never read from a clock (test_event_grid_is_deterministic), which is the only way to test a retry policy.

The consumer is a webhook — a JWT-validated, idempotent HTTP handler exactly like the Phase 09 gateway. Because delivery is at-least-once, that handler must be idempotent on the CloudEvent id (Chapter 8) — Event Grid will occasionally deliver the same event twice.

Chapter 10: Event Hubs — The Streaming Sibling (for Contrast)

You will be asked "Event Hubs or Service Bus?" so hold the contrast even though the full streaming build is in another track. Event Hubs is a partitioned, append-only log — the Azure-native, Kafka-shaped firehose:

• Events are appended to partitions; order is guaranteed within a partition (keyed by a partition key), not across.
• Consumers don't settle messages; they track an offset (a position in the log) and can rewind to replay history within the retention window (1–90 days). There is no per-message lock, ack, or DLQ — the unit is the stream position, not the message.
• Throughput is sized in throughput units / processing units / partitions, scaling to millions of events/second — orders of magnitude past a broker.

So: Event Hubs is for high-volume telemetry/streaming you process in aggregate and may replay (IoT, clickstream, logs, metrics → analytics). Service Bus is for discrete commands you must not lose and may need to order, dead-letter, or transact (orders, payments, workflow steps). The tell: if you find yourself wanting per-message dead-lettering or sessions on Event Hubs, you wanted Service Bus; if you're polling a Service Bus queue at a million msg/s, you wanted Event Hubs.

Chapter 11: Logic Apps — Serverless Workflow Orchestration

The first three services move events. Logic Apps react to and orchestrate them without you writing a service. A Logic App is a serverless workflow: a trigger (an event arrives, a timer fires, an HTTP request comes in — including a Service Bus message or an Event Grid event) followed by a graph of actions drawn from hundreds of connectors (Service Bus, Office 365, Salesforce, SQL, a custom HTTP call, a condition, a loop). It is the low-code integration complement to code-first Functions (P11):

trigger: "Service Bus message in queue 'orders'"
  → action: parse JSON
  → condition: amount > 10000 ?
      → yes → action: call fraud API → action: post to Teams
      → no  → action: write to SQL → action: send confirmation email
  → (built-in: retry policy, run history, per-action error handling)

Why it exists: an enormous amount of "enterprise integration" is glue — when an event happens, call these five SaaS systems in order, with retries and conditionals. Writing and operating a service for that is waste; a Logic App is a declarative workflow with built-in retries, a visual run history (every run, every action, the data that flowed — a debugging gift), and managed connectors. It is the orchestration layer that consumes the events the broker and router deliver. The principal judgment: use a Logic App for integration glue and human-in-the-loop workflows; use a Durable Function (P11) when the orchestration is code with complex state, loops, and determinism requirements. (Logic Apps and Durable Functions are the two "workflow" answers; knowing which to reach for is the signal.)

Lab Walkthrough Guidance

Lab 01 — Messaging Broker, suggested order (each step has tests that go green in turn):

1. Message.__init__ — validate message_id (non-empty string or None), session_id (string or None), scheduled_for (number or None); init delivery_count=0, dead_letter_reason=None. (Ch. 3)
2. Broker.__init__ + send — validate construction; enforce the session_id rule (required iff sessions_enabled); implement duplicate detection (drop a repeat message_id inside dedup_window) and _expire_dedup. (Ch. 6)
3. receive + _next_receivable_index — peek-lock: call tick, find the next receivable message (skip future-scheduled_for, skip already-locked sessions), pop it, delivery_count++, mint a lock token, record the _LockEntry, lock the session. (Ch. 4, 7)
4. complete / abandon / dead_letter + _take_lock / _release_session / _requeue_or_dead_letter — settle (remove), release-now (requeue or DLQ), explicit DLQ. Get the >= max_delivery_count boundary right. (Ch. 4, 5)
5. tick — reclaim every lock with lock_until <= now (the off-by-one), in stable order, routing each through _requeue_or_dead_letter. (Ch. 4)
6. deliver_with_retry — first attempt at now, back off per schedule (clamp at the last interval), treat a raised handler as failure, dead-letter when retries exhaust. (Ch. 9)
7. classify_delivery — the three-way semantics map. (Ch. 8)

Run pytest test_lab.py -v after each step and watch the relevant tests flip green. Then python solution.py to see all six mechanisms in one worked example.

Success Criteria

You are ready for Phase 11 when you can, from memory:

1. Name the three event-plane shapes (broker/router/log = Service Bus/Event Grid/Event Hubs) and the one decision that picks between them.
2. Draw the peek-lock state machine: receive→lock→complete/abandon/dead-letter; lock-expiry→ redeliver + DeliveryCount++; MaxDeliveryCount → DLQ.
3. Explain the lock-expiry off-by-one (now >= lock_until is expired) and why abandon and lock-expiry reach the same redelivery effect two ways.
4. State the three delivery semantics, prove true exactly-once delivery is impossible across a network, and describe the idempotent-consumer pattern that gives exactly-once-effect.
5. Explain duplicate detection (producer side) vs idempotency (consumer side) as the two halves of exactly-once-effect.
6. Say what sessions cost (throughput / bounded parallelism) and when ordering is required.
7. Sketch an Event Grid CloudEvent, its type/subject filter, its backoff-retry policy, and its dead-letter-to-storage target.
8. Choose between a Logic App and a Durable Function for a given orchestration.

Interview Q&A

Q: A consumer pulls a Service Bus message and then the process crashes. Walk me through exactly what happens to that message. In PeekLock mode the message was locked, not removed — it's invisible to other consumers until lock_until = receiveTime + LockDuration, and DeliveryCount was already incremented. Because the crashed consumer never called complete, the lock simply expires; the broker then makes the message visible again and redelivers it (to this or another consumer), incrementing DeliveryCount again. So nothing is lost — the crash-safety is automatic. The one nuance: a slow but alive consumer must renew the lock (RenewMessageLockAsync) or it'll be treated as crashed and get a duplicate; and a message that keeps failing climbs DeliveryCount until it hits MaxDeliveryCount and dead-letters, which is what stops a poison message from looping forever.

Q: Is your event pipeline exactly-once? Yes or no, and prove it. No — and nothing across a network is. Delivery is at-least-once: after my consumer processes a message it must ack, and if that ack is lost the broker can't distinguish "processed, ack lost" from "never processed," so it must redeliver — a duplicate is unavoidable (the two-generals problem). What I build is exactly-once-effect: at-least-once delivery plus an idempotent consumer keyed on the producer's MessageId/CloudEvent id, so a duplicate is a no-op. I also turn on duplicate detection at the broker for producer-side retries. I apply that per data product, sized by what a duplicate costs — a duplicate payment gets it, a duplicate analytics impression doesn't.

Q: Service Bus, Event Grid, or Event Hubs — and how do you decide? By the shape of the workload. A discrete command I must not lose and might need to order or transact → Service Bus (broker: pull, lock, settle, sessions, DLQ). A notification I want many things to react to, push-style, with no consumer to operate → Event Grid (router: CloudEvents, at-least-once push, retry, dead-letter to storage). A firehose of telemetry I process in aggregate and may replay → Event Hubs (partitioned log, offsets, millions/s). The mistake I watch for is reaching for Event Hubs because "high throughput" sounds good, then needing per-message dead-lettering and ordering — that was Service Bus all along.

Q: What does MaxDeliveryCount actually protect against, and what's a sane value? Poison messages and head-of-line blocking. A message that can never be processed — malformed payload, bug, deleted dependency — would otherwise be abandoned and redelivered forever, burning the consumer and blocking the queue behind it. MaxDeliveryCount caps deliveries; after that many attempts the message dead-letters (off the hot path) and an alert fires (a human triages). The default 10 is reasonable for transient failures; I'd lower it if my failures are usually deterministic (no point retrying a parse error 10 times) and pair it with a DLQ-depth alert and a documented replay runbook. The DLQ is a bulkhead, not a graveyard — the failure mode is a DLQ nobody watches.

Q: Why does ordering cost throughput? Because order requires a single writer per ordering key. A session locks an entire SessionId to one consumer, in sequence — so your useful parallelism is bounded by the number of active sessions, not the number of consumers, and one slow session is a head-of-line block for itself. Without sessions, competing consumers drain the queue in parallel and out of order at full throughput. So I only enable sessions where order is genuinely required, and I choose the finest ordering granularity that's still correct (per-customer, not global) to keep parallelism high.

Q: Walk me through Event Grid's retry behavior and where a permanently-failing event ends up. Event Grid is at-least-once push. It POSTs the CloudEvent to the subscriber's webhook and waits for a 2xx. On a non-2xx, timeout, or no response, it retries on an exponential backoff schedule — growing intervals from seconds to minutes — and keeps retrying until the event is accepted or its TTL (up to ~24h) / max attempts are exhausted. At that point it dead-letters the event to a configured blob container, where I can inspect and replay it. Because it's at-least-once, the webhook has to be idempotent on the CloudEvent id. In the lab, deliver_with_retry models exactly this deterministically — first attempt at now, back off per a fixed schedule, treat a thrown handler like a 500, dead-letter when retries run out.

Q: When do you use a Logic App vs a Durable Function? Both orchestrate multi-step workflows. I reach for a Logic App for integration glue — "when this event arrives, call these SaaS connectors in order with retries and a condition" — because it's declarative, has hundreds of managed connectors, and gives me a visual run history that's a debugging gift, with no code to operate. I reach for a Durable Function when the orchestration is code: complex state, dynamic fan-out/fan-in, loops, and tight determinism requirements (P11). Rule of thumb: connectors and human-in-the-loop → Logic App; algorithmic, stateful, code-first orchestration → Durable Function.

References

• Azure Service Bus — Service Bus messaging overview, queues/topics/subscriptions, message transfers, locks, and settlement (peek-lock), dead-letter queues, duplicate detection, message sessions, scheduled and deferred messages: https://learn.microsoft.com/azure/service-bus-messaging/
• Azure Event Grid — Event Grid overview, event delivery and retry, dead-letter and retry policies, event filtering, CloudEvents schema: https://learn.microsoft.com/azure/event-grid/
• CloudEvents 1.0 specification (CNCF) — the event envelope Event Grid speaks: https://github.com/cloudevents/spec and https://cloudevents.io/
• Azure Event Hubs — Event Hubs overview, partitions, consumers, and offsets (for the Service Bus vs Event Hubs contrast): https://learn.microsoft.com/azure/event-hubs/
• Azure Logic Apps — Logic Apps overview, triggers, actions, and connectors, Standard vs Consumption: https://learn.microsoft.com/azure/logic-apps/
• Azure Functions — Service Bus and Event Grid triggers/bindings (the consumer side, P11): https://learn.microsoft.com/azure/azure-functions/
• Gregor Hohpe & Bobby Woolf, Enterprise Integration Patterns — the canonical messaging patterns (Competing Consumers, Dead Letter Channel, Idempotent Receiver, Message Sessions).
• Martin Kleppmann, Designing Data-Intensive Applications, Ch. 11 (Stream Processing) — the delivery-semantics and exactly-once-effect treatment from first principles.
• The track's own CHEATSHEET.md (the messaging numbers) and GLOSSARY.md (every term, defined).