🛸 Hitchhiker's Guide — Phase 10: Event-Driven Messaging

Read this if: you can click "Create Service Bus queue" but "design our event-driven platform" and "is this exactly-once?" still make you sweat. This is the compressed tour the WARMUP derives slowly. Skim it, then read the WARMUP, then build the broker.

0. The 30-second mental model

Three event services, three shapes, one decision:

Service Bus  = BROKER  → pull, LOCK, settle. Commands you must not lose, order, transact, DLQ.
Event Grid   = ROUTER  → push, retry, dead-letter. "React to X" fan-out at scale, no consumer to run.
Event Hubs   = LOG     → stream, offsets, replay. Telemetry firehose (millions/s), process in aggregate.

One sentence to tattoo: decoupling is a buffer plus a delivery guarantee, and the guarantee is at-least-once — so duplicates aren't a bug, they're the contract, and your consumer must be built for them.

1. The peek-lock state machine (draw this in the interview)

receive(now) → LOCKED (invisible until now+LockDuration), DeliveryCount++
  ├─ complete       → removed (success)
  ├─ abandon        → back to READY *now* (DeliveryCount already ++)
  ├─ dead_letter    → DLQ with a reason
  └─ lock expires   → back to READY (consumer crashed/too slow), redeliver, DeliveryCount++
                      └─ once delivered MaxDeliveryCount times → DLQ instead of redeliver

The off-by-one: a lock is expired at now >= lock_until (inclusive). Received at t=0 with LockDuration=10 → locked through t=9, back at exactly t=10.

2. The numbers to tattoo on your arm

3. Delivery semantics (the whole interview, compressed)

no retries                       → at-most-once     (may LOSE)        e.g. ReceiveAndDelete
retries, dumb consumer           → at-least-once    (may DUPLICATE)   the practical default
retries + idempotent consumer    → exactly-once-EFFECT (dup = no-op)  the only "exactly-once" there is

Two halves of exactly-once-effect: producer stamps MessageId/dedup window; consumer is idempotent on that id (dedup store + atomic mark-processed). Apply it per data product, sized by what one dup vs one loss costs.

4. The decision tree (say this out loud)

Must-not-lose command? need order/transactions/DLQ? ───────────────▶ Service Bus
"Something happened, many things react", push, huge fan-out? ──────▶ Event Grid
Telemetry firehose, replay, partitioned, millions/s? ─────────────▶ Event Hubs
Glue together SaaS systems / human-in-the-loop workflow? ─────────▶ Logic App
Code-first stateful orchestration with determinism? ──────────────▶ Durable Function (P11)

5. az one-liners (your subscription, after the lab)

# Service Bus: namespace → queue with sessions, dedup, DLQ-on-expiry
az servicebus namespace create -g rg -n myns --sku Premium
az servicebus queue create -g rg --namespace-name myns -n orders \
  --max-delivery-count 5 --lock-duration PT1M \
  --enable-session true --requires-duplicate-detection true \
  --default-message-time-to-live PT14D --dead-lettering-on-message-expiration true
az servicebus queue show -g rg --namespace-name myns -n orders \
  --query "countDetails"          # activeMessageCount, deadLetterMessageCount ← your alert

# Topic + filtered subscription (pub/sub)
az servicebus topic create -g rg --namespace-name myns -n events
az servicebus topic subscription create -g rg --namespace-name myns \
  --topic-name events -n high-value
az servicebus topic subscription rule create -g rg --namespace-name myns \
  --topic-name events --subscription-name high-value -n amt \
  --filter-sql-expression "amount > 10000"

# Event Grid: topic + a webhook subscription with filters, retry policy, and dead-letter
az eventgrid topic create -g rg -n mytopic -l eastus
az eventgrid event-subscription create -n thumbnailer \
  --source-resource-id $(az eventgrid topic show -g rg -n mytopic --query id -o tsv) \
  --endpoint https://api.example.com/hooks/thumb \
  --included-event-types Microsoft.Storage.BlobCreated \
  --subject-begins-with /blobServices/default/containers/images/ \
  --max-delivery-attempts 5 --event-ttl 1440 \
  --deadletter-endpoint /subscriptions/.../containers/eg-deadletter

Terraform shape: azurerm_servicebus_namespace / _queue (max_delivery_count, lock_duration, requires_session, requires_duplicate_detection) and azurerm_eventgrid_event_subscription (retry_policy { max_delivery_attempts, event_time_to_live }, storage_blob_dead_letter_destination).

6. War story shapes you'll relive

• "We charged the customer twice." → at-least-once redelivered the payment message and the consumer wasn't idempotent. Fix is a dedup key on MessageId + atomic mark-processed, not "turn on exactly-once" (there's no such switch).
• "The queue is stuck and nothing processes." → a poison message at the head, abandoned and redelivered forever, blocking everything behind it. Fix: MaxDeliveryCount → DLQ + an alert on dead-letter count. (Head-of-line blocking is especially nasty with sessions.)
• "Messages get reprocessed every ~30s for no reason." → handler runs longer than LockDuration, the lock expires mid-process, the broker redelivers. Fix: renew the lock or raise LockDuration (max 5 min) — and make the handler idempotent so the dup is harmless.
• "We can't keep up; throughput tanked when we added ordering." → sessions cap parallelism at the active-session count. Fix: finer ordering key (per-customer not global) or drop ordering where it isn't required.
• "Event Grid stopped delivering." → the webhook returned 500s, Event Grid backed off and eventually dead-lettered to the blob you forgot to watch. The events are in the dead-letter container, not lost — go drain it.
• "We picked Event Hubs and now we need per-message DLQ and sessions." → you wanted Service Bus. Event Hubs has offsets, not locks; there's no per-message settle or dead-letter.

7. Vocabulary that signals you've held the pager

• Peek-lock / settle — receive-lock, then complete/abandon/dead-letter. The crash-safety mechanism.
• DeliveryCount / MaxDeliveryCount — the redelivery counter and the poison-message cap.
• DLQ (dead-letter queue) — a bulkhead for unprocessable messages, not a graveyard.
• At-least-once / exactly-once-effect — the real guarantee, and the only "exactly-once."
• Idempotency key — the producer's MessageId/CloudEvent id the consumer dedups on.
• Session / SessionId — FIFO per key, single active consumer; ordering's throughput tax.
• Duplicate detection — the producer-side dedup window on MessageId.
• CloudEvents — the CNCF envelope (id/source/type/subject/time/data) Event Grid speaks.
• Head-of-line blocking — one stuck message blocking everything behind it.
• Competing consumers — many consumers pulling one queue for parallel work distribution.

8. Beginner mistakes that mark you in interviews

1. Saying "exactly-once" without "-effect" — instant tell you haven't built a real consumer.
2. Reaching for Event Hubs because "high throughput" sounds good, then needing DLQ/sessions.
3. Forgetting the consumer must be idempotent because delivery is at-least-once.
4. Using ReceiveAndDelete (at-most-once) for anything that matters, then losing messages.
5. No DLQ alert — the poison messages pile up silently until someone notices the gap.
6. Enabling sessions "to be safe" and tanking throughput for a stream that didn't need order.
7. A handler slower than LockDuration with no lock renewal → mysterious constant redelivery.
8. Treating Event Grid delivery as guaranteed-once instead of at-least-once-with-retry.

9. How this phase pays off later

• P11 (Serverless) — a Service Bus / Event Grid trigger invokes a Function; the scale controller sizes instances from the queue backlog you produce here; Durable Functions are the code-first sibling of Logic Apps.
• P12 (Key Vault / MI) — the secret-free Managed Identity a consumer authenticates with; data-plane RBAC (Service Bus Data Receiver, EventGrid Data Sender) gates it.
• P13 (Observability) — DeliveryCount, active/dead-letter counts, and retry rates are the alerts that catch a stuck consumer before customers do.
• P14 (Reliability) — retry+backoff+jitter, the DLQ-as-bulkhead, and at-least-once+idempotency are the reliability primitives, applied to messaging.
• P15 (Capstone) — the event-driven serverless platform is this broker + Event Grid + Functions, wired under a landing zone.

Now read the WARMUP slowly, then build the broker. The peek-lock state machine you implement is the one you'll draw on a whiteboard for the rest of your career.