Lab 01 — Messaging Broker (Service Bus peek-lock + Event Grid retry)

Phase: 10 — Event-Driven: Event Grid, Service Bus & Logic Apps Difficulty: ⭐⭐⭐☆☆ | Time: 4–6 hours

Every "we lost a message" and every "we processed it twice" incident is one of two mechanisms misunderstood: the peek-lock state machine (lock → process → complete/abandon/dead-letter, with lock-expiry redelivery and a max-delivery poison guard) or at-least-once push delivery with backoff retry (Event Grid). This lab builds both engines — deterministically, offline, with time injected — so the redelivery off-by-one, the FIFO-session lock, the dedup window, and the dead-letter transition stop being folklore and become code you can defend in an interview and reason about at 2 a.m.

What you build

• Message(body, message_id, session_id, scheduled_for) — the unit the broker locks, counts, and dead-letters; carries mutable delivery_count and dead_letter_reason.
• Broker — a single Service Bus queue (or one topic-subscription) with the full peek-lock machine:
  • send(msg, now) — honors duplicate detection (a repeat message_id inside dedup_window is silently dropped) and scheduled messages (scheduled_for in the future is not receivable yet).
  • receive(now) -> (msg, lock_token) | None — peek-lock: the message goes invisible until now + lock_duration; FIFO, and FIFO-per-session with a single active consumer when sessions are enabled.
  • complete / abandon / dead_letter — settle (remove), release the lock now (redeliverable, delivery_count already climbed), or move to the DLQ with a reason.
  • tick(now) — when a lock expires (now >= lock_until), redeliver and increment delivery_count; once a message has been delivered max_delivery_count times it goes to the dead-letter queue instead of being redelivered.
  • dlq() — the dead-letter queue accessor.
• deliver_with_retry(handler, event, *, schedule, max_retries, now) -> Delivery — the Event Grid push path: call handler(event), and on failure (falsy return or a raised exception) retry after fixed exponential-backoff intervals; after max_retries are exhausted, dead-letter. Returns the timestamps of every attempt and the outcome.
• classify_delivery(retries, idempotent_consumer) — at-most-once / at-least-once / exactly-once-effect.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All tests pass against your lab.py and against solution.py.
• You can explain why test_lock_expiry_at_exact_boundary_is_expired treats now == lock_until as expired (the lock has elapsed, so the message is back).
• You can explain why, with max_delivery_count=2, the poison message in test_max_delivery_count_moves_to_dlq_via_abandon reaches the DLQ at delivery_count == 2 — "delivered at most twice," not "twice then one more."
• You can explain why test_session_single_active_consumer skips to session B instead of draining the rest of session A (one active consumer per session = the FIFO guarantee).
• You can explain why deliver_with_retry treats a raised exception identically to a False return, and why a duplicate at-least-once delivery is harmless iff the consumer is idempotent (classify_delivery).

How this maps to real Azure

Limits worth memorizing (Service Bus Standard/Premium): LockDuration ≤ 5 min; MaxDeliveryCount default 10; dedup window up to 7 days; message TTL default 14 days; Standard message size 256 KB, Premium up to 100 MB; sessions and transactions are Premium-friendly features. Event Grid: at-least-once, retry with backoff up to 24 h, dead-letter to a blob container. (The lab models the mechanism; the exact numbers move with tier and are in the phase HITCHHIKERS-GUIDE.md.)

What the lab deliberately simplifies

• No lock renewal — real receivers call RenewMessageLockAsync to extend a lock for a long-running handler. The lab models the consequence of not renewing (expiry → redeliver).
• No deferral / transactions — Defer/ReceiveDeferredMessage and atomic send-and-complete transactions are described in WARMUP.md but not implemented (an extension below).
• Logical time only — Azure uses UTC wall-clock with lock-duration arithmetic; the lab injects integer now so every test is deterministic. The arithmetic is identical.

Extensions

• Lock renewal: add renew_lock(token, now, extend_by) that pushes lock_until forward (and rejects renewal of an already-expired lock) — model a long-running handler.
• Deferred messages: add defer(token) + receive_deferred(sequence_number, now) so a consumer can set a message aside and fetch it by number later (out-of-order processing).
• Topics & subscriptions with filters: wrap N Broker subscriptions behind a Topic that fans a published message to every subscription whose SQL/correlation filter matches — the Event Grid eventType/subject filter, but pull-based.
• Transactions: a transaction() context that makes complete(in) + send(out) atomic (the classic "process and forward exactly once within the broker" pattern).
• Real Azure (your subscription): provision a namespace, queue, and Event Grid topic with az servicebus / az eventgrid (one-liners in the phase guide), publish a CloudEvent, and watch DeliveryCount, the DLQ, and the dead-letter blob fill up — then map each back to a function in this file.

Interview / resume bullets

• "Implemented a Service Bus-style broker with peek-lock receive, lock-expiry redelivery and DeliveryCount tracking, duplicate detection, scheduled messages, FIFO sessions with a single active consumer, and a max-delivery dead-letter queue."
• "Built an Event Grid-style at-least-once push-delivery engine with deterministic exponential-backoff retry and dead-lettering, and a delivery-semantics classifier (at-most / at-least / exactly-once-effect)."