Lab 12 — Observability & Load Test

Load-test harness + metrics math run offline (against a mock); point at a real endpoint for production numbers. Concepts: Phase 7.08, Phase 10.08, cheatsheet 17.

Goal

Measure latency percentiles under load (TTFT/TPOT/p95/p99), attribute cost, and trace requests with correlation IDs — the operate-in-production skill.

Run (offline, zero deps)

python3 load_test.py   # p50/p95/p99 + throughput vs concurrency (1/4/16/64) against a mock call with a heavy tail

load_test.py runs dependency-free (stdlib asyncio) so you can verify the percentile/throughput math; it shows p99 ≫ avg (why you report tails, not averages). Point mock_call at a real async HTTP call to benchmark a live endpoint.

Suggested files

lab-12-observability/
  README.md
  load_test.py     # concurrency driver → p50/p95/p99, throughput
  trace.py         # correlation-ID propagation across retrieval→model→tools
  cost_meter.py    # per-request token+cost accounting (reuse lab-11 math)

Key snippet (percentiles)

import asyncio, time, statistics

async def one(call):
    t0 = time.perf_counter()
    await call()
    return time.perf_counter() - t0

async def load_test(call, concurrency, n):
    sem = asyncio.Semaphore(concurrency)
    async def worker():
        async with sem:
            return await one(call)
    lat = await asyncio.gather(*[worker() for _ in range(n)])
    lat.sort()
    p = lambda q: lat[min(len(lat)-1, int(q*len(lat)))]
    print(f"c={concurrency} n={n} p50={p(.5):.3f} p95={p(.95):.3f} p99={p(.99):.3f} "
          f"thrpt={n/sum(lat)*concurrency:.1f}/s")

Steps

  1. Build the async load driver; run against a mock call (offline) to verify percentile math, then a real endpoint.
  2. Sweep concurrency (1/4/16/64); record p50/p95/p99 + throughput; find the knee.
  3. Add correlation IDs propagated through a (mock) retrieval→model→tool chain; reconstruct one request end-to-end.
  4. Add a cost meter (tokens × price) and a per-tenant breakdown.
  5. Add an alert rule (e.g., p95 > target → print/alert).

Deliverables

  • p50/p95/p99 + throughput vs concurrency.
  • An end-to-end reconstructed trace via correlation ID.
  • A cost-attribution table (per tenant/feature).
  • A note: why p95 under load, not average, is the SLA metric.

Why it matters (interview)

Operating LLM systems = percentiles under load + cost attribution + traceability (interview-prep 02–03). Feeds the runbook template.