« Track Overview

Cheat Sheet — the numbers, formulas and one-liners

The recall pass. Everything here is either arithmetic you should be able to do on a whiteboard or a sentence you should be able to say without hedging. Vendor-specific figures (PTU sizes, per-token prices, quota defaults) change — the method is what is durable, so this sheet gives you the method and tells you which number to look up.

Table of Contents

• The five sentences
• Availability arithmetic
• Error budgets & burn rate
• Reliability of an agent loop
• Latency budgets
• Rate limiting: the token bucket
• Capacity: PTU vs pay-as-you-go
• Cost model for an agent action
• Caching arithmetic
• Serving: prefill, decode, KV cache
• Retrieval: BM25, RRF, recall
• Identity: the claims that matter
• Policy evaluation
• Idempotency, retries and sagas
• Circuit breaker settings
• OWASP LLM Top 10 → control map
• The protocol one-liners
• Review red flags

The five sentences

1. The model proposes, the platform disposes. Every effect passes identity, policy, contract, quota and evidence before it becomes a bank action.
2. The tenant comes from the token, never the request. Anything the caller can set, the caller can forge.
3. Retrieval must be authorized, not merely relevant. A shared index returns the nearest chunk regardless of who owns it.
4. Agents get the task's privileges, not the user's. Scope down at every hop; record the chain.
5. If you cannot produce the evidence, you did not build the control. Auditability is a design input, not a logging afterthought.

Availability arithmetic

Series dependencies multiply; redundant ones combine through their failure probabilities.

$$A_{\text{series}} = \prod_i A_i \qquad A_{\text{parallel}} = 1 - \prod_i (1 - A_i)$$

The five-layer trap: five independent layers at 99.9% each give \( 0.999^5 = 0.995 \) — 99.5%, i.e. 3 h 36 m/month. You cannot promise a platform SLO higher than the product of its serial dependencies. Either reduce serial depth, add redundancy at the weak layer, or degrade gracefully so a layer failure is not a request failure.

Two providers at 99.9% each, independent, with working fallback: \( 1 - 0.001^2 = 0.999999 \) — but only if failover is fast enough to fit the latency budget and the failure modes really are independent (same region ≠ independent).

Error budgets & burn rate

$$\text{error budget} = 1 - \text{SLO} \qquad \text{burn rate} = \frac{\text{observed error rate}}{1-\text{SLO}}$$

Burn rate 1 = you exhaust the budget exactly at the end of the window. Burn rate B exhausts a 30-day budget in \( 30/B \) days.

Multi-window multi-burn-rate (the Google SRE workbook pattern) — page only when a fast window and a slow window both agree, so a 30-second blip does not wake anyone:

Reliability of an agent loop

An agent that takes n dependent steps, each succeeding with probability p:

$$P(\text{task success}) = p^{,n}$$

The lesson to say out loud: 95%-reliable steps give a 36% success rate at 20 steps. You do not fix this with a better prompt; you fix it by reducing n (fewer, coarser tools), raising p (validation, retries, deterministic tools), and making failure recoverable (checkpoints, compensation) so a failed step is not a failed task.

With per-step retry (r attempts, independent failures): \( p_{\text{eff}} = 1-(1-p)^r \). Two attempts turn 0.90 into 0.99 — if the step is idempotent. That "if" is the whole action gateway.

Latency budgets

Write the budget down before designing. A worked example for an interactive banking agent with a 3 000 ms p95 target:

Rules: (1) a fallback that does not fit in the remaining budget is decoration; (2) p95 of a serial chain is worse than the max of the components' p95s — tails compound; (3) parallelize everything that has no data dependency (input guardrails ∥ retrieval, embedding ∥ BM25).

Rate limiting: the token bucket

State: tokens, capacity C, refill rate r (per second), last_refill.

on request costing k at time t:
    tokens = min(C, tokens + (t - last_refill) * r)
    last_refill = t
    if tokens >= k:  tokens -= k;  admit
    else:            reject, retry-after = (k - tokens) / r

• Long-run rate is bounded by r; burst is bounded by C.
• C = r gives no burst tolerance; C = 60r tolerates a one-minute burst.
• Never let tokens go negative — that is the classic boundary bug, and it silently grants free capacity after a large request.
• For LLM traffic, meter tokens-per-minute and requests-per-minute. One request can be 100 000 tokens; an RPM-only limit does not protect you.

Capacity: PTU vs pay-as-you-go

Let \( C_p \) = monthly cost of the dedicated capacity, \( c_t \) = PAYG blended cost per 1 000 tokens, \( T \) = monthly tokens the dedicated capacity can actually serve at your input/output mix.

$$\text{break-even tokens} = \frac{C_p}{c_t}\times 1000 \qquad \text{utilization}_{\text{BE}} = \frac{\text{break-even tokens}}{T}$$

Say it like this in an interview: "Dedicated capacity wins above the break-even utilization; below it, PAYG wins and I keep the dedicated floor only for the latency-sensitive tier. I size the floor to p50 demand, spill the rest to PAYG, and I look up the current unit price and the measured throughput per unit rather than quoting a number from memory — the mix of input to output tokens changes throughput per unit by more than the price changes."

Three things that make this arithmetic wrong if you skip them:

1. Throughput per unit depends on your token mix, because prefill and decode cost differently. Measure with your own traffic shape.
2. Latency, not cost, is often the reason for dedicated capacity — it removes shared-pool congestion and 429s.
3. Reserved commitments are a finance instrument: a 1-year commitment on a model family is a bet against the model being deprecated. Price the exit.

Cost model for an agent action

$$\text{cost}{\text{action}} = \sum{\text{steps}} \Big[ (t_{\text{in}} - t_{\text{cached}}),c_{\text{in}} + t_{\text{cached}},c_{\text{cache}} + t_{\text{out}},c_{\text{out}} \Big] + \text{retrieval} + \text{tools}$$

The unit economic that matters is cost per successful action:

$$\text{CPSA} = \frac{\text{total spend}}{\text{successful actions}} = \frac{\text{cost}_{\text{action}}}{P(\text{success})}$$

A 30% failure rate multiplies your effective cost by 1.43 — quality is a cost lever, which is the sentence that turns an eval budget into a funded programme.

Scratchpad growth: an agent that appends every observation has input tokens growing quadratically over a run — step i re-sends everything from steps 1..i-1. With a base prompt b and average per-step addition a, total input tokens over n steps = \( nb + a,n(n-1)/2 \). At b=1 000, a=2 000, n=10 that is 100 000 input tokens, not 20 000; at n=20 the quadratic term alone is 380 000. Summarization, compaction, and prefix caching all attack this term.

Caching arithmetic

$$\text{effective cost} = (1-h)\cdot c_{\text{miss}} + h\cdot c_{\text{hit}} \qquad \text{savings} = h\left(1 - \frac{c_{\text{hit}}}{c_{\text{miss}}}\right)$$

Three non-negotiables for a semantic cache in a bank: tenant-scoped keys (never cross a tenant boundary), a high similarity floor with negative-example tuning, and no caching of personalized or entitlement-dependent answers. When in doubt, cache the retrieval, not the answer.

Serving: prefill, decode, KV cache

• Prefill processes N input tokens in parallel — compute-bound, roughly \( O(N) \) work per layer for the projections and \( O(N^2) \) for attention.
• Decode emits one token at a time — memory-bandwidth-bound: each step re-reads the weights and the KV cache.
• Therefore batching helps decode enormously (weights are read once for the whole batch) and helps prefill much less. That is why continuous batching exists.

KV cache size (per sequence), for a transformer with L layers, H KV heads, head dim d, sequence length S, and b bytes per element (2 for fp16):

$$\text{bytes} = 2 \cdot L \cdot H \cdot d \cdot S \cdot b$$

(the leading 2 is keys and values). Worked: L=32, H=8, d=128, S=8192, b=2 → \( 2\times32\times8\times128\times8192\times2 \) = 1.07 GB per sequence. On an 80 GB GPU with ~50 GB free after weights, that is ~46 concurrent 8k sequences — this is why max concurrency is a memory question, not a CPU question, and why grouped-query attention (small H) matters so much for serving economics.

Admission control follows: a request is admitted only if its projected KV footprint fits; otherwise it queues. Preemption/swapping trades latency for throughput.

Retrieval: BM25, RRF, recall

BM25 score of document D for query Q:

$$\text{BM25}(D,Q)=\sum_{q\in Q}\text{IDF}(q)\cdot\frac{f(q,D),(k_1+1)}{f(q,D)+k_1\left(1-b+b\frac{|D|}{\text{avgdl}}\right)}$$

with \( k_1 \in [1.2, 2.0] \) (term-frequency saturation) and \( b = 0.75 \) (length normalization). Intuition: term frequency saturates (the 10th occurrence adds little) and long documents are penalized.

Reciprocal Rank Fusion across retrievers, \( k = 60 \):

$$\text{RRF}(d)=\sum_{i}\frac{1}{k+\text{rank}_i(d)}$$

Score-free, so you never calibrate incomparable scales. A document ranked 1st and 10th scores \( 1/61 + 1/70 = 0.0307 \); ranked 3rd by both scores \( 2/63 = 0.0317 \) — consistent agreement beats a single strong signal, which is exactly the behaviour you want from hybrid.

Recall@k is the retrieval metric that bounds everything downstream: if the answer is not in the retrieved set, no amount of prompting recovers it. Measure recall@k on a golden set before tuning the generator.

Identity: the claims that matter

Validate on every token, at every hop:

The three-hop rule. User → Agent A → Agent B → Tool. At each arrow: exchange the token (RFC 8693), narrow the audience and scope, append to the actor chain, and shorten the lifetime. If any hop widens scope or drops the chain, the design is wrong.

Lifetimes: user session hours · agent access token minutes · tool credential seconds · workload SVID minutes with automatic rotation. Never "until someone rotates it."

Policy evaluation

The decision inputs are always the same four: subject (blended user+agent), action, resource, environment (time, risk score, posture, channel).

decision = DENY                      # default-deny
for rule in policies:                # deny-overrides
    if rule.matches(sub, act, res, env):
        if rule.effect == DENY: return DENY(rule)
        candidate = ALLOW(rule)
return candidate or DENY("no matching rule")

Say this in an interview: "Default-deny, deny-overrides, decisions cached with a short TTL and a policy-version stamp in the audit record, and the PDP is fail-static: if the control plane is unreachable the data plane keeps enforcing the last known-good policy bundle rather than failing open or failing shut."

Idempotency, retries and sagas

Side-effect classes and their policy:

Idempotency key store: key → (state, request_hash, response).

• Same key, same request hash, completed → return the stored response.
• Same key, different request hash → 409 conflict, never execute.
• Same key, in-flight → 409 / retry-after, never execute twice.

Saga: forward steps S1..Sn, compensations C1..Cn. On failure at Sk, run Ck-1 … C1 in reverse. Compensations must themselves be idempotent and retryable, because they run in exactly the conditions that made you need them.

Retry policy: exponential backoff with full jitter — sleep = random(0, base * 2^attempt) — capped, with a total budget, and never on a non-idempotent write without a key.

Circuit breaker settings

A breaker without a fallback behaviour just converts a slow failure into a fast one. Decide what "open" does: cached answer, degraded tool, queued action, or an honest refusal.

OWASP LLM Top 10 → control map

The protocol one-liners

• MCP — "one integration surface between an agent and its tools; JSON-RPC 2.0, capability negotiation at initialize, tools/list and tools/call, with resources and prompts alongside. It solved the M×N integration problem; it did not solve authorization, which is why the action gateway exists."
• A2A — "agents discovering and delegating to other agents across vendor and org boundaries. Agent Card for discovery, an explicit long-running task lifecycle, messages made of parts, and artifacts as durable outputs. It is peer-to-peer work handoff, not tool invocation."
• ACP — "a REST-shaped sibling with multipart messages and sync/async execution; treat it as another edge adapter over the same internal task model."
• The design rule — "the kernel speaks an internal task/message/artifact model; MCP, A2A and ACP are adapters at the edge. Otherwise every spec revision is a kernel rewrite."
• Interop with hyperscaler fabrics — "Azure AI Foundry, Bedrock AgentCore, and Google ADK each host agents and each speak some of these. The bank's platform must be able to front them and be fronted by them, which means our identity and policy must survive the translation — that is the actual integration risk, not the wire format."

Review red flags

Ten things that should stop a design review immediately:

1. A tenant_id read from the request body.
2. A long-lived service-account credential shared by an agent fleet.
3. A shared vector index with no namespace and a post-hoc filter.
4. A retry on a payment call with no idempotency key.
5. A fallback chain with no latency budget — it will breach the SLO on every failover.
6. A semantic cache with no tenant scope or no similarity floor.
7. "We'll add observability later" — traces at agent+tool granularity are how you debug non-determinism at all.
8. Policy evaluated only at session start for a task that runs for an hour.
9. An audit record without the policy version, model version, and actor chain — it cannot answer an examiner's question.
10. A control with no evidence artifact. If nothing is emitted, the control does not exist as far as audit is concerned.