« Phase 04 · Warmup · Track Overview

Staff Notes — Judgment, Review Signal & Seniority

Table of Contents

• 1. Build vs buy
• 2. A decision framework for a routing change
• 3. Review red flags
• 4. Production war stories
• 5. The interview signal
• 6. Mentoring notes

1. Build vs buy

The line, again: buy the mechanics, build the policy. And one specific warning: it is very tempting to express the policy in an API-management product's policy language because it is already there. Resist it. A routing decision that must be auditable and testable should be code with a test suite, not XML in a portal.

2. A decision framework for a routing change

Someone proposes "route task X to model Y." Six questions, in order:

1. What changes for the caller? Latency, quality, cost — name the expected direction of each. "Cheaper" alone is not an answer, because a cheaper model that fails more often raises cost per successful action.
2. What is the evidence? An evaluation on a golden set for this task class. Without it you are changing the platform's behaviour on a hunch.
3. What are the residency and classification implications? If the new deployment is in another region, this is a data-flow change, not a config change, and it may need review.
4. What is the fallback, and does it fit the budget? A new primary with a slower fallback can silently make the SLO unachievable.
5. How do we roll it back, and how fast? If the answer is "redeploy the gateway", the config model is wrong.
6. How will we know it worked? Cost per successful action, task success rate, p95 — before and after, on the same traffic slice.

If the change is large, the answer is canary: a small deterministic slice (keyed on session so a user does not flip mid-conversation), watched for a week.

3. Review red flags

In a design document

• Callers specify a model name. A vendor decision hard-coded into twelve repositories.
• One retryable flag. You will either retry a content filter or refuse to fail over on a 429.
• A fallback chain with no latency budget.
• Retry policy that does not mention idempotency or side effects.
• A semantic cache proposal that does not say "tenant-partitioned" in the same paragraph.
• No statement of what happens when the config store is unreachable.
• A synchronous policy or database lookup on the gateway's request path.
• No cost attribution dimension beyond "total".
• "We'll normalize errors later." That is the abstraction layer; without it there is no layer.
• No mention of model version pinning.

In code

# Red flag: the tenant from the body
tenant = body["tenant_id"]

# Red flag: one flag doing two jobs
if err.retryable: try_next_deployment()          # now a content filter shops for a model

# Red flag: fallback with no budget check
except ProviderError: return call(fallback)

# Red flag: retry a side-effecting call
@retry(attempts=3)
def complete(request): ...

# Red flag: cache key without the tenant
key = sha256(prompt + model)

# Red flag: caching whatever came back
cache[key] = response                             # including finish_reason == LENGTH

# Red flag: cost only on success
if response.ok: accounting.record(cost)

# Red flag: price arithmetic in floats, divided early
cost = (tokens / 1000) * price_per_1k             # accumulate a month of these

# Red flag: consuming the request slot before checking tokens
if not rpm.try_consume(1): raise
if not tpm.try_consume(n): raise                  # throttled twice for one attempt

In an incident review

• "We didn't know which teams were affected" → attribution by agent, not just tenant.
• "The fallback made it worse" → budget check, and measure the fallback's real p95.
• "We couldn't roll back the routing change" → config is code, staged and reversible.

4. Production war stories

Shopping for a compliant model. A fallback chain that fired on any error. The provider's safety system refused; the second provider refused; the third answered. Discovered in a model-risk review, and the reviewer's question was verbatim: "So the system tries providers until one agrees to produce the content?"

The fallback that breached the SLO. A 4-second fallback timeout inside a 3-second p95 budget. Every provider blip converted a partial degradation into a fleet-wide breach. The dashboards were maddening: every component healthy, the SLO failing.

The retry that doubled the payments. The gateway retried on timeout. Some of those requests had already caused a downstream tool call. The side_effecting field existed and was optional, so nobody set it. Make it required with no default.

The cache that leaked. Semantic cache keyed on prompt embedding with the tenant applied as a post-filter. A refactor moved the filter one function up. Hit rate 34%, everyone delighted, until a Retail user got a Wholesale answer. A 200, a happy user, and a breach found months later.

The factor of a thousand. A price table entered as per-1M while the code assumed per-1K. Cost reporting was off by 1000× in the reassuring direction for six weeks. Caught by the monthly reconciliation nobody had wanted to build.

The unpinned model. A provider silently updated a model. Output format shifted subtly; a downstream parser started failing on 3% of requests. There was no eval gate because there had been no change to gate — the change happened on the provider's side.

The invoice nobody could decompose. No attribution beyond total spend. Finance asked which team caused a 40% rise; the answer took three weeks and was an estimate.

5. The interview signal

Signal 1 — you say "normalizing errors is the job." Anyone can describe a unified request shape. The candidate who volunteers that the error taxonomy is the hard and valuable part has operated one of these.

Signal 2 — two flags, and the content-filter example. retryable and fall_over as separate concerns, with "shopping for a compliant model" as the reason. This is the single most distinguishing observation in the phase, and it lands especially hard in a regulated interview.

Signal 3 — you do the budget arithmetic unprompted. "2 400 elapsed of 3 000, fallback expects 800, so no — attempting it breaches for every affected request." Numbers end arguments.

Signal 4 — you refuse to fail over side-effecting calls, and you explain why (a timeout is not evidence of non-execution) and where the fix lives (the caller's idempotency key, Phase 10).

Signal 5 — you name the semantic cache as the only silent failure. And give the three non-negotiables without being asked.

Signal 6 — you defend the single point of failure with the counterfactual. "Twelve teams each being their own single point of failure, none of them measured." Plus: stateless, fail-static, no I/O on the request path.

Anti-signals:

• Describing the gateway as a proxy.
• A unified request shape with no mention of errors.
• Failing over on any error.
• Cost per token as the optimization target.
• No answer to "what happens when the config store is down?"
• Enthusiasm for semantic caching with no mention of tenancy.

The question to ask them: "What fraction of your input tokens are billed at the cached rate, and do you reconcile gateway spend against the provider invoice?" Both answers tell you immediately how mature the cost model is, and asking shows you know where the real money is.

6. Mentoring notes

Three exercises:

1. Hand them six provider error responses and ask for the taxonomy. Include a content filter returned as a 200 with an empty completion — the one that breaks naive status-code mapping. Then ask which of their classes should trigger failover, and watch the content-filter realization happen.
2. Give them a latency budget and a failing primary. Ask "fall over?" with three different elapsed times. The moment they say "it depends on the remaining budget" without prompting, they have it.
3. Ask them to design the semantic cache key. If the tenant is not the first thing they write, walk through the leak. It is a five-minute conversation that permanently changes how someone thinks about multi-tenant caching.

And the framing for the platform team: the gateway is where a platform stops being a library. Every control in the rest of this track — residency, classification, quotas, attribution, model governance — is enforced here or nowhere. That is the argument for staffing it properly, and for resisting the pressure to make it a thin proxy that "just forwards requests."