« Phase 04 · Warmup · Track Overview

Deep Dive — Mechanism & Internals

Table of Contents

• 1. The ordering of complete
• 2. Two flags on an exception class
• 3. Routing: first match, two gates
• 4. The fallback loop, line by line
• 5. The token bucket's refill invariant
• 6. Check-both-then-consume-both
• 7. The hashing embedder
• 8. Cache read and write asymmetry
• 9. A traced request
• 10. Invariants, complexity, determinism

1. The ordering of complete

quota.check          → cheapest, protects the budget
rate_limiter.admit   → cheap, protects provider capacity
router.candidates    → pure computation
cache lookup         → keyed on the PRIMARY candidate
execute_with_fallback
cache store

Every step's position is a decision:

Quota and rate limit before routing. They are pure lookups and they reject the request entirely. Routing first would compute a chain nobody uses.

Cache after routing, not before. The exact-cache key includes the deployment name, because the same prompt against a different model is a different answer. So you need the primary candidate before you can build the key. The cost is that a cache hit still pays for routing — microseconds, and worth it for key correctness.

Rate limit before the cache. Debatable, and the lab chooses to charge a cache hit against the tenant's request rate. The argument: the rate limit protects the gateway, not only the provider, and a tenant hammering it with cacheable requests is still load. The counter-argument is that you are throttling the cheap path. Either is defensible; what matters is that it is a decision, and the lab documents which one it made.

Store after execute, unconditionally attempted but conditionally performed. _cache_store itself decides not to store when cacheable=False or the finish reason is not STOP. Putting the condition inside the method rather than at the call site means there is exactly one place the rule lives.

2. Two flags on an exception class

class GatewayError(Exception):
    retryable = False
    fall_over = False

Class attributes rather than instance state, because the semantics belong to the kind of failure, not to a particular occurrence. That makes the taxonomy readable as a table (the tests parametrize over exactly that table) and makes it impossible for a call site to construct a ContentFiltered that fails over.

The two flags are genuinely independent, and all four combinations are meaningful:

The lab uses two of the four; the type system supports all of them, which is the point of two flags rather than one enum.

3. Routing: first match, two gates

for rule in self.rules:                      # sorted by (priority, name)
    if not rule.matches(request): continue
    chain = [self.deployments[n] for n in rule.deployments]
    chain = [d for d in chain if self._admissible(d, request)]
    if chain: return chain                   # first NON-EMPTY match wins
return []

Note if chain: rather than return chain unconditionally. A rule whose deployments are all inadmissible (every one is offshore, and the request demands onshore) falls through to the next rule. That is deliberate: a specific rule that cannot be satisfied should not shadow a general one that can.

The two gates are structurally separate:

• rule.matches — what the policy author expressed: task class, tenant, classification membership.
• _admissible — what the facts require: residency and the deployment's own classification ceiling.

Keeping them apart means a policy author cannot accidentally grant something the deployment cannot do. The lab's test_deployment_classification_ceiling_is_a_second_gate is exactly this case: the restricted-data rule matched, and the confidential-only deployment was still removed.

Validation happens at construction, twice: Router.__init__ rejects a rule naming an unknown deployment, and RoutingRule.__post_init__ rejects a typo'd classification. Both are failures you want at deploy time, not at 3 a.m. — and a typo'd classification is especially nasty because "resticted" would silently match nothing and route everything to the default rule.

4. The fallback loop, line by line

for index, deployment in enumerate(candidates):
    if index > 0:                                        # (1)
        if request.side_effecting:                       # (2)
            raise last
        remaining = request.latency_budget_ms - self._elapsed_ms(started)
        if remaining < deployment.expected_latency_ms:   # (3)
            raise BudgetExhausted(...)
    attempts.append(deployment.name)                     # (4)
    try:
        response = adapter(request, deployment)
    except GatewayError as exc:
        last = exc
        if not exc.fall_over:                            # (5)
            raise
        continue
    ... success path ...                                 # (6)

1. index > 0 — the guards apply to fallbacks, not to the primary. A side-effecting request still gets one attempt; it just does not get a second.
2. Side-effecting raises last, not a new error. The caller needs to know why the primary failed, not merely that the gateway refused to continue.
3. The budget check uses expected_latency_ms, not a measured one. You cannot measure a call you have not made. Using the deployment's declared expectation is the only forward-looking number available, which makes keeping that field accurate an operational task, not a config nicety.
4. attempts is appended before the call, so a failed attempt appears in the record. An attempts list containing only successes would make the failover-rate metric useless.
5. fall_over is checked on the exception, not on the loop position. A non-failover error raises immediately even if there are candidates left.
6. On success the response is replaced with the real cost, the accumulated attempts, and the measured latency — the adapter's latency_ms was a stand-in.

The loop exits by raising last or NoRouteAvailable(...). The or matters for the degenerate case where every candidate had no registered adapter: last is an InvalidRequest, and without it the caller would see a confusing NoRouteAvailable for a chain that clearly had routes.

5. The token bucket's refill invariant

def _refill(self):
    elapsed = max(0.0, self.now() - self.last)
    self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
    self.last = self.now()

Three details:

• max(0.0, …) guards a clock that goes backwards. With an injected monotonic counter this is impossible; with NTP on a real machine it is not, and a negative elapsed would remove tokens.
• min(self.capacity, …) is what bounds the burst. Without it, an idle bucket accumulates forever and the first burst after a quiet night is unbounded.
• self.last is advanced on every refill, including one that adds nothing. Otherwise a sequence of sub-tick reads accumulates elapsed time repeatedly.

try_consume refills first, then compares. The order cannot be reversed: comparing against a stale token count refuses requests that the elapsed time has already paid for.

6. Check-both-then-consume-both

rpm, tpm = self._buckets(tenant)
if rpm.tokens < 1 or tpm.tokens < estimated_tokens:
    rpm._refill(); tpm._refill()
    if rpm.tokens < 1:   raise RateLimited(...)
    if tpm.tokens < estimated_tokens: raise RateLimited(...)
rpm.try_consume(1)
tpm.try_consume(estimated_tokens)

The shape looks redundant — why check, refill, check again? Because the first comparison is against possibly stale counts (cheap), and only if it looks like a refusal do we pay for a refill and check properly. On the common path (plenty of capacity) this is two comparisons.

The property being protected is the one the lab tests directly (test_a_rejected_request_does_not_spend_the_request_budget): a request refused on tokens must not consume an RPM slot. The naive implementation —

if not rpm.try_consume(1): raise ...
if not tpm.try_consume(n): raise ...     # the RPM slot is already gone

— throttles a tenant twice for one attempt, and the symptom is a tenant that appears to be over its request limit while sending very few requests. That is a genuinely confusing incident.

7. The hashing embedder

for token in text.lower().split():
    digest = blake2b(token, digest_size=8)
    index = int.from_bytes(digest[:4], "big") % dimensions
    sign = +1 if digest[4] % 2 == 0 else -1
    vector[index] += sign

This is the signed hashing trick (feature hashing / the hashing trick with random signs). The sign matters: without it, hash collisions always add constructively and every pair of documents looks more similar than it is. With random signs, collisions cancel in expectation, so the dot product remains an unbiased estimator of the true sparse dot product.

L2 normalization at the end makes cosine a plain dot product, which is both faster and less error-prone than dividing by norms at comparison time. The all-zero case (empty text) returns the zero vector rather than dividing by zero, and cosine against it is 0 — a miss, which is correct.

What this is not: a semantic embedder. It captures lexical overlap, not meaning. "The payment is held" and "the payment is not held" are near-identical under it — which is exactly the negative example the WARMUP warns about, and a good reason the lab's tests use a high threshold for the "must not collide" case and a lower one for the "should hit" case.

8. Cache read and write asymmetry

Read (_cache_lookup): exact first, then semantic. Exact is cheaper and more precise; if it hits there is no reason to embed.

Write (_cache_store): both, unconditionally (subject to the two rules). A response that missed both caches populates both, so a later exact repeat is cheap and a later near-duplicate still hits.

Two replace calls on a read that are easy to miss:

return replace(hit, cache="exact", cost_micros=0, attempts=())

• cost_micros=0 — cost means "what this call cost". Returning the original double-counts spend, and the lab's accounting would then report savings as spending.
• attempts=() — the stored response carries the attempts of the call that produced it, which have nothing to do with this one. Leaving them in would corrupt the failover-rate metric with historical data.

Both are the same underlying lesson: a cached object carries facts about its own creation, and several of those facts are wrong for the request being served. Enumerate them deliberately.

9. A traced request

complete(NormalizedRequest(task_class=REASONING, tenant="wholesale", data_classification="internal", latency_budget_ms=3000)), with the primary Azure deployment scripted to 429 once.

Step 12 is worth pausing on: the response came from eu, and it is cached under the key built from ptu — the primary. That is deliberate and it is the only consistent choice, because the next identical request will also compute ptu as its primary and must find the entry. The alternative (key on the deployment that answered) produces a cache that never hits after a failover, which is precisely when you most want it to.

The trade-off is honest and worth stating: the cached answer came from a different model than the key implies. For a bank, that is a reason to record the serving deployment in the accounting record (which the lab does) so the evidence trail is accurate even when the cache key is not.

10. Invariants, complexity, determinism

Invariants (each tested):

1. ContentFiltered.fall_over is False; every class's flags match the taxonomy table.
2. A rule naming an unknown deployment, or a typo'd classification, fails at construction.
3. candidates() returns [] rather than an inadmissible deployment.
4. A token-limit rejection does not consume an RPM slot.
5. The token bucket never goes negative and never exceeds capacity.
6. Quota refusal is inclusive at the boundary.
7. Quota and rate-limit refusals call no provider.
8. A cache hit reports cost_micros == 0 and calls no provider.
9. A cache never crosses a tenant boundary.
10. A non-STOP response is never cached.
11. A side-effecting request never has more than one attempt.
12. Every accounting record — including failures — has an outcome and a latency.
13. Two identically-constructed gateways produce identical responses and identical records.

Complexity:

The semantic cache's linear scan is the one that does not survive scale: at 10 000 entries per tenant and 64 dimensions that is 640 000 multiply-adds per lookup. Production uses an ANN index (the same structures as Phase 06), per tenant partition — which is where the partition decision starts costing memory, and where the "one index with a filter" temptation reappears with the same answer as before.

Determinism. No wall clock (injected), no RNG, no uuid4. hash_embed uses blake2b rather than hash(). Accounting.cost_by sorts its output. make_scripted_adapter consumes a scripted failure list, so "the primary 429s once then works" is reproducible byte-for-byte.