« Phase 05 · Warmup · Track Overview

Deep Dive — Mechanism & Internals

Table of Contents

• 1. The memory budget as a subtraction
• 2. The decode expression, and what each term does
• 3. The continuous batcher's loop order
• 4. Admission on the projected length
• 5. Why static batching is modelled as one number
• 6. A traced simulation
• 7. The break-even algebra
• 8. Invariants, complexity, determinism

1. The memory budget as a subtraction

free = gpu.total_memory_bytes - model.weight_bytes()
if free <= 0: return 0
return int(free * (1.0 - overhead_fraction))

Three decisions in four lines.

total_memory_bytes multiplies by count. Modelling a tensor-parallel group as one aggregate device is a simplification that is exactly right for memory (each device holds 1/N of the weights and 1/N of each sequence's KV) and approximately right for bandwidth (they read in parallel) and FLOPs. It is wrong about communication, which does not scale — and the docstring says so, because an undocumented simplification in a capacity model becomes a procurement error.

The <= 0 floor returns 0 rather than a negative. A negative KV budget would propagate into max_concurrent_sequences as a negative concurrency, which would then compare > against thresholds and pass. Returning 0 makes "the model does not fit" a representable, testable state, and the lab asserts it for a 70B fp16 model on one 80 GiB card.

The overhead fraction is a multiplication, not a subtraction of a fixed amount. Working space scales roughly with the model and the batch, not with a constant, so a fraction generalizes across model sizes. It is still a crude model — real systems profile at startup — and 10% is a starting point, not a law.

2. The decode expression, and what each term does

bytes_read = model.weight_bytes() + batch_size * model.kv_bytes(context_tokens)
seconds    = bytes_read / gpu.total_bandwidth_bytes_per_s
return (seconds / batch_size) * 1000.0

Two terms, and the whole phase lives in their asymmetry:

Divide by batch_size at the end and you get per-token time:

$$t = \frac{W}{\text{BW} \cdot B} + \frac{\text{KV}}{\text{BW}}$$

The first term falls as \( 1/B \); the second is constant in B. So batching helps until the second term dominates — which happens when \( B \cdot \text{KV} \approx W \).

Worked for the lab's 70B model on an 8-way node: weights 140 GB, KV at 2 000 tokens is 0.66 GB. The crossover is at \( B \approx 140 / 0.66 \approx 213 \). Below that, batching is buying you throughput; above it, you are mostly paying for KV reads. At 8 000 tokens of context the crossover drops to \( B \approx 53 \).

That is a design number. It tells you that long-context workloads saturate the batching benefit early, which is a direct argument for context compaction (Phase 01) framed as a serving cost rather than a token-price cost.

The two lab tests pin both regimes: with context_tokens=0, doubling the batch exactly halves per-token time; with context_tokens=200_000, it does not come close.

3. The continuous batcher's loop order

1. arrive   — move newly-arrived requests into the queue
2. retire   — remove finished slots, emit results
3. admit    — fill freed capacity from the queue
4. check    — terminate if nothing running, queued, or pending
5. step     — every slot produces one token, context += 1

Retire before admit is what makes the batching continuous. Reverse them and a finishing sequence's slot stays empty for one step, every time — which at high turnover is a measurable throughput loss and, more importantly, is not what continuous batching means.

Arrive before retire matters less but is worth being deliberate about: a request that arrives on the same tick a slot frees should be able to use it.

The termination check sits between admit and step, not at the top. At the top, a simulation that has just retired its last slot would run one more empty step. Between them, it exits immediately — which is why total_ticks is comparable between the two batchers rather than off-by-one.

first_token_tick is set inside the step, on a slot's first execution, rather than at admission. That distinguishes admitted from producing, which is the difference between queue time and prefill time in a real system. Our model conflates prefill into a single tick; a richer one would separate them, and the field is where that extension attaches.

4. Admission on the projected length

def _projected(self, request):
    return self.model.kv_bytes(request.prompt_tokens + request.output_tokens)

The naive alternative — budgeting prompt_tokens — passes every test with a short workload and fails catastrophically in production. Here is the failure, concretely:

• Budget 10 GiB. Ten requests, each 4 000-token prompt (0.5 GiB) and 4 000-token output.
• Admit on prompt: all ten fit (5 GiB). ✓
• Each grows to 8 000 tokens → 1 GiB each → 10 GiB. At the last step, allocation fails.
• The OOM does not kill the new request; there is no new request. It kills the batch, including sequences that were 99% complete.

That asymmetry — an over-admission failure destroys completed work — is why the conservative choice is right for a bank platform, and why real systems that do over-commit (via paging) pair it with preemption: a sequence is swapped out and resumed rather than lost. Over-commitment without preemption is not a strategy.

The second rule is the never-fits check:

if projected > self.kv_budget:
    rejected.append(request.request_id)
    continue

Placed before the capacity check, so a too-large request is rejected regardless of current occupancy. Placed after, it would sit in the queue being re-evaluated forever while the caller waits — a leak with a customer attached. The lab tests that ok-1 and ok-2 are still served while too-big is rejected: one bad request must not block the queue.

5. Why static batching is modelled as one number

longest = max(r.output_tokens for r in batch)
for request in batch:
    finished_tick = start + longest        # everyone waits for the longest

This is the entire pathology in one line, and modelling it exactly (rather than simulating step-by-step) makes the comparison honest: static batching's only difference from continuous is that slots are not reused until the whole batch drains. Everything else — admission, KV budget, max batch — is identical between the two classes, so the measured difference is attributable to the scheduling policy and nothing else.

That is a deliberate experimental design choice. If the two simulators differed in more than one respect, the 2.8× would not be evidence of anything.

6. A traced simulation

Workload: 24 requests arriving 4 per tick over 6 ticks, 512-token prompts, output length 400 for every 8th request and 20 for the rest. max_batch=8, 8-way node, 70B model.

Static: fills a batch of 8 (arrivals permitting), finds the longest output in it, and advances the clock by that. Three of the eight batches contain a 400-token generation, so those batches each occupy 400 ticks while seven slots idle after tick 20.

batch 1 (r00..r07) → contains r00 (400)  → 400 ticks
batch 2 (r08..r15) → contains r08 (400)  → 400 ticks
batch 3 (r16..r23) → contains r16 (400)  → 400 ticks
                                    total 1200 + arrival slack = 1220

Continuous: the three long generations occupy three slots for 400 steps each; the remaining 21 short generations flow through the other five slots at 20 ticks apiece. 21 × 20 = 420 ticks of short work spread over 5 slots ≈ 84 ticks, so the wall-clock is dominated by the long generations — 440 ticks total.

The mean-latency difference (5×) is larger than the throughput difference (2.8×) — because static batching penalizes the short requests, which are the majority. The user-visible effect of continuous batching is bigger than its throughput effect, which is worth knowing when you are justifying the migration to someone who only looks at GPU utilization.

7. The break-even algebra

blended_per_1k = c_in * (1 - f) + c_out * f
tokens = int(monthly / blended_per_1k * 1000)
utilization = tokens / capacity

The subtlety is that f is the fraction of tokens that are output, and output is typically 4× input. So the blended price is not near the input price unless f is tiny:

int() truncates, which is deliberate — reporting a break-even below the true value would make dedicated capacity look better than it is, and truncation errs the safe way.

break_even_utilization returns inf when capacity is zero rather than dividing by zero. That case is reachable (a misconfigured tokens_per_unit_month=0) and inf is the honest answer: you never break even on capacity that serves nothing.

cheapest_option sorts on (cost, rank, choice) with PAYG at rank 0, so a tie goes to the option with no commitment and no operational burden. Encoding the tie-break rather than relying on enum ordering makes the preference explicit and testable, which the lab does directly.

8. Invariants, complexity, determinism

Invariants (each tested):

1. A model that does not fit yields a KV budget of exactly 0, never negative.
2. peak_kv_bytes <= kv_budget, always.
3. A request whose projected KV exceeds the whole budget is rejected, and does not block others.
4. Both batchers serve exactly the same request set.
5. Static batching gives every member of a batch the same finished_tick; continuous does not.
6. With zero context, decode(2B) == decode(B) / 2 exactly.
7. A heavier output mix lowers both break-even tokens and break-even utilization.
8. Spillover's total == provisioned + payg, and payg == 0 below capacity.
9. cheapest_option ties resolve to PAYG.
10. Two identical simulations produce identical results.

Complexity:

The continuous batcher's per-tick queue scan is \( O(Q) \), so a workload with a large standing queue is \( O(T \cdot Q) \). Fine at lab scale; a real scheduler keeps the queue sorted by admission feasibility and stops at the first request that does not fit.

Determinism. No clock (a tick counter), no RNG, no floating-point accumulation in the money path (integer micro-USD throughout). Requests are sorted by (arrival_tick, request_id) before simulation and results are sorted by request_id after, so output is diffable across runs and across machines. int() truncation rather than rounding keeps money comparisons exact.