Lab 01 — Truss-Style Model Packaging & the Serving Contract

Phase: 20 — Custom Models in Production Difficulty: ⭐⭐⭐☆☆ (no math; the invariants — reproducible build, readiness gate, released slot — are the lesson) Time: 3–4 hours

Every "my model works locally but the deploy is broken" story is one of five bugs, and this lab builds the machine that makes all five impossible. You write the two halves of what truss push actually does: the packager (validate the config, then turn it into ordered, content-addressed image layers so the registry cache hits instead of re-downloading 16 GB of weights on every one-line code edit) and the server (nothing heavy in __init__, load() exactly once, 503 before ready, 429 past predict_concurrency, and a concurrency slot that comes back even when predict() explodes). Then you model where a 6-minute cold start actually goes, and wrap the output in the OpenAI envelope that lets your custom model drop into any existing client with a one-line base_url change.

What you build

• parse_config(raw) → TrussConfig — the packager's validation, and it is deliberately hostile: an unknown top-level key is a typo you must reject (a silently ignored resourcses: is how you serve a 70B model on a CPU); every requirement must be pinned (torch==2.4.0, not torch>=2.4); every weight source must name an immutable revision (revision: main means today's build and tomorrow's build serve different weights under the same image tag); and secrets are names only — the value is injected at runtime and must never enter a layer. use_gpu and total_vram_gb are derived, never hand-written.
• build_plan(config, code_digest) → ordered Layers — base_image → system_packages → python_requirements → model_weights → user_code, each addressed by sha256(canonical_json)[:16]. That order is the cache policy.
• plan_cache_reuse(old, new) — the longest common prefix by digest. Not a set intersection: OCI caching is positional, so the first differing layer invalidates everything downstream.
• cold_start_breakdown(...) — image pull + weight fetch + model load + engine build, with a weights_cached branch that swaps a cross-internet hub download for a local NVMe read. This one boolean is worth minutes per replica start.
• ModelServer — start() (load exactly once), try_admit()/release() (the admission gate), handle() (readiness → admission → preprocess → predict → postprocess, slot released in a finally), handle_stream() (eager validation, slot released even when the client disconnects mid-stream), and health().
• to_openai_chat_completion(...) — the response envelope, with a deterministic id and a created you pass in rather than read from the clock.

Key concepts

Files

Run

pip install -r requirements.txt
pytest test_lab.py -v                       # against your lab.py
LAB_MODULE=solution pytest test_lab.py -v   # against the reference
python solution.py                          # the worked example

Success criteria

• All 62 tests pass against your implementation (and LAB_MODULE=solution).
• You can explain why test_cache_reuse_dependency_bump_invalidates_weights_and_code is a soul test: bumping one pinned dependency re-runs every downstream layer including the multi-GB weight pull, which is why a "one-character change" can turn a 40-second deploy into a 9-minute one — and why you order layers by volatility.
• You can explain why test_slot_is_released_when_predict_raises is the other soul test, and describe the production failure it prevents (capacity decaying by one per failed request until someone restarts the pod).
• You can state, without looking, what parse_config rejects and why each rejection corresponds to a real incident.
• You can trace handle() end to end and name the HTTP status each guard maps to (503 not-ready, 429 over-capacity, 400 malformed, 500 model failure).
• Given image_gb=4, weights_gb=16, you can say roughly how long a cold start takes from the hub vs from a warm cache, and which lever you'd pull first.

How this maps to the real stack

Limits of the miniature (be honest in the interview): there is no Docker here — the "layers" are hashes, not filesystems, and real BuildKit cache keys also include the command string and build args, so a changed RUN line busts a layer even with identical content. Our admission gate is a counter in one thread, not a real semaphore across an async event loop with a request queue and timeouts. Cold start is modeled as four serial phases; real systems overlap image pull with node provisioning and stream weights layer-by-layer. And the OpenAI envelope here is the non-streaming shape — streaming uses SSE chat.completion.chunk deltas with a terminating [DONE].

Extensions (build these for real)

• Do it for real: truss init, drop a small HF model in load(), truss push, then truss predict -d '{"prompt":"hi"}'. Time the first deploy vs. the second.
• Emit a real Dockerfile from build_plan and confirm with docker build that a code-only edit prints CACHED for the first four layers.
• Add a request queue in front of the admission gate with a max depth and a per-request deadline, and return 429 only when the queue is full — then measure how queueing changes p99 vs. shedding immediately.
• Add engine_build_s for real: build a TensorRT-LLM engine, cache the artifact, and show the second cold start skipping the build entirely.
• Extend to_openai_chat_completion into the streaming shape (chat.completion.chunk, delta, SSE framing) and point the real OpenAI SDK at it.
• Add a truss watch-style dev loop: hash model/, and on change push only the user_code layer.

Interview / resume

• Talking points: "Why does the weight layer go before the code layer?" "What happens to a request that arrives while the model is still loading, and what status do you return?" "Where does your model server leak capacity?" "Your cold start is 6 minutes — walk me through where the time goes and what you'd fix first." "Why is revision: main a production bug?"
• Resume bullet: Built a dependency-free model-packaging and serving harness — hostile config validation (pinned deps, immutable weight revisions, secrets by reference), content-addressed build layers ordered by volatility with prefix-based cache-reuse accounting, a cold-start cost model, and a serving contract with readiness gating, concurrency admission control, leak-free slot release and eager stream validation — proven by a 62-test deterministic suite.