« Phase 24 README · Warmup · Hitchhiker's · Principal Deep Dive · Core Contributor · Staff Notes

Phase 24 — Deep Dive: Amazon Bedrock, the Managed Foundation-Model Platform

The load-bearing mechanism in this phase is not "a model behind an API." It is a schema translation layer wrapped in three admission-control disciplines (throttle, queue, defer) and a two-phase content-safety filter. Everything else is packaging. This doc traces those mechanisms at the level a person implementing them has to reason about — data shapes, ordering, invariants, failure surfaces — because that is exactly what Lab 01 and Lab 02 make you build.

The translation layer is the whole game

InvokeModel is a passthrough: bytes in provider schema, bytes out provider schema. There is no abstraction — Bedrock is plumbing, not a contract. The moment you route to more than one provider, your call sites grow an if model_id.startswith("anthropic") ladder that leaks four incompatible JSON shapes into your application. Converse exists to move that ladder inside the platform.

The mechanism is a pair of pure functions per provider: to_native(unified_request) and from_native(native_response). The unified request is a small, fixed record — a system string, an ordered list of {role, content} turns, optional toolConfig, and inference parameters (maxTokens, temperature, topP). Each adapter maps that record onto its provider's native shape:

• Anthropic — system as a top-level field, messages as a typed content array, tools as a first-class tools list. Nearly 1:1 with the unified shape because Converse was modeled on it.
• Titan — a single inputText completion string and a textGenerationConfig. The multi-turn message list has to be flattened into one prompt string, and there is no native tool concept — so to_native must reject toolConfig rather than silently drop it.
• Llama — a raw prompt string with [INST] ... [/INST] instruction templating; system and turns are serialized into that template. Also no structured tool calling.
• Cohere — a message plus a chat_history array of {role: USER|CHATBOT, message} — close in spirit to the unified shape but with different role tokens and field names.

The invariant that makes this a layer and not a leak: from_native(to_native(x)) must return the same unified response record regardless of which adapter ran. Lab 01's forty tests are, at their core, a proof of that round-trip commutativity across four schemas.

Why the naive approach fails at the mechanism level. The tempting shortcut is a lowest-common JSON with optional fields and "just ignore what a provider can't do." That fails precisely on tool use. Titan and Llama have no tool-calling grammar; there is nothing to translate toolConfig into. If you silently drop it, the model answers in prose, your caller's tool-dispatch code sees no toolUse block, and the failure surfaces three layers away as "the agent stopped calling tools" — undebuggable. The correct mechanism is to make the adapter raise UnsupportedFeatureError at translation time, at the exact boundary where the capability gap is knowable. Real Bedrock does the same thing with a ValidationException. The ceiling of a translation layer is the least-capable target, and the only safe design surfaces that ceiling loudly at the edge instead of hiding it.

Admission control: three shapes of "no"

A model endpoint is a finite resource. Every serving platform must decide what happens when demand exceeds capacity, and Bedrock exposes three mechanically distinct answers. Getting these straight is the difference between "add more retries" and "do the capacity math."

On-Demand is a token bucket. State: capacity (burst), tokens (current), refill_per_tick. consume(n) succeeds and decrements iff tokens >= n; otherwise it raises ThrottlingException. Refill is monotonic up to capacity. The invariant is 0 <= tokens <= capacity at all times, and the key property is that rejection is immediate and total — the request is not held. The caller owns the retry, canonically exponential backoff with jitter. Lab 01 injects the tick instead of reading a wall clock so the bucket is deterministic; the algorithm is otherwise the real one. The failure mode is a 429 storm, and the bug is treating that storm as an application error rather than a signal you have crossed a capacity threshold.

Provisioned Throughput is a bounded queue. State: capacity (reserved slots), in_flight, a FIFO _queue. invoke() occupies a slot if one is free; if all slots are saturated it enqueues rather than rejects. complete() frees a slot and admits the head of the queue. The invariant is in_flight <= capacity, and the load-bearing behavioral difference from On-Demand is that saturation produces latency, not an error. You paid for capacity, so the platform holds your request. This is why "we're getting throttled" and "requests are slow" are diagnoses of two different capacity regimes, and why the fix for one (back off and retry) is wrong for the other (you are already admitted, retrying just adds load).

Batch is deferral. A job moves SUBMITTED → IN_PROGRESS → COMPLETED|FAILED. There is no real-time SLA; the platform schedules the work when it has slack, in exchange for roughly a 50% discount on the same per-token rate. The mechanism is an async state machine, not an admission gate on a live request. The lab processes in submission order in-memory for testability and explicitly notes real Bedrock reads/writes S3 and does not guarantee output order matches input order — an ordering invariant you must not assume.

The crossover is arithmetic, and you should be able to derive it live. Provisioned Throughput is fixed cost C_PT per Model-Unit-hour; On-Demand is variable cost C_OD per request. Set them equal over H hours with U units: N* = (C_PT × U × H) / C_OD. Below N* requests in the window, On-Demand is cheaper; above it, Provisioned Throughput is and you get throttle-free latency as a bonus. Plugging illustrative numbers ($21/unit-hour, 1 unit, 24 h, $0.02/request) gives ~25,200 requests/day. This is the calculation the "just add retries" reflex skips.

Cross-region routing sits on top

A cross-region inference profile is one logical modelId (ARN-shaped, conventionally prefixed like us.anthropic.claude-...) that fans out across a fixed member-region set. The mechanism is a router that, per request, picks an available member region — the lab models it as "lowest simulated queue depth wins." Two concrete wins: your per-region On-Demand quotas sum into one logical stream (higher effective throughput before you hit any single region's token bucket), and you can reach a model live in a region where it is available even if your home region lacks it. The lab reproduces the decision shape; real AWS routing telemetry is internal.

The two-phase safety filter

Guardrails is a filter over text, evaluated at up to two points: input (before the model is invoked) and output (after generation, before the caller sees it). The load-bearing invariant, which Lab 02 proves with a direct test: a blocked input guardrail means the model is never called at all — zero tokens spent, and denied content never enters the context window. This is why a guardrail is cheaper and safer than a post-hoc filter bolted onto the response.

Five independent policies compose into one verdict:

1. Denied topics — natural-language topic classification; the only subject-matter policy.
2. Content filters — six categories (hate, insults, sexual, violence, misconduct, prompt attack), each scored None/Low/Medium/High, each with a configurable filter strength. The threshold is the tuning knob, not the detector — a High-severity hit under a Low-strength config behaves differently than under High-strength.
3. Word filters — exact denylist plus built-in profanity.
4. PII filters — per entity type, either mask (redact just the matched span, keep the rest of the response) or block (replace the whole response). This is a real per-type UX decision, not a fixed behavior.
5. Contextual grounding — output-only, scores groundedness (is the claim supported by supplied sources?) and relevance against thresholds. It is the only policy that catches a fluent, polite, non-toxic, entirely fabricated answer — content filters look for toxicity, not truth.

Every intervention emits a trace naming the policy, the rule, and the action; the aggregate result is one GuardrailAction: NONE or GUARDRAIL_INTERVENED. Contextual grounding is output-only by construction — there is nothing to ground a bare prompt against.

Knowledge Bases: retrieval as a pipeline, not a database

A Knowledge Base is Phase 05's hybrid retriever run as a managed pipeline. Ingestion: a data source feeds documents through a chunking strategy, chunks are embedded, and vectors are written to a store you provision (OpenSearch Serverless, Aurora pgvector, Pinecone, Redis, MongoDB Atlas). Bedrock owns the pipeline; you own the store.

The mechanism worth tracing is hierarchical parent-child chunking. Split each document into large parent chunks, then split each parent into small child chunks. Embed and index only the children (small chunks make cosine similarity sharper and more precise). On query, match a child, but return the child's parent text for generation (large chunks give the model more context). This is the one strategy that buys precision and context instead of trading them; the data structure is simply a child→parent map alongside the vector index, and retrieve does the parent lookup after the nearest-child search. Fixed-size-with-overlap, by contrast, forces one chunk size to compromise both goals.

Retrieval exposes two APIs with different ownership boundaries: Retrieve returns ranked chunks and lets you build the prompt (right when the KB is one of several context sources); RetrieveAndGenerate does retrieval, prompt construction, generation, and citation assembly in one call, returning output.text plus a citations array tying each span back to the source chunk. Metadata filtering constrains retrieval to tagged chunks — the mechanism that makes one KB serve coarse multi-tenant RAG without one index per tenant.

What to hold onto

Strip the AWS branding and the mechanisms are provider-agnostic: normalize invocation with a per-target translation pair whose ceiling is the weakest target; pick admission control by failure shape (reject/queue/defer) and by the breakeven arithmetic; filter text in two phases with the model-never-sees-blocked-input invariant; retrieve by embedding small and returning large over a store you own. Learn the mechanism and every vendor's docs read the same.