JD2 Track — AI Inference, Model Optimization & Accelerator Solutions Architecture

Target role: AI Specialist / Solutions Architect for AI accelerators — the jd2.md role. This is the Qualcomm Cloud AI / NVIDIA Solution Architect / AWS Annapurna-Inferentia / Groq / Cerebras / SambaNova archetype: the person who takes a customer's model and hardware and makes them meet in production at the right latency, throughput, accuracy, and cost.

Why this track exists. The original AI Specialist/ folder targets a different job — the Parsons air-gapped Digital Twin role (jd.md): RAG, QLoRA, function-calling, guardrails. That is an applied-LLM-application role. JD2 is an inference-systems and model-optimization role. The two share the word "LLM" and almost nothing else. This track fills the ~85% of JD2 that the original labs never touch: GPU/accelerator microarchitecture, quantization math, model porting/compilation, high-throughput serving, distributed inference, performance profiling, CV/video pipelines, capacity sizing, and customer-facing solutions architecture.

The promise of this track: from zero understanding of how a GPU executes a matmul → to a principal engineer who can walk into a customer, size their cluster, port and quantize their model, debug a 3× throughput regression with a profiler, and explain every trade-off on a whiteboard. And, beyond the role, to the level where you could found an inference-infrastructure company.

Table of Contents

• 1. What JD2 Actually Demands (decoded)
• 2. Gap Analysis vs. the Existing Folder
• 3. The Mental Model: The Inference Stack
• 4. Curriculum — Knowledge Modules
• 5. Curriculum — Labs
• 6. The 12 Numbers Every Principal Knows Cold
• 7. Demand Primer: The Business of Inference
• 8. Study Order & Milestones
• 9. From Specialist to Founder
• 10. References

1. What JD2 Actually Demands (decoded)

The job description is written in HR-compressed language. Decoded into engineering competencies:

The one-sentence job: Bridge AI model requirements ↔ hardware capabilities ↔ customer expectations — guiding customers from model selection → hardware sizing → deployment → production readiness. Everything in this track serves that bridge.

2. Gap Analysis vs. the Existing Folder

Verdict: the original folder, on its own, would get you screened out of JD2 at the phone stage — it demonstrates application skill, not systems skill. This track is the missing 85%.

3. The Mental Model: The Inference Stack

Everything in this track maps to one layered picture. Memorize it; it's the whiteboard you'll draw in every customer meeting and interview.

┌──────────────────────────────────────────────────────────────────────┐
│  L7  Application / pipeline   (RAG, agent, det+track+recognize, video) │  ← original folder lives here
├──────────────────────────────────────────────────────────────────────┤
│  L6  Serving layer           (vLLM, TGI, TensorRT-LLM, Triton)         │  ← continuous batching, KV mgmt, scheduling
├──────────────────────────────────────────────────────────────────────┤
│  L5  Runtime / execution     (CUDA Graphs, kernels, fused ops)         │  ← where latency is won or lost
├──────────────────────────────────────────────────────────────────────┤
│  L4  Compiler / graph        (ONNX, TensorRT, XLA, torch.compile)      │  ← op fusion, layout, lowering
├──────────────────────────────────────────────────────────────────────┤
│  L3  Numerics                (FP16/BF16/FP8/INT8, quant schemes)        │  ← accuracy ↔ speed ↔ memory
├──────────────────────────────────────────────────────────────────────┤
│  L2  Parallelism / topology  (TP, PP, EP, SP; NVLink/PCIe/fabric)      │  ← scale beyond one chip
├──────────────────────────────────────────────────────────────────────┤
│  L1  Accelerator hardware    (SMs/tensor cores, HBM, on-chip SRAM)     │  ← the roofline that bounds everything
└──────────────────────────────────────────────────────────────────────┘

A principal engineer can start at any layer from a symptom ("p99 latency doubled at 32 concurrent users") and walk down to root cause ("KV cache spilled, PagedAttention block table thrashing, you're memory-capacity-bound — shard KV with TP=2 or quantize KV to INT8"). This track builds that traversal skill layer by layer.

4. Curriculum — Knowledge Modules

Each is a deep, from-first-principles document with its own TOC, worked math, and references.

5. Curriculum — Labs

Hands-on, runnable. Each lab has its own README with steps, a measurable result, and a resume bullet. (Labs are scaffolded as you progress; the knowledge modules are the prerequisite reading.)

6. The 12 Numbers Every Principal Knows Cold

You should be able to recite and use these without looking them up. (Derivations and current values live in the knowledge modules.)

1. A matmul of M×K·K×N costs 2·M·N·K FLOPs. (the factor of 2 = multiply + add)
2. Decoding a token of a P-parameter model costs ~2P FLOPs and reads ~2P bytes of weights (in FP16). This is why decode is memory-bandwidth-bound.
3. Arithmetic intensity = FLOPs / bytes moved. Compare it to the hardware's FLOPs/byte ratio (the roofline ridge point) to know if you're compute- or memory-bound.
4. HBM bandwidth (≈1–8 TB/s) is ~10–30× slower than on-chip SRAM, and SRAM is ~100× smaller. This gap is why FlashAttention and op fusion exist.
5. KV cache size = 2 · n_layers · n_kv_heads · head_dim · seq_len · batch · bytes_per_elem. Memorize this; it sizes your serving memory.
6. Roofline ridge point of an H100 ≈ ~300+ FLOPs/byte (FP16). Below it you're memory-bound; above, compute-bound.
7. Prefill is compute-bound; decode is memory-bound. Different bottlenecks → different optimizations.
8. INT8 ≈ 2× the throughput and ½ the memory of FP16, typically <1% accuracy loss with good calibration. INT4 ≈ 4×/¼ with more care.
9. Tensor parallelism adds an all-reduce per layer; its cost scales with hidden_size and is bounded by interconnect bandwidth (NVLink ≫ PCIe).
10. Continuous batching can 5–20× throughput over static batching by refilling slots as sequences finish.
11. MFU (Model FLOPs Utilization) of a well-tuned LLM serving system is often only 30–60% on decode — the rest is lost to memory stalls and small batches.
12. $/1M tokens is the unit of competition. It's (GPU $/hr) ÷ (tokens/hr). Halving it is the whole game.

7. Demand Primer: The Business of Inference

A principal engineer doesn't just optimize kernels — they understand why a customer pays. This is the commercial context that makes you a "trusted technical advisor," not just an implementer.

Why this role exists now

Training is a one-time capex bet; inference is the recurring opex that scales with usage. As GenAI moves from demos to production, the cost of serving dominates. A 2024–2026 shift: enterprises realize their LLM bill is 80–95% inference, not training. Every accelerator vendor (NVIDIA, AMD, Qualcomm, AWS, Google, Groq, Cerebras, SambaNova, Tenstorrent) is fighting for that recurring spend, and each needs solutions architects who can land a customer's model on their silicon at a winning $/token. That is JD2.

The competitive landscape (know the players)

• NVIDIA: incumbent. CUDA moat, TensorRT-LLM, Triton, NVLink/NVSwitch, H100/H200/B200/GB200. The thing every competitor is benchmarked against.
• AMD: MI300X (huge HBM, good for large models), ROCm software catching up.
• Qualcomm Cloud AI (AI100/AI80): power-efficiency play, edge + data center, qaic toolchain. (JD2's likely employer archetype.)
• AWS: Inferentia/Trainium (Neuron SDK), captive cloud demand.
• Google: TPU (XLA, JAX), captive + Cloud.
• Groq: deterministic, single-token-latency champion (LPU, SRAM-based, no HBM).
• Cerebras: wafer-scale, weight-streaming.
• SambaNova, Tenstorrent, Etched (Sohu, transformer-ASIC): bets on specialization.

The whole industry runs on one tension: NVIDIA's software ecosystem vs. everyone else's better-or-cheaper silicon that's harder to program. A JD2 specialist is the human bridge across that gap.

The metrics customers actually buy on

The principal's commercial instinct

A customer says "we need lower latency." The junior reaches for a smaller model. The principal asks: which latency (TTFT or TPOT), at what concurrency, against what SLO percentile, and what's the accuracy floor and budget? Then sizes the trade-off. That diagnostic reflex — turning a vague ask into a measurable spec and a Pareto choice — is what gets you hired and promoted. The whole of Knowledge 08 trains it.

8. Study Order & Milestones

Phase A — Foundations (you can read a roofline). Knowledge 00 → 01. Milestone: derive the KV-cache size and decode FLOPs for Llama-3-70B at 4k context from memory.

Phase B — Make it small and portable. Knowledge 02 → 03 + Labs 01, 02. Milestone: quantize a 7B model to INT8, port it to ONNX/TensorRT, and produce an accuracy/latency table.

Phase C — Make it serve at scale. Knowledge 04 → 05 + Lab 03. Milestone: stand up vLLM, tune it to a target SLO, and explain continuous batching + PagedAttention on a whiteboard.

Phase D — Make it fast & prove it. Knowledge 06 + Lab 04. Milestone: read an Nsight trace and correctly name a memory-bound vs compute-bound stall.

Phase E — The full pipeline & the customer. Knowledge 07 → 08 + Lab 05. Milestone: write a one-page sizing memo for a hypothetical customer and defend the $/token.

Phase F — Interview & demand. interview-prep/. Milestone: pass a mock system-design ("size and design inference for X") and a trade-off whiteboard.

9. From Specialist to Founder

JD2 asks for someone who can "found companies and startups." The bridge from principal IC to founder in this space:

• The wedge products: inference cost reduction (the $/token war), model porting/compilation tooling, serving infrastructure, eval/observability for inference, specialized hardware enablement, on-prem/sovereign AI appliances. Every one of these is a real 2024–2026 startup category (Fireworks, Together, Baseten, Modal, OctoAI[acq], Predibase, etc.).
• The unfair advantage a JD2 principal has: you've seen many customers' workloads and the gap between vendor benchmarks and real goodput. That pattern library is a founder's moat.
• What you'd still need to learn: GTM, pricing, the capital-intensity of hardware bets, and the brutal NVIDIA-moat reality. The technical depth in this track is necessary but not sufficient — pair it with the commercial primer in §7 and Knowledge 08.

This track makes you technically fundable. It is honest about the rest.

10. References

Authoritative sources, current as of 2026. Each knowledge module has its own deeper reference list.

• Roofline model — Williams, Waterman, Patterson, "Roofline: An Insightful Visual Performance Model" (CACM 2009).
• FlashAttention — Dao et al., FlashAttention (2022) and FlashAttention-2 (2023); FlashAttention-3 (2024, FP8/Hopper).
• PagedAttention / vLLM — Kwon et al., "Efficient Memory Management for LLM Serving with PagedAttention" (SOSP 2023).
• GPTQ — Frantar et al. (2022). AWQ — Lin et al. (2023). SmoothQuant — Xiao et al. (2022). LLM.int8() — Dettmers et al. (2022).
• Speculative decoding — Leviathan et al., Chen et al. (2023). Medusa (2023). EAGLE (2024).
• Megatron-LM (tensor/pipeline parallelism) — Shoeybi et al. (2019), Narayanan et al. (2021).
• TensorRT-LLM, NVIDIA Triton, vLLM, Hugging Face TGI — official docs.
• NVIDIA Hopper/Blackwell architecture whitepapers; Qualcomm Cloud AI 100 product/SDK docs; AWS Neuron, Google TPU/XLA docs.
• DeepStream / GStreamer / FFMPEG / NVDEC — NVIDIA Video Codec SDK and DeepStream docs.
• "Efficiently Scaling Transformer Inference" — Pope et al. (Google, 2022) — the canonical sizing-math paper.

Each module restates the load-bearing results in full so you never have to take a number on faith — per the depth standard of this curriculum.