👨🏻 Brother Talk — Phase 10, Off the Record

GenAI on edge — the JD's headline: "latest GenAI techniques and adapt those for NPUs." This is the frontier, the reason this role exists right now, and the hardest constraint set.

Brother, this is the phase — the one the JD leads with, the reason Qualcomm is hiring Senior Staff ML engineers in 2026. Running a multi-billion-parameter generative model on a phone — on a battery, with a few gigabytes of memory, with no datacenter to fall back on — is one of the hardest problems in applied ML right now, and the people who can do it are vanishingly rare. Every technique you've learned converges here, under the most brutal constraints in the whole field. If you nail this phase, you're not interviewing for the job; the job is interviewing for you.

The constraint that dominates everything: memory. A 7B model in FP16 is ~14GB — that doesn't fit on a phone, full stop. So edge GenAI is, first and foremost, a fit the model in memory problem, and that's why weight-only quantization to 4-bit (and below) is non-negotiable here. AWQ (the lab) is the technique to internalize: Activation-aware Weight Quantization. Its insight is that not all weights matter equally — a small fraction of "salient" weight channels (identified by the activation magnitudes, not the weights themselves) carry most of the accuracy. Protect those (keep them high-precision or scale them cleverly), crush the rest to 4-bit, and you hold accuracy where naive quantization collapses. AWQ vs GPTQ (P03) is a great interview compare-and-contrast: GPTQ uses second-order error compensation, AWQ uses activation-aware scaling — both fight the same battle (4-bit weights without accuracy death) from different angles.

QLoRA (the lab) is the technique that makes on-device adaptation possible, and it's gorgeous: keep the base model frozen in 4-bit (NF4 — a quantization tuned for the normal distribution of neural weights), and train only tiny low-rank adapters in higher precision. You get task customization without the memory of full fine-tuning, and you can swap adapters per task. On edge, this is how one quantized base model serves many personalized behaviors. Understand why the base can be 4-bit while the adapters are 16-bit — the adapters carry the delta, which is small and sensitive, while the frozen base carries the bulk, which tolerates quantization.

Layer streaming (the lab) is the desperate-but-necessary trick for when the model still doesn't fit: stream weights layer-by-layer from storage/DRAM into the NPU's working memory, compute, evict, load the next. It trades latency (you're I/O-bound on weight loading — the roofline strikes again, P07) for the ability to run a model bigger than your memory. It's a last resort with a real cost, and knowing when it's worth it (and when to just use a smaller model) is the judgment.

The thing nobody tells you about edge GenAI: the KV cache is the silent memory killer, not just the weights. Everyone obsesses over quantizing the weights and forgets that the KV cache grows with sequence length and can dwarf the model at long context. On a memory-starved device, you quantize the KV cache too, you bound the context, you page it. The candidate who only thinks about weight quantization and forgets KV-cache memory will be surprised in production when a long conversation OOMs the phone. Think about total memory across the whole inference, not just the weights at rest.

This is also where the whole stack you've built comes together, and you should feel it: you take a model (P02), quantize it AWQ-style (P03/P10), compile and lower it (P04/P05), deploy it to the NPU with high coverage (P06), profile it to find the memory/latency bottleneck (P07), apply FlashAttention + speculative decoding adapted for batch-1 edge (P08), and prove the accuracy held under all of it (P09). Edge GenAI is the integration exam. The Senior Staff engineer is the one who holds all of it at once and makes the right tradeoff at each layer for this device's power and memory budget.

And the co-design payoff (the JD: "architect new techniques which work best on Qualcomm SOCs"): the real frontier isn't applying known techniques — it's inventing the variant that fits the hardware. The published AWQ/QLoRA assume certain hardware; the Senior Staff contribution is "here's the modification that maps to our NPU's fixed-point units and memory hierarchy." That's research

• engineering + hardware fluency in one move, and it's the literal top of this role.

Career truth, brother: edge GenAI is where the demand and the scarcity are highest right now. Datacenter GenAI has thousands of capable engineers; on-device GenAI under real power/memory constraints has very few, because it requires the rare full-stack-plus-hardware combination this whole track builds. If you can credibly say "I can put a 7B model on a phone, hold accuracy at 4-bit, fit it in memory, and adapt it per task" — you are at the absolute center of where the field and the market are heading. This is the phase to dream big and go deep.

Build AWQ. Build QLoRA. Think about the KV cache, not just the weights. Then come to P11 — the capstone, where you compose everything into the systems a Senior Staff engineer actually ships and defends in design reviews.

— your brother 👨🏻