👨🏻 Brother Talk — Phase 01, Off the Record

Framework internals — the phase that decides whether you're a framework user or a framework engineer. At Qualcomm Senior Staff level, only the second one gets hired.

Brother, let me tell you the single thing that separates the people who pass this interview loop from the people who don't, because it took me too long to see it: almost everyone can use PyTorch, and almost nobody can explain it. They can write loss.backward() a thousand times and have no idea what happens after they hit enter. And the Qualcomm Senior Staff role is specifically about what happens after you hit enter — the dispatcher, the autograd graph, the kernel that actually runs. You are being hired to work below the API that everyone else lives on top of. So this phase isn't "review PyTorch." It's "stop being a tourist in the framework you've used for years."

Here's the mental reframe that unlocks it. PyTorch is not one thing — it's three layers pretending to be one: the Python eager frontend (what you type), the C++ dispatcher (the c10::Dispatcher that routes every op by dispatch key — CPU, CUDA, Autograd, Functionalize, Meta), and the backend kernels (the code that does the math). Every bug you'll ever debug at this job lives at a boundary between these layers: a custom op that works eager but breaks under torch.compile (boundary: dispatcher ↔ Dynamo), a quantized model that NaNs after a graph pass (boundary: Autograd ↔ Functionalize), a TF model that gives different answers in graph mode (boundary: eager ↔ tf.function tracing). If you can name the layer a bug lives in, you've already done 80% of the debugging. Build the autograd engine by hand in Lab 01 and this stops being abstract — you'll feel the graph being constructed on the forward pass and walked on the backward.

The thing nobody tells you about the autograd engine: it's simpler than you fear and more beautiful than you expect. It's just a DAG built during forward (each op records how to compute its gradient), walked in reverse-topological order during backward, accumulating gradients at the leaves. That's it. Once you've written Engine::execute() in miniature — even in pure Python — you'll never again be intimidated by "how does PyTorch know the gradient?" And you'll be the person in the room who can explain retain_graph, create_graph for double-backward, and why AccumulateGrad exists, while everyone else mumbles. That clarity is worth the week it takes.

Now the custom-operator stuff (Labs 02), because this is where the Qualcomm job actually lives. When you write a custom op — say, an NPU-accelerated kernel — it has to behave correctly under every dispatch key: CPU, CUDA, autograd (does it have a gradient?), functionalization (is it functional or does it mutate?), torch.compile (can Dynamo trace through it?). The classic disaster: someone registers a custom op that works in eager, ships it, and three months later it silently produces garbage under torch.compile because they never registered a Meta kernel or a functionalization rule. That is the bug you're being hired to prevent. The C++ operator lab feels like plumbing; it's actually the core skill — writing ops that are safe under all the ways PyTorch can call them. (And yes, the lab may skip on a machine without a C++ toolchain — that's expected; read the code, understand the TORCH_LIBRARY schema, build it for real when you have a compiler.)

On TensorFlow: don't skip it because PyTorch won. Qualcomm ships both, and the tf.function tracing model is a gold mine of interview questions precisely because it's subtle. The trap everyone falls into: Python side-effects inside a tf.function run once (at trace time), not every call, and retracing triggers on new input signatures. The "model gives different results in graph mode" bug is almost always a Python int where a tf.Variable belonged, or a closure capturing a stale value. Understand tracing-vs-execution once and you'll debug a class of bugs that mystifies whole teams.

A word on numerical precision, because it's the quiet thread through this entire role: FP32 vs FP16 vs BF16 is not just "smaller is faster." It's where you accumulate, what cancels, and when you get NaN. BF16 has FP32's exponent range with fewer mantissa bits — great for training stability, lossy for precision. The day you debug a model that's fine in FP32 and NaNs in FP16, you'll be grateful you understand catastrophic cancellation and why accumulation dtype matters. This thread runs straight into quantization (P03) and accuracy (P09) — it's the same fight at lower and lower precision.

Career truth, brother: this phase is the foundation that makes every later phase possible, and it's also the most legible signal of seniority in an interview. When you can trace loss.backward() to the kernel, explain dispatch keys, and write a custom op that's safe under compile — the interviewer knows you've operated below the API. That credibility carries the whole loop. Spend the time here. Build the autograd engine by hand. It's the difference between "I know PyTorch" and "I know how PyTorch works," and only the second sentence gets the Senior Staff offer.

Build the labs. Trace one op end to end. Then come to P02, where we go up a level to the architectures — and you'll understand them more deeply because you know what they compile down to.

— your brother 👨🏻