Feed-Forward Layers (FFN / MLP)

Phase 2 · Document 03 · Transformer Foundations Prev: 02 — Attention and Self-Attention · Next: 04 — LayerNorm and Residuals

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Attention gets the attention, but the feed-forward network (FFN) is where most of a model's parameters, compute, and stored "knowledge" actually live — typically two-thirds of the weights. That single fact explains a lot of your job: why quantization focuses on these big matrices, why MoE (08) targets the FFN to scale parameters cheaply, and why a model's factual recall correlates with FFN capacity. If attention is "which tokens matter," the FFN is "now think hard about each one." Understanding it demystifies model size, memory, and the entire MoE phenomenon.

2. Core Concept

Plain-English primer (matrix multiply + activation, from zero)

The FFN is the simplest part of a transformer once two ideas are clear (both build on 2.00's math primer).

• A "linear layer" is one matrix multiply (y = x @ W): take a vector, multiply by a learned matrix, get a new vector. If W has shape (in, out), it maps an in-sized vector to an out-sized one. Purpose: reshape/expand information. Example: a (4 → 8) layer "widens" a 4-number vector into 8 numbers; an (8 → 4) layer "narrows" it back.
• An activation is a simple non-linear function applied to each number (e.g. ReLU = "keep positives, zero out negatives"; GELU/SiLU = smooth versions). Why it's needed: stacking pure matrix multiplies collapses into one matrix (no extra power); inserting a non-linear bend between them lets the network learn curved, complex patterns instead of only straight-line ones. Analogy: without an activation, ten linear layers = one linear layer; the activation is the "kink" that makes depth meaningful.

So an FFN is just: widen → bend → narrow:

def relu(v): return [max(0.0, x) for x in v]
def matmul(x, W):                                   # x:(in,)  W:(in,out)  -> (out,)
    return [sum(x[i]*W[i][j] for i in range(len(x))) for j in range(len(W[0]))]

x   = [1.0, -2.0]                                   # a 2-dim token vector
W1  = [[0.5,0.1,0.9,0.2],[0.3,0.7,0.4,0.6]]         # (2 -> 4): EXPAND (~4x wider)
W2  = [[0.2,0.1],[0.4,0.3],[0.1,0.5],[0.6,0.2]]     # (4 -> 2): PROJECT back
hidden = relu(matmul(x, W1))                        # widen, then bend (negatives -> 0)
out    = matmul(hidden, W2)                         # narrow back to 2-dim
# out is the token's vector after "thinking" — same size in, same size out

Key facts that follow: the FFN runs on each token independently (no token-mixing — that's attention's job); the wide middle (d_ff, ~4× hidden) holds most parameters, so it's where most of the model's memory and knowledge sit — and what quantization (Phase 1.06) and MoE (2.08) both target. Modern models use a gated activation called SwiGLU (a fancier "bend" with an extra gate matrix) — you'll see gate_proj/up_proj/down_proj in configs.

Plain English

After attention has each token gather information from its context, the FFN processes each token independently through a small neural network: expand it into a much wider space, apply a nonlinearity, then compress it back. This is where the model does its per-token "computation" and where learned facts/patterns are stored.

Technical depth

A classic transformer FFN is a 2-layer MLP applied position-wise (same weights for every token, no mixing between tokens):

FFN(x) = W2 · activation(W1 · x + b1) + b2
   x:  hidden_dim           (e.g. 4096)
   W1: hidden_dim → d_ff    (expand, often d_ff ≈ 4× hidden)
   W2: d_ff → hidden_dim    (project back)

• Expansion factor: d_ff is usually ~4× hidden_dim (e.g. 4096 → 11008/14336). The wide intermediate space is where capacity lives.
• Activation: older models used ReLU/GELU; modern ones use SwiGLU (a gated variant), which adds a third matrix and tends to improve quality — you'll see gate_proj, up_proj, down_proj in Llama-style configs.
• Position-wise & independent: unlike attention, the FFN does not mix tokens. Each token is transformed on its own — which is exactly why it parallelizes trivially and why MoE can route different tokens to different experts.

Where parameters and compute go

For a typical dense LLM, the FFN holds ~2/3 of the parameters (attention projections hold most of the rest). Because d_ff is large, the FFN also dominates the FLOPs per token. Two consequences:

1. Quantization (Phase 1.06) cares most about these big matrices — shrinking them is most of the memory win.
2. MoE replaces the single FFN with many expert FFNs and activates only a few per token: you multiply total parameters (capacity/knowledge) without multiplying active compute. This is the entire mechanism behind "26B-A4B" style models (Phase 1.02).

"Knowledge lives here"

Interpretability research treats FFN layers as key-value memories: the first matrix detects patterns, the second writes associated information back into the residual stream. This is the intuition for why bigger/ wider FFNs (and more experts) tend to store more facts.

3. Mental Model

ATTENTION = "gather": each token collects info from other tokens (mixes across positions)
FFN       = "think":  each token is transformed ALONE (no mixing)

FFN(x):   hidden ──W1(expand ~4×)──► wide ──activation──► wide ──W2(project)──► hidden
                                   (where capacity / knowledge lives)

PARAMETER BUDGET (typical dense LLM):
   FFN ≈ 2/3 of weights   ·   attention ≈ 1/3
   → quantization & MoE both target the FFN because that's where the mass is.

MoE = swap ONE FFN for MANY experts; a router picks a few per token
   → total params ↑↑ (knowledge), active compute ~flat (speed).

4. Hitchhiker's Guide

What to understand first: the FFN is a per-token expand→activate→project, it holds most parameters/compute, and it does not mix tokens.

What to ignore at first: SwiGLU gating math and exact activation choices — know that modern models use gated activations and move on.

What misleads beginners:

• Thinking attention is "the whole model" — most weights and FLOPs are in the FFN.
• Assuming MoE makes a model "smaller" — it makes active compute smaller while total parameters/memory grow.
• Believing wider FFN automatically = better — capacity helps only with the data/training to fill it.

How experts reason: they read the config (intermediate_size/d_ff, gate/up/down projections, num_experts) to estimate where memory goes, and they recognize MoE as "FFN sharding by router." When memory is tight, they know quantizing the FFN matrices yields the most savings.

What matters in production: FFN width and (for MoE) expert count/active-experts drive both memory (total) and speed (active); quantization of these matrices is the main local-inference lever.

How to verify: in a model config, identify the FFN matrices and compute their parameter share; for MoE, read num_local_experts and num_experts_per_tok and compute total vs active params.

Questions to ask: What's the FFN expansion factor? Gated activation (SwiGLU)? Dense or MoE — how many experts, how many active? What fraction of params is FFN?

What silently gets expensive: MoE total memory (all experts resident) catching teams who budgeted for "active" size; assuming a small active-param MoE fits on small hardware.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The FFN is a position-wise 2-layer MLP: expand (~4×), activate, project back.
• It typically holds ~2/3 of a dense model's parameters and dominates per-token FLOPs.
• The FFN does not mix tokens — each token is transformed independently (attention does the mixing).
• Modern LLMs use gated activations (SwiGLU) — hence gate_proj/up_proj/down_proj.
• Knowledge/facts are largely stored in FFN weights (key-value-memory interpretation).
• MoE replaces the FFN with multiple experts, routing a few per token: total params ↑, active compute ~flat.
• Quantization targets the big FFN matrices for most of its memory savings.
• For MoE, memory tracks total experts; speed tracks active experts.

9. Observations from Real Systems

• Llama/Qwen/Mistral configs expose intermediate_size and SwiGLU gate/up/down projections — the FFN in the wild.
• Mixtral, Qwen-MoE, DeepSeek-MoE replace the FFN with experts; their cards report total vs active params (the Phase 1.02 distinction).
• Quantization toolkits (GPTQ/AWQ/GGUF) spend most bits on FFN matrices because that's where the mass is.
• vLLM/llama.cpp must hold all MoE experts in memory even though only a few activate per token — the classic MoE memory surprise.
• Interpretability tools (e.g. logit lens, activation patching) localize many factual associations to FFN layers.

10. Common Misconceptions

11. Engineering Decision Framework

Estimating memory:
  FFN ≈ 2/3 of dense weights → quantizing FFN matrices = most of the savings.

Dense vs MoE (FFN lens):
  Need more knowledge/quality, have memory, want speed?
     → MoE: many experts (total memory) but few active (compute) — verify TOTAL fits.
  Memory-constrained?
     → smaller dense model; MoE total may not fit even if "active" is small.

Reading a config quickly:
  hidden_size, intermediate_size → FFN width/share.
  gate_proj/up_proj/down_proj    → SwiGLU.
  num_local_experts, num_experts_per_tok → MoE total vs active.

12. Hands-On Lab

Goal

Implement a transformer FFN (with and without SwiGLU), and measure the FFN's share of a real model's parameters.

Prerequisites

• Python 3.10+, numpy; optional transformers for the param-share measurement.

Setup

pip install numpy transformers torch

Steps

import numpy as np

def gelu(x): return 0.5*x*(1+np.tanh(np.sqrt(2/np.pi)*(x+0.044715*x**3)))

def ffn(x, W1, b1, W2, b2):                       # classic 2-layer FFN
    return gelu(x @ W1 + b1) @ W2 + b2

hidden, d_ff = 16, 64                              # 4x expansion
x = np.random.randn(3, hidden)                     # 3 tokens
W1, b1 = np.random.randn(hidden, d_ff)*0.1, np.zeros(d_ff)
W2, b2 = np.random.randn(d_ff, hidden)*0.1, np.zeros(hidden)
print("FFN out shape:", ffn(x, W1, b1, W2, b2).shape)  # (3, 16): per-token, same shape

# Measure FFN parameter share of a real small model
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
ffn_params = sum(p.numel() for n,p in m.named_parameters() if "mlp" in n)
total = sum(p.numel() for p in m.parameters())
print(f"FFN/MLP share: {ffn_params/total:.1%}")

Expected output

• FFN output keeps the (tokens, hidden) shape — confirming per-token processing.
• A printed FFN parameter share (a large fraction of the model).

Debugging tips

• Param name filter empty? Different models name FFN modules differently (mlp, feed_forward); inspect named_parameters().
• SwiGLU extension shapes off? Gated variant needs a third (gate) matrix; output of gate multiplies the up-projection.

Extension task

Implement SwiGLU: down( (up(x)) * silu(gate(x)) ); compare parameter count to the classic FFN at equal d_ff.

Production extension

For an MoE config, compute total vs active FFN parameters (num_local_experts × per-expert vs num_experts_per_tok × per-expert) and state the memory-vs-speed implication.

What to measure

FFN parameter share; output shape (per-token); SwiGLU vs classic param delta; MoE total vs active params.

Deliverables

• Working FFN (+ SwiGLU extension).
• Measured FFN parameter share for one model.
• A note: why MoE memory tracks total experts.

13. Verification Questions

Basic

1. What does the FFN do, and how does it differ from attention?
2. Roughly what fraction of a dense model's parameters is the FFN?
3. What is SwiGLU and why do modern models use it?

Applied 4. Why does quantization focus on FFN matrices? 5. In a 26B-A4B MoE, what's resident in memory vs computed per token, and why?

Debugging 6. An MoE "4B active" model OOMs on a 16 GB GPU. Explain. 7. You widened d_ff 2× and quality didn't improve. Give a plausible reason.

System design 8. Choose dense vs MoE for a knowledge-heavy assistant on a fixed memory budget; justify via the FFN.

Startup / product 9. Pitch why an MoE model can give you frontier-ish quality at lower serving cost — and the one infrastructure caveat (total memory).

14. Takeaways

1. The FFN is a per-token expand→activate→project MLP — no token mixing.
2. It holds ~2/3 of parameters and most per-token compute; knowledge largely lives here.
3. Modern LLMs use gated activations (SwiGLU).
4. MoE = FFN replaced by routed experts: total params ↑ (quality), active compute ~flat (speed).
5. Quantization targets FFN matrices for the biggest memory savings.
6. For MoE, memory tracks total, speed tracks active — budget for total.

15. Artifact Checklist

• Code: FFN implementation (+ SwiGLU).
• Measurement: FFN parameter share of a real model.
• MoE note: total vs active param computation for one MoE config.
• Notes: "gather vs think" (attention vs FFN) and where knowledge lives.
• Decision record: dense vs MoE for one workload + memory budget.
• Diagram: the expand→activate→project FFN and MoE routing.

Next: 04 — LayerNorm and Residuals