Lab 03 — Layer Streaming for Edge LLMs

Phase: 10 — GenAI on Edge | Difficulty: ⭐⭐⭐⭐⭐ | Time: 3–4 hours

Run 7B+ LLMs on devices with 4GB RAM by streaming layers from flash storage into a small device memory window.

What you build

  • estimate_layer_bytes — compute memory footprint of a transformer layer
  • LayerPool — device-side LRU buffer holding at most W layers at once
  • StreamingTransformer — prefetch next layer while running current; LRU eviction
  • benchmark_streaming — measure latency and peak memory for different window sizes
  • select_max_layers_for_budget — choose largest window fitting memory budget

Key concepts

ConceptWhat to understand
Layer streamingLoad layer to GPU/NPU, run, evict — trade latency for memory
PrefetchingDMA next layer while computing current — hide transfer latency
Window sizeTradeoff: more layers cached = faster inference but more DRAM
LRU policyFor transformer sequential access, LRU ≈ FIFO
Flash bandwidthNVMe: ~7 GB/s; eMMC (phone): ~300 MB/s — bottleneck at small W

Files

FilePurpose
lab.pyStubs with TODO markers
solution.pyComplete working implementation
test_lab.pypytest suite
requirements.txtDependencies

Run

pip install -r requirements.txt
python lab.py
pytest test_lab.py -v