Knowledge 07 — CV & Video Pipelines

The JD bullets: "Multi-model workflows (e.g., detection + tracking + recognition)," "Video pipeline choices (e.g., FFMPEG vs GStreamer)," "real-time AI systems, including computer vision, video, and analytics." JD2 is not LLM-only — a huge fraction of accelerator deployments are real-time vision: cameras, video analytics, smart-city, retail, industrial inspection. This module covers the cascade-pipeline architecture, hardware video decode, the FFMPEG-vs-GStreamer decision, and how to think about a 30-camera real-time system. The performance principles (roofline, batching, profiling) all transfer; the workload shape is different.

Table of Contents

• 1. Why video is its own discipline
• 2. The end-to-end video AI pipeline
• 3. Decode is the hidden bottleneck: NVDEC and hardware codecs
• 4. FFMPEG vs GStreamer (the JD's explicit question)
• 5. The cascade: detection → tracking → recognition
• 6. Batching frames and multiplexing streams
• 7. DeepStream and the production stacks
• 8. Keeping data on the GPU: the zero-copy principle
• 9. Sizing a multi-stream deployment
• 10. Worked example: 50-camera smart-city pipeline
• 11. References

1. Why video is its own discipline

An LLM request is text in, text out. A video pipeline is a continuous, real-time stream of frames that must each pass through several models within a frame budget (33 ms at 30 fps), across many concurrent streams, forever. The differences that matter:

• Throughput is fixed by reality: 30 fps × N cameras frames/second arrive whether you're ready or not. Fall behind and you drop frames (or latency grows unboundedly). It's a hard real-time constraint, not a best-effort one.
• The model is rarely the bottleneck — decode, color conversion, resize, and memory copies often dominate. Teams that only profile the neural net miss 60% of the cost.
• Multi-model by default: useful vision is a cascade of models (§5), not one.
• Memory movement is everything: a 4K frame is ~25 MB raw; moving it CPU↔GPU per stage per frame per camera saturates PCIe instantly. The whole game is keeping pixels on the GPU (§8).

The reframing for an LLM-centric engineer: in video, the "tokens" are frames, the "model" is a cascade, and the memory wall bites at the pixel-movement layer before it ever reaches the network. Same physics, different surface.

2. The end-to-end video AI pipeline

Every video analytics system is some version of this chain. Know each stage and its cost:

Camera/RTSP ─► DECODE ─► PREPROCESS ─► INFERENCE (cascade) ─► POSTPROCESS ─► TRACK/ANALYTICS ─► OUTPUT
 (H.264/265   (compressed (resize,     (detect→classify→     (NMS, decode  (multi-object      (alerts,
  stream)      → frames)   normalize,   recognize)            boxes)        tracking, rules)   overlay, DB)
                           color cvt)

1. Ingest/decode: receive an H.264/H.265 RTSP stream, decode compressed video into raw frames. Often the #1 cost (§3).
2. Preprocess: resize to model input size, convert color space (NV12→RGB), normalize. Per frame, per model — surprisingly expensive; do it on GPU.
3. Inference: the model cascade (§5).
4. Postprocess: non-max suppression (NMS), bounding-box decode, thresholding.
5. Track/analytics: associate detections across frames (multi-object tracking), apply business rules (line crossing, dwell time, counting).
6. Output: alerts, overlays, metadata to a database/dashboard.

The principal's instinct: profile the whole chain, not just step 3. The neural net might be 8 ms of a 33 ms budget while decode + color-convert + copies eat 20 ms. (Knowledge 06 applies wholesale.)

3. Decode is the hidden bottleneck: NVDEC and hardware codecs

Video arrives compressed (H.264/AVC, H.265/HEVC, AV1). Decoding it to raw frames is computationally heavy. Two ways:

• CPU (software) decode (libavcodec on CPU): flexible, but a CPU core decodes only a handful of 1080p streams, and the decoded frame lands in host memory → must be copied to GPU → PCIe traffic + CPU saturation. Doesn't scale past a few streams.
• Hardware decode (NVDEC): GPUs have dedicated fixed-function decode engines (NVDEC) separate from the CUDA cores. They decode dozens of streams in parallel directly into GPU memory — the frame never touches the CPU or PCIe. This is the only way to do many-stream real-time. (Encode has the analogous NVENC.)

Numbers to know: a single GPU's NVDEC engines can decode on the order of tens of 1080p30 streams (varies by GPU and codec); some GPUs have multiple NVDEC engines. Decode capacity, not GPU FLOPs, frequently caps the stream count — a fact that surprises customers who sized purely on model throughput.

Sizing catch (pure JD2): a customer plans "100 cameras on one GPU because the detector runs at 2000 fps." But the GPU's NVDEC can only decode ~30 1080p streams. Decode is the binding constraint, not inference. Catching that in the sizing meeting is the value-add. Always ask: codec, resolution, fps, and stream count → check it against decode capacity first.

Qualcomm/edge accelerators and SoCs similarly have dedicated video decode blocks — the principle (offload decode to fixed-function hardware, keep frames off the CPU) is universal across vendors.

4. FFMPEG vs GStreamer (the JD's explicit question)

The JD literally names this choice, so have a crisp answer.

The principal's answer: "FFmpeg is a Swiss-army codec tool — perfect for transcoding and batch frame extraction. GStreamer is a streaming pipeline framework — perfect for live, multi-camera, low-latency analytics where you need a dynamic graph of decode→infer→track→output with zero-copy GPU memory. For a real-time 50-camera deployment I'd build on GStreamer (likely via DeepStream); for an offline 'process this archive of video files' job I'd use FFmpeg." That nuance — tool vs framework, batch vs live — is exactly what the JD probes.

(They're not mutually exclusive: GStreamer can use libav elements; many systems use FFmpeg for ingest/transcode and GStreamer/DeepStream for the live inference pipeline.)

5. The cascade: detection → tracking → recognition

The canonical multi-model workflow the JD names. Each stage feeds the next; understanding the data flow is the architecture skill.

1. Detection (e.g., YOLO, RetinaNet, DETR): per frame, find what and where — bounding boxes + classes ("person at (x,y,w,h)", "vehicle at …"). The workhorse; runs every frame.
2. Tracking (e.g., SORT, DeepSORT, ByteTrack): associate detections across frames into persistent tracks ("this is person #7, same as last frame"). Gives temporal identity → enables counting, dwell time, trajectories. Often cheap (Kalman filter + association) but DeepSORT adds a re-ID embedding model.
3. Recognition / classification / re-ID (e.g., face recognition, license-plate OCR, attribute classification): on the cropped region from detection, do fine-grained identification. Runs only on detected objects, not the whole frame → input-dependent, variable load.

Frame ─► [Detector] ─► boxes ─► [Tracker] ─► tracks ─┐
                          │                            ├─► analytics (count, dwell, alerts)
                          └─► crops ─► [Recognizer] ───┘

Why cascade instead of one big model: each stage is specialized and cheap on its narrow job; you run the expensive recognizer only on the few crops that matter, not every pixel. It's the vision analog of a retrieval-then-rerank pipeline — coarse-to-fine to save compute.

The serving challenge: these are different models with different input shapes and variable per-frame call counts (1 detector call, N recognizer calls where N = #objects). Orchestrating them with per-model batching is exactly what Triton ensembles/DeepStream (Knowledge 04 §9) are for.

6. Batching frames and multiplexing streams

The batching economics apply to vision too, with a twist: you batch across streams. Instead of running the detector on one camera's frame at a time (tiny batch, idle GPU), you gather one frame from each of 30 cameras into a batch of 30 and run the detector once. This amortizes weight reads and fills the GPU — the same win as continuous batching, applied to multiplexed video.

• Stream muxing: combine N camera streams into a batched tensor (DeepStream's nvstreammux).
• Batch the detector across streams; demux results back per stream for tracking (tracking is per-stream stateful).
• Trade-off: bigger frame-batches = better GPU efficiency but add latency (wait to collect frames) — the same throughput↔latency dial, bounded here by the frame budget (can't wait longer than ~33 ms at 30 fps).

So the vision serving objective mirrors LLM serving: maximize GPU efficiency (batch across streams) subject to the real-time latency budget. Different units, identical shape of problem.

7. DeepStream and the production stacks

NVIDIA DeepStream is the production framework for GPU video analytics: built on GStreamer, it provides plugins for NVDEC decode, nvstreammux (batch streams), nvinfer (TensorRT inference), nvtracker (multi-object tracking), and sinks — all operating on zero-copy GPU memory (NVMM) so frames never leave the GPU through the whole pipeline. It's how you go from "50 RTSP URLs" to "real-time analytics" without hand-building the plumbing.

Other stacks: NVIDIA Triton (Knowledge 04 §9) for the model-serving layer (DeepStream can call Triton), Intel OpenVINO + DL Streamer (the Intel-world equivalent), and vendor SDKs on edge SoCs (Qualcomm, etc.). The transferable concept: a GPU/accelerator-native streaming pipeline that keeps pixels on-device and batches across streams.

8. Keeping data on the GPU: the zero-copy principle

The dominant performance principle in video, and the most common mistake to catch:

Every CPU↔GPU copy of a frame is expensive. A naive pipeline does: NVDEC → copy frame to CPU → resize on CPU → copy back to GPU → infer → copy result to CPU. That's 2+ PCIe round-trips of ~25 MB per frame per camera — PCIe saturates and the CPU melts at a few streams.

The fix: decode (NVDEC) → preprocess (GPU resize/color-convert) → infer (GPU) → postprocess all in GPU memory, copying only the tiny result metadata (boxes, IDs) back to the CPU at the end. DeepStream's NVMM memory and zero-copy plugins enforce this; in a custom stack you use CUDA-accelerated resize/color (NPP, CV-CUDA, nvJPEG) and keep tensors on-device.

The diagnostic: if a video pipeline's nsys trace (Knowledge 06 §4) is full of big HtoD/DtoH memcpy bars, it's doing CPU round-trips → keep it on the GPU and the stream count multiplies. This is the single highest-leverage fix in most real-world vision deployments, and spotting it fast is principal-level.

9. Sizing a multi-stream deployment

The sizing equation for video differs from LLMs — you balance three capacities, and the smallest binds:

Max streams per accelerator = min(
   decode_capacity,      // NVDEC streams the GPU can decode (codec/res/fps dependent)
   inference_capacity,   // detector_fps / (streams × fps_per_stream), across the cascade
   memory_capacity,      // frame buffers + model weights + intermediate tensors fit in VRAM
   bandwidth/PCIe        // only if you (wrongly) copy frames off-GPU
)

You compute each and take the minimum — and it's frequently decode, not inference, as in §3. Then add latency: each stage's latency must sum within the frame budget for real-time, or you accept buffering/latency for throughput-oriented (non-real-time) analytics.

The deliverable (same shape as Knowledge 08): "On this GPU you can run 30 cameras at 1080p30 — bound by NVDEC decode, not the detector, which has headroom for 80. To reach 100 cameras you need a GPU with more NVDEC engines or a second GPU, not a faster model." That sentence is the JD's "hardware sizing and architecture discussions" for vision.

10. Worked example: 50-camera smart-city pipeline

Customer: 50 cameras, 1080p H.265 @ 25 fps, need vehicle + person detection, tracking, and license-plate recognition, real-time alerts.

1. Decode: 50 × 1080p25 H.265. Check the GPU's NVDEC HEVC capacity — say one GPU does ~35 1080p25 HEVC streams → need 2 GPUs (or a multi-NVDEC GPU) just for decode. This is the binding constraint; surface it first.
2. Preprocess: GPU resize + NV12→RGB via DeepStream/CV-CUDA, zero-copy (§8). No CPU round-trips.
3. Inference cascade (§5): YOLO detector batched across streams (nvstreammux batch=50), TensorRT FP16/INT8 (Knowledge 02–03). Tracker (nvtracker, ByteTrack) per stream. Plate recognizer runs only on vehicle crops → variable load; size for peak vehicle density.
4. Batching: gather one frame per camera → detector batch 50 → demux → per-stream tracking. Latency budget 40 ms/frame (25 fps); confirm decode+preprocess+infer+track sums within it.
5. Serving: DeepStream (GStreamer-based) for the pipeline; TensorRT engines for the models; optionally Triton for the recognizer ensemble.
6. Sizing output: "2 GPUs: decode-bound at ~35 streams/GPU; detector has compute headroom; plate recognizer is the variable cost — size for peak traffic. Real-time met at 40 ms budget with zero-copy. FFmpeg only for the archival transcode side job; GStreamer/DeepStream for the live path."

Every JD2 video bullet appears here: multi-model cascade, FFmpeg-vs-GStreamer, decode-bound sizing, real-time constraint, accelerator deployment. Practice articulating it.

11. References

• NVIDIA DeepStream SDK documentation; NVIDIA Video Codec SDK (NVDEC/NVENC) docs.
• GStreamer documentation and the gst-nvinfer/nvstreammux/nvtracker DeepStream plugin guides; FFmpeg documentation and hardware-acceleration guide.
• Redmon et al., YOLO; Carion et al., DETR — detection. Bewley et al., SORT; Wojke et al., DeepSORT; Zhang et al., ByteTrack — tracking.
• NVIDIA CV-CUDA, NPP, nvJPEG — GPU preprocessing (zero-copy, §8).
• Intel OpenVINO DL Streamer; MLPerf mobile/edge vision benchmarks.

➡ Next: Knowledge 08 — Capacity Sizing & Solutions Architecture — turning all of this into a customer recommendation and a business case.