What Is an LLM Engineer?

Phase 0 · Document 00 · Orientation Prev: Phase 0 Index · Next: 01 — Roles and Career Paths

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

"LLM engineer" is not one job — it's a spectrum of specializations that mostly did not exist five years ago, every one of which is being hired for right now. If you don't know the landscape, you study the wrong things: you grind transformer math when the job wants RAG and evals, or you learn prompt tricks when the role needs GPU serving. This document maps the terrain so the rest of the curriculum has a destination. It answers: which engineer am I becoming, what do they actually do, and which phases should I prioritize?

2. Core Concept

Plain-English primer (the systems these roles touch — one line each)

This is the first page of the whole curriculum, and the roles below mention systems you haven't met yet. Here's each in one plain line so the role descriptions make sense. Every one gets a full from-zero phase later — this is just a map of the territory.

• LLM (large language model) — the AI that reads and writes text by predicting one token (≈ ¾ word) at a time. The thing all these roles build around. → Phase 1
• API vs self-hosted — you can call a model over the internet (an API, like OpenAI/Anthropic) or run the model's downloaded files on your own machines (self-hosted). → Phase 5
• Inference / serving — "inference" = running a trained model to get answers; "serving" = doing that reliably for many users. → Phase 7
• RAG (retrieval-augmented generation) — fetch relevant documents and paste them into the prompt so the model can answer from your data instead of guessing. → Phase 9
• Embeddings — turning text into a list of numbers so you can measure which texts are similar (the engine behind search/RAG). → Phase 9
• Agent / tool calling — letting the model request actions (call a function, search the web, run code) in a loop; your code actually executes them. → Phase 10
• Gateway / proxy / router — a service that sits in front of many models and handles keys, budgets, routing, and fallbacks (think "load balancer for LLMs"). → Phase 8
• Eval (evaluation) — measuring quality/cost/latency on your task with a test set, instead of trusting leaderboards. → Phase 12
• Inference engines (vLLM, etc.) — specialized software that runs open models fast on GPUs. → Phase 7
• KV cache / quantization — two efficiency ideas: the KV cache is the running memory that limits how many users you can serve; quantization shrinks a model to fit smaller hardware. → Phase 2.06 / Phase 1.06
• Fine-tuning / LoRA — further-training a model on your own data to specialize it. → Phase 13

Don't memorize these now — just know they're systems different roles own. The role map below tells you which roles touch which.

There is a value-chain of LLM roles, from "uses a model" to "builds the infrastructure models run on." Each role owns different systems, optimizes different metrics, and is interviewed on different topics.

The role spectrum

The deep per-role breakdown (responsibilities, skills, tools, metrics, example tasks, interview bar, portfolio, how to train, startup relevance) is in 01 — Roles and Career Paths.

The two organizing axes

1. Application ↔ Infrastructure: do you build features (app, RAG, agents) or the systems features run on (gateway, serving, infra)?
2. Build ↔ Evaluate ↔ Operate: are you creating capability, measuring it, or keeping it reliable in production?

Almost every role is a point on these two axes. Knowing your point tells you which phases of this curriculum are core vs optional.

3. Mental Model

                 builds FEATURES                         builds INFRASTRUCTURE
   ┌──────────────────────────────────────┐   ┌────────────────────────────────────────┐
   │ AI Application Eng · RAG Eng          │   │ LLM Platform Eng · Inference Eng         │
   │ Agent Eng · AI Coding Tools Eng       │   │ AI Infra Eng · ML Eng                    │
   └──────────────────────────────────────┘   └────────────────────────────────────────┘
                 measures it                                keeps it safe/reliable
   ┌──────────────────────────────────────┐   ┌────────────────────────────────────────┐
   │ Model Evaluation Engineer             │   │ AI Safety/Governance Engineer            │
   └──────────────────────────────────────┘   └────────────────────────────────────────┘

   spanning all of it →  Applied AI Architect  →  Startup CTO

   EVERY role is judged on the same three numbers: QUALITY · COST · LATENCY

4. Hitchhiker's Guide

What to figure out first: which point on the two axes you're aiming for. That decision orders the whole curriculum for you.

What to ignore at first: job titles. Titles are inconsistent across companies ("AI Engineer" can mean any of five roles). Read the responsibilities and metrics, not the title.

What misleads beginners:

• Thinking "LLM engineer" = "knows transformer math." Most hiring is application/platform work where you rarely touch the math.
• Thinking prompt engineering is a durable standalone career — it's merging into AI Application Engineering.
• Believing you must operate GPUs to be an LLM engineer — most roles never do.

How experienced people reason: they pick a first role that builds on their existing strengths (a backend engineer → AI Application or Platform), ship a portfolio project for it, then specialize toward inference/RAG/agents based on what they enjoyed.

What matters in production / careers: demonstrated, shipped artifacts (a working RAG system, a gateway, a benchmark report) beat certifications and beat being a generalist who's read everything but built nothing.

How to verify your target fit: read 3 real job descriptions for the role, list the systems and metrics they name, and check them against the per-role cards in 01. If you can't see yourself owning those metrics, pick a different point.

Questions to ask when evaluating a role/team: What systems will I own? What metrics am I accountable for? Do I operate GPUs? Is there an eval culture? Where does cost responsibility sit?

What silently derails people: studying breadth-first forever without shipping; chasing the most "advanced" role (inference) before mastering the foundational one (application/platform); ignoring cost/latency because they're "ops problems" — they're everyone's problems here.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• "LLM engineer" is a family of roles, not one job — titles are inconsistent; read responsibilities and metrics.
• Most LLM engineering hiring is application/platform work that builds on normal software skills and needs no GPU operations.
• Prompt engineering as a standalone role is consolidating into AI Application Engineering.
• Every role is judged on quality, cost, and latency — these three numbers follow you everywhere.
• Portfolios beat certifications in this space; a shipped artifact is the strongest signal.
• The fastest entry path for a strong software engineer is AI Application or LLM Platform Engineer, then specialize.
• Inference/infra roles require GPU and systems depth and are usually not the right first target.

9. Observations from Real Systems

• OpenRouter-style gateways are staffed by Platform + Inference engineers (routing, metering, multi-provider reliability).
• Cursor-style AI IDEs need AI Coding Tools + Application + RAG engineers (indexing, context, patch apply, model routing).
• Enterprise internal gateways lean on Platform + Safety/Governance + ML engineers (budgets, audit, fine-tunes).
• RAG platforms are RAG + Application + Eval engineers (retrieval quality, grounding, regression evals).
• Agent platforms need Agent + Application + Inference engineers (loop safety, tool reliability, latency).
• Early-stage AI startups collapse all roles into the Applied AI Architect / CTO, who must reason across the whole stack (the reason this curriculum is end-to-end).

10. Common Misconceptions

11. Engineering Decision Framework

Which role should I target FIRST?

Background = backend/full-stack, want fastest entry?
  → AI Application Engineer. (Phases 1,3,4,5,9,10,12,13)

Like building platforms/infra abstractions others use?
  → LLM Platform Engineer. (Phases 1,5,7,8,12,15)

Love systems/perf, comfortable with GPUs/CUDA?
  → LLM Inference Engineer (later specialization). (Phases 2,5,6,7)

Drawn to search/knowledge products?
  → RAG Engineer. (Phases 1,9,12,13)

Drawn to autonomy/automation?
  → Agent Engineer. (Phases 1,10,12,14)

Care most about correctness/trust?
  → Model Evaluation Engineer. (Phases 12,13,14)

Building a company?
  → Applied AI Architect → Startup CTO (all phases; prioritize 5,8,9,12,15,16).

12. Hands-On Lab

Goal

Produce a personal role-targeting plan grounded in real job postings — so the rest of the curriculum has a concrete destination.

Prerequisites

• None. A text editor and access to a job board.

Setup

Create role-plan.md in your notes repo.

Steps

1. Pick two candidate roles from Section 2 that fit your background.
2. Find 3 real job postings for each. For every posting, extract: systems owned, metrics named, required tools, and whether GPUs are involved.
3. Build a comparison table of the two roles from this real data.
4. Map each role to the curriculum phases (use Section 11).
5. Choose one primary role and one portfolio project (Section 11 table) to build by the end of the curriculum.
6. Write a 5-line justification.

Expected output

A role-plan.md containing two role profiles built from real postings, a chosen target, a portfolio project, and a phase priority order.

Debugging tips

• Postings vague? Look for the metrics and systems lines; ignore buzzwords.
• Two roles look identical? You picked adjacent points on one axis — pick one farther apart to clarify the choice.

Extension task

Add a "skills gap" column: for your chosen role, list which curriculum phases close which gaps.

Production extension

Revisit role-plan.md after each phase and check off demonstrated skills — turning it into a living portfolio readiness tracker.

What to measure

Number of distinct systems/metrics you can already speak to vs. those you still need; phase coverage of your target role.

Deliverables

• role-plan.md with two real-posting-based role profiles, a chosen target, and a portfolio project.

13. Verification Questions

Basic

1. Name three LLM engineering roles and the systems each owns.
2. Which roles typically require operating GPUs, and which don't?
3. What three metrics is every LLM role judged on?

Applied 4. A backend engineer wants the fastest entry into LLM work. Which role and why? 5. Map the RAG Engineer role to four curriculum phases.

Debugging 6. You've studied for months but no interviews. What's the most likely missing signal, per this document? 7. A job titled "AI Engineer" — how do you figure out which actual role it is?

System design 8. Staffing a new Cursor-style IDE startup: which roles do you hire first and why?

Startup / product 9. As a solo technical founder, which phases of this curriculum do you prioritize, and what's the one portfolio artifact that doubles as your product demo?

14. Takeaways

1. "LLM engineer" is a family of roles along two axes: application↔infrastructure and build↔evaluate↔operate.
2. Read responsibilities and metrics, not titles — titles are inconsistent.
3. Most hiring is application/platform work that needs no GPU operations.
4. Start with AI Application or LLM Platform Engineer, then specialize into inference/RAG/agents.
5. Every role is judged on quality, cost, and latency.
6. Ship a portfolio project for your target role — it beats any certification.

15. Artifact Checklist

• role-plan.md with two real-posting-based role profiles.
• Chosen target role + justification.
• Portfolio project selected for that role.
• Phase priority order mapped to your role.
• Skills-gap table (role → phases that close gaps).
• Notes: the two-axis role map in your own words.

Next: 01 — Roles and Career Paths