Phase 15 — Embodied AI & Robotics: RL, Planning & VLA

The phase where the model stops talking and starts acting. Every phase before this produced a token, a string, a tool call — something that lives in a buffer and can be regenerated. An embodied agent perceives a physical world, takes an action in it, and the world changes irreversibly. This is the key differentiator for a Robotics Research Center role: the JD asks for reinforcement learning, simulation, embodied AI, and agentic planning — and this phase builds the control core of all of it from first principles.

Why this phase exists

The job is at a Robotics Research Center. The "nice-to-haves" — reinforcement learning, simulation environments, embodied AI — are the things that separate a strong LLM engineer from someone who can contribute to robots. And the frontier of robotics in 2025 is exactly where this curriculum has been heading: Vision-Language-Action models take the VLMs of Phase 04, the agentic planning of Phase 12, the alignment RL of Phase 07, and the verification of Phase 14, and point them at a gripper. Actions become tokens an LLM decodes.

But you cannot reason about RT-2 or OpenVLA without the substrate underneath: the MDP that formalizes a decision problem, the Bellman equations that define optimality, the split between planning (you have the world model) and learning (you don't), the exploration that makes learning possible, the reward design that makes it safe, and the task planning that sequences skills into goals. This phase gives you that substrate as muscle memory — and then shows you exactly where the modern stack plugs in. Three facts reframe everything: in an embodied system latency is a safety property, mistakes are irreversible (no regenerate), and the world is partially observed. Internalize those and you think like a roboticist, not a chatbot engineer.

Concept map

                 ┌─────────────────────────────────────────────┐
                 │  An agent that ACTS in a world (the loop)    │
                 │  perceive → state → plan → act → (world)     │
                 └─────────────────────────────────────────────┘
                       │                │                 │
            ┌──────────┘        ┌───────┘         ┌───────┘
            ▼                   ▼                 ▼
      THE MDP              CONTROL            THE MODERN STACK
   (S,A,P,R,γ)        ┌──────┴───────┐     ┌──────┴────────┐
   Bellman eqs        PLANNING    LEARNING  imitation /  task planning
   V*(s)=max_a        value iter   Q-learn    BC          STRIPS / PDDL
   [r+γV(s')]         (have P)    (sample)    │             │
       │              MPC,MuZero  DQN,SARSA   VLA          SayCan / CaP
       │                  │        PPO=P07   (RT-2,        (LLM planner)
       │                  │          │       OpenVLA,        │
       └────────┬─────────┴────┬─────┘       Octo)──┐        │
                ▼              ▼              actions │   verify (P14)
         exploration vs    reward design     as tokens│        │
         exploitation      reward hacking      │      │        ▼
         (ε-greedy)        shaping             diffusion   sim-to-real
                                               policies    domain rand.
                                                           (Isaac/MuJoCo)
                                                           latency = SAFETY

The lab

The lab is a runnable, test-verified miniature — see the lab standard. Run it red (pytest test_lab.py), make it green, then read solution.py. The two soul tests: Q-learning converges so the greedy policy reaches the goal in the optimal number of steps, and that learned policy matches value iteration's optimum (cross-checked on value, since optimal policies are not unique).

Integrated scenario ideas

• Plan-then-learn a pick-and-place: use the STRIPS planner to sequence pick → move → place at the symbolic level, then imagine each operator backed by a learned (Q-learning / VLA) skill — the hierarchical pattern every real manipulation stack uses.
• Ground an LLM planner: have an LLM (or a scripted stub) propose an action sequence, and use the lab's execute_plan as the verifier that rejects infeasible plans — SayCan/Code-as-Policies grounded by a checker (a direct tie to Phase 14).
• Defend "VLA over from-scratch RL": explain why borrowing a VLM's web-scale grounding (RT-2) lets a robot generalize to objects it never saw in training, where tabular/DQN RL cannot.
• Size the safety budget: argue why the learned policy runs under a fast safety filter and why a 2-second LLM planner cannot sit in the kHz control loop — latency as a safety property.
• Cross the reality gap: design a domain-randomization schedule (friction, latency, lighting) for a sim-trained policy and justify why more variation beats more accuracy.

Deliverables checklist

• lab.py passes pytest test_lab.py -v (and LAB_MODULE=solution does too).
• You can write the Bellman optimality equation and the Q-learning update from memory.
• You can explain why Q-learning is off-policy and what that enables (replay buffers, DQN).
• You can explain why ε-greedy must be seeded, and why you train stochastic but deploy greedy.
• You can explain "actions as tokens" and name RT-2 / OpenVLA / Octo.
• You can explain why an LLM planner needs grounding/verification and why latency is a safety property in an embodied loop.

Key takeaways

• The MDP is the substrate; everything else is how you solve it. Planning when you have the model, learning when you don't, and the Bellman backup \(V^*(s) = \max_a[r + \gamma V^*(s')]\) is the same idea under value iteration, DQN, and MuZero alike.
• Q-learning is off-policy, and that is the whole game. Behave exploratorily, learn the greedy optimum — which is what makes replay buffers, DQN, and most of modern value-based RL possible.
• VLA is the convergence point of this whole curriculum. RT-2/OpenVLA turn a VLM (Phase 04) into a robot policy by treating discretized actions as tokens it generates like words — inheriting web-scale grounding no from-scratch RL policy has. PPO (Phase 07) fine-tunes it.
• In embodied AI, generation needs verification and latency is safety. An LLM plan is creative but can be infeasible; ground it (SayCan) and verify it (Phase 14). A physical action is irreversible, so the learned policy runs under a fast, verified safety layer — always.

Next: Phase 16 — Evaluation, Observability & Guardrails.