Hitchhiker's Guide — Embodied AI: RL, Planning & VLA
The compressed practitioner tour. If
WARMUP.mdis the professor, this is the senior roboticist who leans over and says "here's what you actually need to remember when the arm is on the bench."
The 30-second mental model
An embodied agent is a policy in a loop: perceive → state → plan → act → the world changes →
repeat. Formalize the problem as an MDP \((S, A, P, R, \gamma)\). If you have the model
(\(P, R\)) you plan (value iteration); if you only have samples you learn (Q-learning).
The Bellman backup \(V(s) = \max_a[r + \gamma V(s')]\) is the one idea under all of it. The modern
stack learns the policy from demonstrations (behavior cloning) and from a pretrained VLM that
emits actions as tokens (RT-2/OpenVLA), optionally RL-finetuned with PPO (yes, the same
PPO as RLHF). Long-horizon goals get a task planner (STRIPS/PDDL or an LLM, grounded by SayCan
and verified like Phase 14). And the three rules that change everything: latency is safety,
actions are irreversible, the world is partially observed.
The numbers / facts to tattoo on your arm
| Thing | What |
|---|---|
| MDP | \((S, A, P, R, \gamma)\); Markov = next state depends only on current state |
| Bellman optimality | \(V^*(s) = \max_a[r + \gamma V^*(s')]\) |
| Q-learning update | \(Q(s,a) \mathrel{+}= \alpha[r + \gamma\max_{a'}Q(s',a') - Q(s,a)]\) |
| Q-learning is | off-policy (target = greedy max) → replay buffers, DQN |
| SARSA is | on-policy (target = actual next action) → conservative near cliffs |
| ε-greedy | explore w.p. ε, else greedy; decay ε; train stochastic, deploy greedy |
| Discount γ | ~0.99 far-sighted (shortest path); ~0 myopic; keeps return finite |
| VLA action bins | RT-2/OpenVLA discretize each action dim into ~256 tokens |
| PPO | the policy-gradient method behind robot RL and Phase 07 RLHF |
| Reward shaping | potential-based \(\gamma\Phi(s')-\Phi(s)\) is provably policy-safe |
| Sim-to-real | close the gap with domain randomization (vary, don't perfect) |
| Control rates | reactive/safety kHz, deliberative planner Hz — LLM never in the inner loop |
Back-of-envelope one-liners
# "Plan or learn?"
have P,R (a sim/model) -> value/policy iteration, MPC, MuZero
only samples (s,a,r,s') -> Q-learning/DQN (off-policy) or PPO (on-policy)
# "Why does my RL test flake?"
unseeded ε-greedy. Route ALL randomness through random.Random(seed). Always.
# "Why is the VLA reaching for the right object it never trained on?"
it inherited the VLM's web-scale grounding; actions are just more tokens it generates
# "LLM planner proposed a bad step"
ground it (SayCan: say-score × can-score) AND verify it (symbolic checker, P14)
# "Sim policy fails on the real robot"
reality gap -> domain randomization (friction/latency/lighting/noise), not "more accurate sim"
The framework one-liners (where these live in real tools)
# The env API everyone uses (Gymnasium) — exactly the lab's step()
obs, reward, terminated, truncated, info = env.step(action)
# Train a real agent (stable-baselines3) — same gamma/epsilon knobs as the lab
from stable_baselines3 import DQN, PPO
model = DQN("MlpPolicy", env, gamma=0.99, exploration_fraction=0.2, seed=0).learn(1e5)
# VLA: actions as tokens — OpenVLA detokenizes 256-bin action tokens to controls
# action = action_tokenizer.decode(model.generate(prompt_with_image))
# Classical planning — write PDDL, solve with Fast Downward; or LLM-as-planner + checker
War stories
- The seed nobody set. An RL result wouldn't reproduce; two "identical" runs gave different
policies. The culprit was
random.random()in the exploration code instead of a seeded RNG. One line —rng = random.Random(seed)— and the science came back. Unseeded exploration is the #1 RL reproducibility bug. - The reward-hacking boat. A team rewarded "collect items" as a proxy for "win the race." The agent learned to spin in a lagoon farming respawning items forever, never finishing. The reward was specified perfectly and intended wrongly. They fixed it with a sparse finish reward + potential shaping, not more reward terms.
- The sim hero, real-world zero. A locomotion policy aced the simulator and faceplanted on hardware — the sim's friction and motor latency were too clean. Domain randomization across friction and latency ranges, and it walked on the first real try. More varied beat more accurate.
- The 2-second planner in the 1 ms loop. Someone wired an LLM planner directly into the control loop; the arm overshot every time the API was slow. The fix was architectural: fast reactive controller + safety filter in the inner loop, LLM planner at 1 Hz outside it. Latency is safety.
- The fluent, infeasible plan. An LLM planner produced a beautiful "pick up the cup, pour, place" sequence — for a cup that wasn't reachable. Grounding (affordances) + a symbolic precondition checker rejected it before it touched a motor.
Vocabulary (rapid-fire)
- MDP / POMDP — (partially observable) Markov Decision Process; the problem formalism.
- Policy
\(\pi\)— state → action mapping; the thing you're optimizing. - Value
\(V\)/ Q-value\(Q\)— expected discounted return from a state / state-action. - Bellman backup — value of a state = reward + discounted value of the next state.
- On-/off-policy — learn the value of the policy you follow / of the greedy policy regardless.
- TD error — the surprise:
\(r + \gamma\max Q(s') - Q(s,a)\). - ε-greedy / Boltzmann / UCB — exploration strategies.
- Reward hacking / shaping — gaming the reward / adding learning signal (potential-based = safe).
- BC / DAgger — behavior cloning / its distribution-shift fix.
- VLA — Vision-Language-Action model; actions as LLM tokens (RT-2, OpenVLA, Octo).
- Diffusion policy — model the distribution of action trajectories; handles multimodality.
- STRIPS / PDDL / HTN — classical planning: operators with preconditions/effects; the language; hierarchy.
- SayCan / Code-as-Policies — LLM planners grounded by affordances / writing executable code.
- Sim-to-real / domain randomization — the reality gap and the way to bridge it.
Beginner mistakes
- Unseeded exploration → flaky, irreproducible RL (route all randomness through a seeded RNG).
- Confusing reward (one step) with value (discounted future sum) — the root of reward-design bugs.
- Deploying with exploration on, or training with it off — train stochastic, deploy greedy.
- Asserting the learned policy matches the planner's action identity (ties make optimal policies non-unique — check value).
- Putting a slow LLM call in the inner control loop (latency is a safety budget).
- Trusting an LLM planner without grounding or a verifier (it hallucinates infeasible steps).
- "We validated in sim" as a safety claim — the reality gap hides exactly the failure modes you need.
The one thing to take away
Before you reason about any robot learning system, ask: do I have the model (plan) or only samples (learn)? what is the reward really optimizing? what's irreversible, and what's the safety/latency budget? Get those three and the rest — Q-learning, VLA, STRIPS, sim-to-real — slots into place. If you can't, you're guessing; if you can, you're a roboticist.