Warmup Guide — Embodied AI & Robotics: RL, Planning & VLA

Zero-to-senior primer for Phase 15. Every phase before this built a model that emits text. This one builds an agent that acts in a world — and that single change rewrites the rules. We start from "what does embodied actually change?" and end with a control stack you could defend in a robotics design review: the Markov Decision Process and the Bellman equations, the split between planning (you have the model) and learning (you don't), exploration vs exploitation, reward design and reward hacking, the modern stack (imitation learning, Vision-Language-Action models that decode actions as tokens, diffusion policies), classical and LLM task planning (STRIPS/PDDL, SayCan, Code-as-Policies), simulation and sim-to-real, and why in an embodied system latency is a safety property. The lab makes all of it concrete in stdlib Python, with the VLM replaced by an injected deterministic policy so you see the mechanism without a model in the way.

Table of Contents

Chapter 1: What "Embodied" Changes

From zero. Everything in this curriculum so far produced an artefact that lives in a buffer: a token, a string, a retrieved document, a tool call. If it's wrong, you regenerate. The environment is patient, stateless, and forgiving. Embodied AI breaks every one of those assumptions. An embodied agent — a robot arm, a mobile base, a drone — perceives a physical world, takes an action in that world, and the world changes because of the action. There is no undo. There is no regenerate. The coffee is on the floor.

The loop that defines the field. Strip a robot of its hardware and what's left is a control loop running forever:

        ┌──────────────────────────────────────────────────────┐
        │                                                        │
        ▼                                                        │
   ┌─────────┐    ┌────────┐    ┌────────┐    ┌────────┐         │
   │ PERCEIVE│ →  │  STATE │ →  │  PLAN  │ →  │  ACT   │ ────────┘
   │ (camera,│    │(estimate│    │(choose │    │(motors,│
   │  lidar) │    │  world) │    │ action)│    │gripper)│
   └─────────┘    └────────┘    └────────┘    └────────┘
        ▲                                          │
        └──────────  the world changes  ───────────┘

Perception (Phase 04's VLMs feed this), state estimation, planning, action, and then the world hands you a new observation. The agent never escapes the loop; it just gets better at closing it.

Three things that are now true and weren't before.

  1. Latency is a safety property, not a UX metric. When ChatGPT is slow you're annoyed. When a robot's control loop is slow, the arm overshoots, the drone drifts into the wall, the car brakes late. A 200 ms planning stall is a physical event. This is why embodied systems run a fast reactive controller (kHz) under a slow deliberative planner (Hz) — and why you cannot just bolt a 2-second LLM call into the inner loop.
  2. Partial observability is the default. A camera sees one side of an object; an occluded shelf hides the cup. The agent rarely knows the true state — it maintains a belief over states (a POMDP, the partially-observable cousin of the MDP). The lab's GridWorld is fully observed to keep the algorithms clean; reality is not.
  3. Mistakes are expensive and irreversible. You cannot "sample 10 times and pick the best" with a physical action. This is why verified plans (Phase 14) and safety constraints matter so much more here than in a chatbot, and why so much training happens in simulation first.

The senior framing. An embodied agent is a policy \(\pi(a \mid s)\) — a mapping from what it perceives to what it does — wrapped in a loop, optimized against a reward, and constrained by safety. The next eight chapters are the machinery for building, learning, and grounding that policy.

Common misconception. "Robotics is just RL on a robot." RL is one tool. Most deployed robots run classical planning + control with learning bolted onto perception and a few skills; the frontier (RT-2, OpenVLA) is learning the whole policy end-to-end, but even those lean on planners and safety layers. A senior knows the whole stack, not just the part that's trending.

Chapter 2: The MDP — States, Actions, Rewards, Discount

The formalism the whole field stands on. A Markov Decision Process is the mathematical object that defines a sequential decision problem. It is a five-tuple \((S, A, P, R, \gamma)\):

  • \(S\) — the set of states (in the lab, grid cells).
  • \(A\) — the set of actions (up/down/left/right).
  • \(P(s' \mid s, a)\) — the transition probability of landing in \(s'\) after action \(a\) in state \(s\). (The lab is deterministic, so \(P\) puts all its mass on one \(s'\).)
  • \(R(s, a)\) — the reward for that transition (goal \(+1\), otherwise a small step penalty).
  • \(\gamma \in [0, 1]\) — the discount factor: how much a future reward is worth now.

The Markov property is the load-bearing assumption: the future depends only on the current state, not the path that got you there. \(P(s_{t+1} \mid s_t, a_t)\) — \(s_t\) is a sufficient statistic. This is what makes everything tractable; it is also what fails under partial observability (you don't have the true \(s_t\)), forcing the POMDP machinery.

The objective. The agent wants a policy \(\pi\) (a mapping from states to actions) that maximizes the expected discounted return — the sum of future rewards, each discounted by how far off it is:

$$ G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}. $$

Why discount at all? Three reasons, and you should be able to say all three: (1) mathematically, \(\gamma < 1\) keeps the infinite sum finite and the math well-posed; (2) it encodes a preference for sooner reward (a coffee now beats a coffee in an hour); (3) it models uncertainty about the future — the episode might end, the world might change. In the lab, \(\gamma = 0.99\) says "the future matters a lot, but reaching the goal in fewer steps is still better," which is exactly what makes the optimal policy take the shortest path.

The value functions. Two objects measure "how good":

  • The state-value \(V^\pi(s)\) — expected return starting from \(s\) and following \(\pi\).
  • The action-value \(Q^\pi(s, a)\) — expected return starting from \(s\), taking action \(a\), then following \(\pi\). \(Q\) is the one you can act on directly: pick the action with the highest \(Q\).

Common misconception. "The reward is the value." No — the reward is the immediate, one-step signal; the value is the expected discounted sum of all future rewards. A state can have low immediate reward but high value because it's on the path to the goal. Confusing the two is the most common beginner error in RL, and it's the source of most reward-design disasters (Chapter 6).

Chapter 3: The Bellman Equations & Value Iteration (Planning)

The recursive insight. Richard Bellman's observation in 1957 is the spine of the field: the value of a state is the immediate reward plus the discounted value of where you land next. For the optimal value function \(V^*\), you take the best action:

$$ V^(s) = \max_a \Big[ R(s, a) + \gamma \sum_{s'} P(s' \mid s, a), V^(s') \Big]. $$

This is the Bellman optimality equation. For the lab's deterministic world the sum collapses to a single term — there is only one \(s'\) per \((s, a)\) — so it's just:

$$ V^(s) = \max_a \big[ R(s, a) + \gamma, V^(s') \big]. $$

Value iteration: turn the equation into an algorithm. The Bellman optimality operator is a contraction (it shrinks the distance between any two value estimates by a factor of \(\gamma\) each application), so iterating it from any starting point converges to the unique fixed point \(V^*\):

initialize V(s) = 0 for all s
repeat:
    delta = 0
    for each non-terminal state s:
        v_old = V(s)
        V(s) = max_a [ R(s,a) + gamma * V(next(s,a)) ]   # Bellman backup
        delta = max(delta, |V(s) - v_old|)
until delta < theta          # converged
policy(s) = argmax_a [ R(s,a) + gamma * V(next(s,a)) ]    # greedy w.r.t. V*

theta is the stopping tolerance: when no value moved more than theta in a sweep, you're at the fixed point. Then the optimal policy is greedy with respect to \(V^*\) — at each state, pick the action whose backup is largest.

This is planning, and it has a hard prerequisite: you need the model. Value iteration reads \(R\) and \(P\) directly — it asks "if I take action \(a\) here, what reward do I get and where do I land?" In the lab that's a free call to env.step. On a real robot you usually don't have \(P\) (you don't know the physics exactly), which is the entire reason Chapter 4 exists.

 PLANNING (this chapter)          LEARNING (next chapter)
 ─────────────────────────        ─────────────────────────
 you HAVE P and R                 you DON'T — only samples
 value/policy iteration           Q-learning / SARSA / PPO
 dynamic programming              temporal-difference / gradients
 one offline solve                online, episode by episode
 MuZero's planner, MPC            DQN, robot RL, RLHF (Phase 07)

Common misconception. "Value iteration is old and irrelevant." The exact tabular version is limited to small state spaces, but the idea — bootstrap a value estimate from the next state's estimate — is the beating heart of MuZero (which plans with a learned model), MPC controllers, and every TD method. Understanding it cold is non-negotiable.

Chapter 4: Q-Learning — Control Without a Model (Learning)

The problem. You're on a real robot. You don't know \(P\). You can't run value iteration because you can't ask "where would action \(a\) land me?" without actually trying it. All you get is experience: tuples \((s, a, r, s')\) — I was in \(s\), did \(a\), got \(r\), ended up in \(s'\). Can you learn the optimal policy from samples alone? Yes — this is the breakthrough of temporal-difference learning, and Q-learning is its cleanest form.

The Q-learning update. Keep a table \(Q(s, a)\), start it at zero, and after every experienced transition nudge the estimate toward the TD target — the reward plus the discounted best you think you can do from the next state:

$$ Q(s, a) ;\leftarrow; Q(s, a) + \alpha\Big[, \underbrace{r + \gamma \max_{a'} Q(s', a')}{\text{TD target}} - \underbrace{Q(s, a)}{\text{old estimate}} ,\Big]. $$

The bracket is the TD error: how surprised you were. \(\alpha\) (the learning rate) controls how much each surprise moves the estimate. That's the entire algorithm — one line, run over a stream of experience.

Why it converges to the optimum. The target \(r + \gamma \max_{a'} Q(s', a')\) is a sampled, bootstrapped version of the Bellman optimality backup from Chapter 3. Under mild conditions (every state-action visited infinitely often, \(\alpha\) decayed appropriately), \(Q\) provably converges to \(Q^*\). You are doing value iteration one stochastic sample at a time, without ever knowing \(P\).

Off-policy is the magic word. Notice the target uses \(\max_{a'} Q(s', a')\) — the greedy action — regardless of what action you actually take next. So you can behave however you like (exploring randomly, replaying old data) while learning about the optimal greedy policy. That decoupling is what makes a replay buffer legal and is exactly what powers DQN (Q-learning with a neural net as the table and a buffer of past transitions). Contrast SARSA, the on-policy sibling, whose target uses the action you actually took next — it learns the value of the policy you're following, which makes it more conservative near cliffs.

The soul of the lab. Start \(Q\) at all zeros. Feed it nothing but (s, a, r, s') tuples from ε-greedy wandering. After enough episodes, greedy_policy(Q) walks straight to the goal in the fewest possible steps — and every action it picks achieves value iteration's optimal value. A table of zeros became an optimal controller from experience alone. That is RL.

Common misconception. "Q-learning learns the path it took." No — it learns the value function, from which the optimal path falls out. The path it explored was full of random detours; the policy it learned is optimal. And note: optimal policies are not unique — when two actions tie on value, either is optimal, so the learned policy may differ in identity from value iteration's while being identical in value. The lab's cross-check tests value, not identity, for exactly this reason.

Chapter 5: Exploration vs Exploitation

The dilemma. To learn which action is best you must try actions — including ones that currently look bad (explore). But to collect reward you should take the action that currently looks best (exploit). Always exploiting means you never discover the better route you haven't tried; always exploring means you learn a lot and achieve nothing. Every RL algorithm needs a knob for this trade.

ε-greedy: the workhorse. With probability \(\varepsilon\) take a uniformly random action; otherwise take the greedy (current-best) action.

$$ a = \begin{cases} \text{random action} & \text{with probability } \varepsilon \ \arg\max_a Q(s, a) & \text{with probability } 1 - \varepsilon. \end{cases} $$

In practice you decay \(\varepsilon\) over training: explore hard early (the table is garbage, exploration is free information), exploit late (the table is good, random moves only cost you). The lab uses a fixed \(\varepsilon\) for clarity, but the decay schedule is what you tune in real runs.

The determinism rule that bites everyone. The exploration draw is random — so if you call the global random module, two runs disagree and your tests are flaky. All randomness must go through an injected, seeded random.Random. Same seed → same sequence of "random" actions → same learned table → reproducible bytes. This isn't pedantry: reproducing a paper's RL result requires the seed, and a flaky RL test is worse than no test. The lab enforces this, and the determinism test will fail the moment you reach for unseeded randomness.

Beyond ε-greedy (know the names). Boltzmann/softmax exploration samples actions in proportion to \(e^{Q(s,a)/\tau}\) (temperature \(\tau\)), so near-best actions are tried more than terrible ones. UCB (upper confidence bound) adds an optimism bonus to under-tried actions. Intrinsic motivation / curiosity rewards the agent for visiting novel states — essential in sparse-reward worlds where ε-greedy would wander forever before ever hitting reward.

Common misconception. "Set ε to zero once it's learned." During training you need exploration; at deployment you typically run greedy (\(\varepsilon = 0\)). Confusing the two — deploying with heavy exploration, or training with none — is a classic failure. Train stochastic, deploy greedy.

Chapter 6: Reward Design & Reward Hacking

Reward is the only thing the agent optimizes. This is the most underestimated fact in RL. The agent does not want what you want; it wants what the reward function says. If those differ — and they always differ a little — you get reward hacking: the agent finds a high-reward behaviour that violates your actual intent.

The canonical disasters. A boat-racing agent that learned to spin in circles collecting power-ups forever instead of finishing the race (the reward credited power-ups, not finishing). A simulated robot that learned to vibrate to exploit a physics-engine bug for "forward velocity." A cleaning bot that knocks things over so it has more to clean. Each is the reward function being specified correctly and intended wrongly — the agent is doing exactly what you said, not what you meant.

Sparse vs dense reward — the core tension.

  • Sparse (+1 only at the goal, like the lab's main signal): unambiguous and hard to hack, but the agent gets no learning signal until it stumbles onto the goal by chance — brutal in large spaces (this is why exploration matters so much).
  • Dense / shaped (a little reward for progress, e.g. the lab's small step penalty that nudges toward shorter paths): faster learning, but every shaping term is a new surface for the agent to exploit. The lab's step penalty is a safe shaping (it can't be gamed without reaching the goal); a naive "reward for being near the goal" can create a local optimum that parks near the goal forever without entering it.

Potential-based shaping — the one safe trick. Ng et al. (1999) proved that shaping of the form \(F(s, s') = \gamma\,\Phi(s') - \Phi(s)\) (a potential difference) provably does not change the optimal policy — it only speeds learning. If you must shape, shape this way. Knowing this theorem is a senior signal.

Common misconception. "We'll just add more reward terms until it behaves." Every term is a constraint the agent will satisfy in the cheapest way it can find, which is rarely the way you imagined. The senior discipline is the smallest reward that specifies the task, plus potential-based shaping, plus watching what the agent actually does — because the gap between "specified" and "intended" is where embodied systems hurt people.

Chapter 7: The Modern Stack — Imitation, VLA & Diffusion Policies

RL from scratch is sample-hungry and dangerous on real hardware. The 2023–2025 frontier shifted to learning policies from demonstrations and from internet-scale vision-language pretraining. Three families you must be able to compare:

1. Imitation Learning / Behavior Cloning (BC). Instead of a reward, give the agent demonstrations — (observation, expert action) pairs from teleoperation or motion capture — and train a supervised policy to copy them: \(\min_\theta \sum \ell(\pi_\theta(s_i), a_i^{\text{expert}})\). Dead simple, no reward design, no dangerous exploration. The catch is compounding error / distribution shift: a tiny mistake takes the robot to a state the expert never visited, where the policy has no idea what to do, and errors snowball. DAgger fixes this by iteratively querying the expert on the states the learner actually visits. BC is the foundation most modern robot policies build on.

2. Vision-Language-Action (VLA) models — "actions as tokens." The headline idea of RT-2, OpenVLA, and Octo, and the heart of the lab. Take a pretrained Vision-Language Model (Phase 04 — it already grounds images and language), and make it output robot actions by treating each continuous action dimension as just another token in the vocabulary:

   instruction: "pick up the red block"   image: [camera frame]
            │                                     │
            ▼                                     ▼
        ┌──────────────────────────────────────────────┐
        │     Vision-Language Model (a transformer)      │
        │   tokens in  →  tokens out, autoregressively   │
        └──────────────────────────────────────────────┘
                              │
                              ▼
        action tokens:  [137]  [200]  [42]  ...  <eos>
                              │  detokenize (bin → centre)
                              ▼
        continuous actions:  Δx=0.07  Δy=-0.02  grip=close

A continuous control value (a joint delta, a gripper command) is discretized into bins — RT-2/ OpenVLA use 256 bins per dimension — and each bin is an integer token. The VLM generates the action tokens exactly as it generates words, then a de-tokenizer maps each token back to its bin's centre. The payoff is enormous: the policy inherits the VLM's web-scale common sense and language grounding, so "pick up the extinct animal" can reach for the toy dinosaur — semantic generalization no from-scratch RL policy has. The lab's ActionTokenizer + vla_decode is exactly this interface (tokenize → decode via an injected policy → de-tokenize), with the giant VLM swapped for a deterministic stub so you can see the mechanism.

3. Diffusion Policies. Instead of predicting one action, model the distribution of good action trajectories and sample from it via a denoising diffusion process (the same machinery as image diffusion, applied to action sequences). The win is multimodality: when there are several good ways to do something (go left or right around the obstacle), a single-action policy averages them into a bad middle, while a diffusion policy commits to one mode. State of the art for fine-grained manipulation.

Common misconception. "VLAs replaced RL." They're complementary. BC/VLA gets you a strong starting policy cheaply and safely from demonstrations; RL (often PPO — the same algorithm you met in Phase 07's RLHF, just with environment reward instead of a reward model) fine-tunes it past what the demonstrations showed. The frontier recipe is pretrain a VLA on demonstrations, then RL-finetune — pretrain-then-RL, exactly like the LLM alignment pipeline.

Chapter 8: Task Planning — STRIPS, HTN & the LLM Planner

RL/VLA learns skills ("grasp the cup"). Task planning sequences skills into a multi-step plan ("clear the table" = pick cup → move → place → pick plate → …). Long-horizon goals need both.

STRIPS — the classical formalism. A planning problem is a start state, a goal, and a set of operators, each with preconditions (what must be true to apply it) and effects (what it adds and deletes). The world state is a set of true predicates; planning is search for a sequence of operators that transforms start into a state satisfying the goal. The lab implements exactly this:

Operator "pick":
    preconditions: {at(block), hand_empty}      # must hold to apply
    add_effects:   {holding(block)}             # become true after
    del_effects:   {hand_empty}                 # become false after

plan(start={at(home), hand_empty}, goal={on(block,target)}, operators=[...])
  → ["move_to_block", "pick", "move_to_target", "place"]     # shortest valid sequence

The lab plans with BFS over symbolic states, which guarantees the shortest plan. Real planners (Fast Downward, Graphplan) use heuristics because the state space is exponential — STRIPS planning is PSPACE-hard — but the bones are identical: applicable operators expand the frontier, the goal is a subset test. PDDL is the standard language for writing these problems; HTN (Hierarchical Task Networks) add a layer of abstraction — high-level tasks decompose into subtasks decompose into primitive operators — which is how you keep long-horizon plans tractable.

The LLM as planner. The 2022+ move: let an LLM propose the plan in natural language or code, because it has common-sense task knowledge a hand-written operator set lacks. Two landmark patterns:

  • SayCan (Google, 2022): the LLM scores what would help ("say"); a learned affordance model scores what's possible right now ("can"); the agent picks the action maximizing the product. The LLM's eloquence is grounded by what the robot can actually do — it won't say "grab the apple" if no apple is reachable.
  • Code-as-Policies (Google, 2022): the LLM writes executable code that calls perception and control primitives (get_obj_pos, move_to), so the plan is a runnable program with loops and conditionals.

Why you still need the classical checker. An LLM plan can be fluent and wrong — preconditions unmet, steps in the wrong order, an impossible action. The robust pattern (and a direct tie to Phase 14's neurosymbolic verification) is LLM proposes, symbolic planner/checker verifies: the lab's plan/execute_plan is precisely that verifier — it confirms a proposed sequence is applicable end-to-end and achieves the goal, and rejects it otherwise. Generation is creative; verification is sound; you want both.

Common misconception. "Just let the LLM plan, it's smart enough." LLM planners hallucinate infeasible steps and silently violate constraints. Without grounding (SayCan) or a verifier (Phase 14), you get plans that read beautifully and break the robot. Generation without verification is how embodied LLM systems fail.

Chapter 9: Simulation, Sim-to-Real & Safety

Why simulate. Real-robot data is slow (real-time), expensive (hardware, operators), and dangerous (a learning policy will do stupid things). Simulators — MuJoCo, Isaac Sim/Gym, Habitat — run thousands of parallel environments faster than real time, with perfect state access and zero broken hardware. Almost all modern robot learning starts in sim.

The reality gap. A policy trained in sim and dropped on a real robot usually fails, because the simulator is wrong in a thousand small ways: friction, motor latency, sensor noise, lighting, contact dynamics. The distribution the policy learned ≠ the distribution it faces. Closing this gap is sim-to-real transfer, and the dominant technique is:

Domain randomization. Instead of trying to make the simulator perfectly accurate, make it randomly varied — randomize friction, masses, textures, lighting, latencies, sensor noise across a wide range during training. A policy forced to succeed across that whole distribution treats the real world as just one more sample from the distribution it already handles. Counterintuitive but field-defining: you bridge the gap by making sim less realistic and more varied. (OpenAI's Rubik's-cube hand and most quadruped locomotion ship this way.)

Safety & verification. Because actions are irreversible and physical, embodied systems wrap the learned policy in guardrails that a chatbot never needs:

  • Hard constraints / safety filters — a verified layer (control barrier functions, geofences, torque/velocity limits) that can override the policy. The learned policy proposes; the safety layer disposes. This is the embodied version of Phase 14's verified plans and Phase 16's guardrails: never let an unverified policy command a motor directly.
  • The latency budget is a safety budget (Chapter 1, again). The safety filter and the reactive controller run fast and deterministically under the slow learned planner, so even if the VLA stalls for a second, the robot doesn't drive into the wall.

Common misconception. "We validated in sim, so it's safe for the real world." Sim validation is necessary, not sufficient — the reality gap means the real failure modes (a sensor glitch, an unmodeled contact) are exactly the ones sim didn't show you. Real deployment needs the safety filter, real-world testing, and graceful degradation, not just a good sim score.

Lab Walkthrough Guidance

The lab (lab-01-rl-planner-vla) turns these nine chapters into code. Suggested order matches the file:

  1. GridWorld (Chapter 2) — the MDP. The only subtlety is step: bump-into-wall stays put, reaching the goal returns (goal, goal_reward, True), and you must reject stepping from a terminal state. Keep it a pure function so planning and learning can both call it.
  2. value_iteration (Chapter 3) — the Bellman backup over non-terminal states until delta < theta, then a greedy argmax for the policy. This is your ground-truth optimum for the soul tests.
  3. q_learning + _epsilon_greedy (Chapters 4–5) — the one-line TD update, with seeded ε-greedy. Get the seeding right or the determinism test fails.
  4. greedy_policy / rollout — extract behaviour from a value table; tie-break to fixed action order so the policy is deterministic.
  5. Operator / plan / execute_plan (Chapter 8) — applicable is a subset test, apply is (state − del) | add, plan is BFS returning the shortest operator sequence (or None).
  6. ActionTokenizer / vla_decode (Chapter 7) — uniform binning with a round-trip bounded by one bin; vla_decode concatenates instruction + perception, drives the injected policy, and de-tokenizes until the policy returns None (the <eos>).

Run red → green: pytest test_lab.py -v, then LAB_MODULE=solution pytest test_lab.py -v, then python solution.py for the worked example. The two tests to stare at: test_q_learning_converges_to_optimal_open_grid (the RL soul test) and test_q_learning_matches_value_iteration_policy (the planner cross-check — note it tests value, not action identity).

Success Criteria

  • All tests pass against your lab.py and against LAB_MODULE=solution.
  • You can write the Bellman optimality equation and explain why iterating it converges (contraction).
  • You can write the Q-learning update from memory and explain why it's off-policy (target uses the greedy \(\max_{a'}\) regardless of behaviour) and what that enables (replay buffers, DQN).
  • You can explain why ε-greedy must be seeded and why you train stochastic but deploy greedy.
  • You can explain why the cross-check soul test asserts action value, not action identity (optimal policies are not unique).
  • You can explain "actions as tokens" — discretize → the VLM generates action tokens → de-tokenize — and name RT-2/OpenVLA/Octo as the production systems.
  • You can explain why an LLM planner needs grounding (SayCan) or a verifier (the lab's execute_plan, Phase 14), and why latency is a safety property in an embodied loop.

Interview Q&A

  • "Define an MDP, and what is the Markov property?" — \((S, A, P, R, \gamma)\); the Markov property is that the next state depends only on the current state and action, not the history — the state is a sufficient statistic. Partial observability breaks it → POMDP.
  • "Planning vs learning — when do you use each?" — Planning (value/policy iteration, MPC) when you have the dynamics model \(P\); learning (Q-learning, SARSA, policy gradient, PPO) when you only have samples. Real robots rarely have an exact model, so they learn — or plan with a learned model (MuZero).
  • "Write the Q-learning update and explain each term." — \(Q(s,a) \mathrel{+}= \alpha[r + \gamma \max_{a'} Q(s',a') - Q(s,a)]\): TD target (reward + discounted best next value) minus old estimate is the TD error; \(\alpha\) scales the correction.
  • "Why is Q-learning off-policy and SARSA on-policy?" — Q-learning's target uses the greedy next action (\(\max\)) regardless of what you do → learns \(Q^*\) while behaving exploratorily. SARSA's target uses the actual next action → learns the behaviour policy's value (more conservative near cliffs). Off-policy is what makes replay buffers and DQN legal.
  • "How does PPO relate to what you did in Phase 07?" — Identical algorithm. RLHF is RL where the reward is a learned reward model and the "environment" is text generation; embodied PPO uses an environment reward and physical actions. Same clipped policy-gradient objective, different reward source.
  • "What is reward hacking and how do you defend against it?" — The agent optimizes the specified reward, not the intended goal, and finds a degenerate high-reward behaviour. Defend with the smallest sufficient reward, potential-based shaping (provably policy-invariant), and watching actual behaviour — not by piling on reward terms.
  • "Explain 'actions as tokens' (RT-2/OpenVLA)." — Discretize each continuous action dimension into bins (≈256), treat each bin as a vocabulary token, and have a pretrained VLM generate action tokens autoregressively like words, then de-tokenize to bin centres. The policy inherits the VLM's web-scale language/vision grounding — semantic generalization from-scratch RL can't match.
  • "Behavior cloning vs RL — tradeoffs?" — BC is supervised on demonstrations: simple, safe, no reward design, but suffers compounding error / distribution shift (fixed by DAgger). RL explores and can surpass the demos but is sample-hungry and dangerous on hardware. Frontier recipe: BC/VLA pretrain, then RL-finetune.
  • "What is the sim-to-real gap and how does domain randomization close it?" — A sim-trained policy fails on real hardware because the sim is subtly wrong (friction, latency, noise). Domain randomization varies those parameters widely in training so the real world is just one more sample from a distribution the policy already handles — bridge the gap by making sim more varied, not more accurate.
  • "How do you make an LLM planner safe for a robot?" — Ground it (SayCan: multiply LLM "say" scores by affordance "can" scores) and/or verify it (a symbolic planner/checker confirms preconditions hold and the goal is achieved — the lab's execute_plan, tying to Phase 14). Never let an unverified plan command a motor; keep a fast safety filter under the slow planner.
  • "Why is latency a safety property here but not in a chatbot?" — A physical action is irreversible and the world keeps moving; a stalled control loop means overshoot, drift, or a late brake — a physical event. So embodied systems run a fast reactive controller and safety filter under the slow deliberative planner, and you cannot drop a 2-second LLM call into the inner loop.
  • "Why discount future reward?" — To keep the infinite-horizon return finite, to prefer sooner reward, and to model uncertainty about the future. With \(\gamma\) close to 1 the agent is far-sighted (and, with a step penalty, takes the shortest path); \(\gamma\) near 0 makes it myopic.

References

  • Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — the MDP, Bellman equations, value iteration, Q-learning, SARSA, exploration. The canonical text.
  • Bellman, Dynamic Programming (1957) — the original recursive value formulation.
  • Mnih et al., Human-level control through deep reinforcement learning (DQN, Nature 2015) — Q-learning + neural net + replay buffer.
  • Schulman et al., Proximal Policy Optimization Algorithms (PPO, 2017) — the policy-gradient method behind both robot RL and Phase 07's RLHF.
  • Brohan et al., RT-2: Vision-Language-Action Models (Google DeepMind, 2023) — actions as tokens.
  • Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model (2024); Octo (2024) — open VLAs.
  • Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan, 2022).
  • Liang et al., Code as Policies: Language Model Programs for Embodied Control (2022).
  • Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023).
  • Tobin et al., Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (2017); OpenAI, Solving Rubik's Cube with a Robot Hand (2019).
  • Ng, Harada, Russell, Policy Invariance Under Reward Transformations (potential-based shaping, 1999).
  • Fikes & Nilsson, STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving (1971); the PDDL standard and the Fast Downward planner.
  • Todorov et al., MuJoCo (2012); NVIDIA Isaac Sim/Gym; Gymnasium (the Env.step API).