WAM-Policy
The University of Sydney, Australian Centre for Robotics UNSW Sydney Maincode La Trobe University

WAM-Policy: World Action Models with On-Policy Distillation for Quadrupedal Loco-Manipulation

1Australian Centre for Robotics, USYD 2UNSW Sydney 3Maincode 4La Trobe University

*Equal contribution. †Corresponding authors. ‡Project lead.

Sim to Real

We deploy WAM-Policy on a Unitree Go2 Air with the two cameras without real-robot data.

Real WorldMuJoCo Simulation
Exo-centric Camera
Exo-centric Camera
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination

MuJoCo Simulation

Covering 10 tasks from QLM-Bench.

Run #1
Exo-centric Camera
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination
Run #2
Exo-centric Camera
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination
Run #3
Exo-centric Camera
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination
Run #4
Exo-centric Camera
Ego-centric Observation
Ego-centric Imagination
Top-down Observation
Top-down Imagination

Method

Stage 1 transfers an arm-centric WAM to the quadruped and accelerates it with OPSD; Stage 2 improves it per task and merges the teachers with M-OPD.

Stage 1

Cross-Morphology Policy Adaptation and Acceleration

Stage 1.1: Cross-Morphology Policy Transfer
1Arm-centric WAM
Source embodiment
Task
instruction
Text Encoderℓ
Image
observation
Video Streamzk → Zk
Future
imagination
Autoregressive
Action StreamAk
Robot
action
2Interface Adaptation
Visual Adaptation
Head
Left wrist
Right wrist
Ego-centric
Top-down
Action Adaptation
Joint + Gripper(30D)
Body + Foot End(9D)
Controller Execution
Low-level Whole-Body Controller 𝒞
Joint torque
3QLM-Bench Data Synthesis
Task prompts
Press the button.
Push the box inside the outlined area.
Kick the ball through the goal.
…
Push the oven door until the door closes.
 
LLMs
Reference Trajectory
Scene Randomization
Automatic Synthesis
Demonstrations
press button
push box
kick ball to goal
close oven door
4Stage 1.2: On-Policy Self-Distillation (OPSD)
High-NFE Quadrupedal WAM  πT
On-Policy Self-Distillation (OPSD)
Low-NFE Quadrupedal WAM  πS
Generation time / chunk
High-NFE2.226 s
Low-NFE0.159 s
14.0× Speed-up
5Closed-loop Inference
“Press the button.”
Real State
Inverse Dynamics
Future Imagination
Stage 1: Cross-Morphology Policy Adaptation and Acceleration. Stage 1.1 fine-tunes an arm-centric WAM on quadruped demonstrations synthesized in QLM-Bench. Visual Adaptation replaces the head and wrist views (images from Mobile ALOHA) by the ego-centric and top-down views, Action Adaptation replaces the 30-dimensional arm command by the 9-dimensional quadruped command, and the whole-body controller 𝒞 converts each command into joint torques. Stage 1.2, OPSD, distills the transferred policy, as the high-NFE teacher πT, into the 1/2-NFE student πS; the generation time counts the sampling of video and action only. Bottom right shows the closed loop of the policy on the task “Press the button.” Imagined views and rendered demonstrations are illustrations.

Stage 1.2: On-Policy Self-Distillation

frozen fixed targets regression trainable gradient ‖stop-gradient
Training Context
Teacher πT1
Closed-loop trajectory
Textℓ
Videoz1
ActionA1
Videoz2
ActionA2
…Terminal
Teacher Cache H, εZ, εA, ZT, AT
Environment-level exposure bias
Student πS· streaming rollout2
Context  Hk = (ℓ, z≤k, A<k)
Textℓ
Videoz1
ActionA1
…
Videozk−1
ActionAk−1
Current obs.zk
New Contexts
teacher cache
Context  H ∼ dd = (1 − η) dπS + η dπT
new contexts
Target & Loss
Teacher πT4
25 video steps50 action steps
Video StreamZT = ΦZT(εZ | H)
Action StreamAT = ΦAT(εA | ZT, H)
Shared
noise
εZ
ℒZ
εA
ℒA
Student πS5
1 video step2 action steps
Video StreamẐ = ΦZS(εZ | H)
‖
Action Stream = ΦAS(εA | sg(Ẑ), H)
Student UpdateθS = θ0 + Δ
On-Policy Self-Distillation (Stage 1.2). Left (Training Context): training contexts mix the teacher cache, which the frozen teacher collects in closed loop before training, with the contexts that the student reaches in its own streaming rollouts (H ∼ d). Right (Target & Loss): on each context, teacher and student sample from the same noise, and the student’s one-step video stream and two-step action stream are regressed to the targets of the teacher’s high-NFE sampler, with a stop-gradient on the plan that conditions the student’s action. Plan images and rendered actions are illustrations, not model outputs.
Stage 1.1 · fine-tuning
Video loss
Action loss
Training (per update)Training (100-update mean)Validation
Stage 1.2 · OPSD
OPSD: video
OPSD: action
Teacher cache only (η = 1)Streaming only (η = 0)OPSD (Ours, η = 0.5)
Left: fine-tuning losses of the transferred policy (Stage 1.1), on a logarithmic scale. Light: training loss of each update; dark: its centered moving average over 100 updates, i.e. one pass over the 800 training demonstrations; dashed: validation loss on 10 held-out demonstrations per task, every 100 updates.
Right: distillation loss of OPSD. The held-out loss is evaluated every 50 updates. Held-out loss of the students of the three training-context sources, on the same 64 held-out contexts, with a square-root horizontal axis; the dotted line marks 300 updates, after which the students are evaluated. The held-out contexts are drawn from the teacher cache, so the curves do not rank the sources: they disadvantage Streaming only, which trains on no cached contexts.
Stage 2

Task-Specific Improvement and Merging

1Stage 2.1: Task-Specific Reinforcement Learning
OPSD teacherHigh-NFE
Candidate Windows
Task-specific Reward
DiffusionNFT Update
Specialist teacher(push box)
Candidate Windows
Task-specific Reward
DiffusionNFT Update
Specialist teacher(open door)
2Stage 2.2: Multi-Teacher On-Policy Distillation
Task Router
M-OPD
M-OPD student 1/2-NFE
OPSD teacherHigh-NFE
OPSD studentInit.
Task-Specific Improvement and Merging (Stage 2). Stage 2.1 tunes the OPSD teacher by reinforcement learning into a specialist teacher for each selected task, and Stage 2.2, M-OPD, distills the teachers, routed by task, into one 1/2-NFE student that starts from the OPSD student.
Stage 2.1 · specialist teachers
Push box (mm)
Open door (cm)
Specialist teacher reward (±1 SE)Specialist teacher successOPSD teacher rewardOPSD teacher success
Stage 2.2 · M-OPD
M-OPD: video
M-OPD: action
Training lossTraining, 25-update meanHeld-out
Left: specialist teachers during Stage 2.1. Solid line (left axis): reward gain over the OPSD teacher on held-out start states after each round, paired by start state and noise, with a band of ±1 standard error over scenes; the two rewards measure different quantities (the box reward in mm, the offset of the foot on the door handle in cm). Squares (right axis): closed-loop success over 32 episodes after rounds 4, 9, and 14; the dotted line is the success of the OPSD teacher. For opening the door, the success falls from 19 of 32 episodes after round 4 to 6 after round 9 while the reward gain stays positive, so the reward is only a proxy of success there; the specialist teacher is the update after round 4.
Right: distillation loss of M-OPD. The held-out loss is evaluated every 50 updates. The M-OPD student starts from the OPSD student and learns from the specialist teachers routed by task. The light line is the training loss of every update, computed on the contexts sampled for that update rather than on the held-out contexts; the dark line is its centered moving average over 25 updates, with shorter windows at both ends; the circles are the held-out loss. The held-out contexts of M-OPD come from its own teacher cache, so its loss is not on the scale of the OPSD losses.

QLM-Bench

The ten tasks of QLM-Bench, one demonstration per task from the observer camera of the dataset.

Close a fridge door“Push against the fridge door until the door closes.”
Close an oven door“Push the oven door until the door closes.”
Turn a valve“Rotate the valve lever upward to the upright position.”
Pull open a drawer“Slide your foot into the drawer handle opening and pull the drawer outward.”
Open a room door“Press down the lever handle, then push the door open.”
Press a button“Press the button.”
Lift a basket“Lift the basket by its handle.”
Kick a ball through a goal“Kick the ball through the goal.”
Push a box into a target area“Push the box inside the outlined area.”
Sweep a block into a target area“Sweep the block into the outlined area.”