Coordinating Humanoid Loco-Manipulation
through Action-Conditioned Latent Foresight

Ju Dong^1,†Xinyu Xiao⁴Yiming Zhou^4,†Tianshuo Yang²Xuerong Liu²Lei Zhang¹Jian Chen⁵Haocheng Zhao³Yu Fu¹Yujie Wei⁴Diwen Zheng³Alois Christian Knoll³Angela P. Schoellig³Ping Luo^2,*Jianwei Zhang¹

¹ TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg, Hamburg, Germany.
² The University of Hong Kong, Hong Kong, China.
³ Technical University of Munich, Germany.
⁴ Kinetix AI, China.
⁵ University of Science and Technology of China, Hefei, China.

† Project leads * Corresponding author

Predict visual progress and evaluate body–hand temporal consistency before execution.

Training and inference

Network architecture

Open interactive figure ↗

Demonstrations → frozen visual features → endpoint prediction → action consistency → joint training

Paused

Swipe or scroll horizontally to explore the full architecture.

OVERVIEW

Training uses demonstrated commands. Current and future images are encoded separately; a shared inverse decoder reconstructs the first command from observed and predicted feature changes.

Method

Predict progress.
Coordinate commands.

Action-conditioned latent foresight

An action generator proposes body-motion and hand command sequences from an egocentric image, instruction and robot state. An endpoint predictor estimates visual features after each proposed prefix. Visual progress measures how much closer these features move toward the active demonstration-derived subgoal.

Action consistency regularization

During training, one inverse action decoder reconstructs the first demonstrated command from observed and predicted feature changes. This connects latent prediction to the commands responsible for the change.

Joint temporal prior

A joint prior measures next-command prediction errors across body-motion and hand components. Visual progress and temporal-consistency energy score the first h commands of each sequence. The robot executes the first K commands of the best-scoring sequence and replans.

V-JEPA 2.1 is adapted on HIW-500, a dataset of whole-body humanoid demonstrations collected in real homes, then frozen. Ordered visual subgoals are extracted from task demonstrations and reused under standard and unseen object configurations.

Real-world demonstrations

Loco-manipulation on Unitree G1

Seven tasks under standard conditions
and 18 successful recordings with unseen object configurations.

TASK 01 / 07

Dispose of Trash

Pick up the object, open the bin with the foot pedal, and dispose of the trash.

Download view ↓

ZED2 provides the policy’s egocentric input; two external cameras show the full execution. Supplied video edits include accelerated segments.

Experiments

Task performance

Ten SIMPLE tasks and seven real-robot tasks.

Download chart data ↓

Six real-robot task scenes and ten SIMPLE simulation tasks; basket placement appears in the method overview. — Task scenes from the real robot and SIMPLE humanoid loco-manipulation benchmark.

Platforms and implementation settings · Table I

Setting	Value
Real robot	Unitree G1
Egocentric camera	ZED2
Training hardware	8× NVIDIA A100
Evaluation (sim./real)	1× NVIDIA RTX 4090D (48 GB)
Onboard deployment test	Jetson Orin NX (16 GB)
Proposal length	Sim.: H=30; real: H=40
Evaluation horizon	h=20
Executed prefix length	K=10; commands at 30 Hz
Command chunks	N=8
Joint temporal prior	GRU, 2 layers, hidden width 512
Loss weights	λwm=0.1, λΔ=0.05, λacr=0.02, λprior=0.01
Score weight	β=0.1
Subgoals per task	J=5
Subgoal tracker	εg=0.15

Each method trained in our experiments uses a separate policy per task. Simulation policies use 40,000 training steps and a per-GPU batch size of 32; real-robot policies use 90,000 steps and a per-GPU batch size of 4.

01 · Simulation

SIMPLE benchmark

Evaluation scope

Average success across ten tasks and three difficulty levels.

Ours: 89.7%; DiT4DiT: 82.0%; Ψ₀: 73.7%; VLA-JEPA: 49.0%; Fast-WAM: 27.3% — Ten episodes per task and difficulty level.Download SVG ↓

View chart counts

Method	Successes / trials	Success rate

Six main tasks · Table II

Method	XMovePick	BendPick	Handover	Mobile P&P	Grasp	XMoveBendPick	Mean (%)
Ψ₀	10 / 10 / 6	10 / 10 / 10	7 / 7 / 10	7 / 5 / 6	10 / 10 / 8	10 / 9 / 9	85.6
GR00T N1.6	10 / 10 / 7	7 / 7 / 6	1 / 3 / 3	0 / 0 / 0	9 / 9 / 7	4 / 4 / 1	48.9
OpenPI 0.5	7 / 5 / 1	10 / 10 / 8	5 / 4 / 5	3 / 3 / 3	10 / 10 / 8	0 / 0 / 0	51.1
InternVLA-M1	0 / 0 / 0	5 / 5 / 0	0 / 0 / 0	0 / 0 / 0	0 / 0 / 0	3 / 5 / 7	13.9
H-RDT	0 / 0 / 2	0 / 0 / 1	0 / 1 / 0	0 / 0 / 0	0 / 0 / 0	0 / 0 / 0	2.2
DreamZero	10 / 10 / 10	9 / 9 / 8	7 / 8 / 9	5 / 3 / 3	9 / 10 / 7	0 / 0 / 1	65.6
EgoVLA	0 / 1 / 2	7 / 5 / 8	0 / 4 / 3	0 / 0 / 0	10 / 10 / 7	3 / 5 / 4	38.3
Diffusion Policy	3 / 3 / 2	10 / 8 / 6	3 / 2 / 4	4 / 0 / 0	8 / 9 / 8	0 / 0 / 0	38.9
ACT	10 / 10 / 5	10 / 9 / 9	7 / 7 / 10	5 / 5 / 5	10 / 10 / 8	9 / 10 / 10	82.8
DiT4DiT	10 / 9 / 9	9 / 9 / 10	9 / 9 / 10	7 / 7 / 7	10 / 9 / 7	9 / 9 / 9	87.8
Fast-WAM	10 / 10 / 6	0 / 3 / 1	6 / 5 / 7	0 / 0 / 0	10 / 7 / 7	0 / 0 / 0	40.0
VLA-JEPA	0 / 0 / 0	1 / 0 / 0	7 / 8 / 9	7 / 7 / 9	9 / 8 / 5	0 / 3 / 4	42.8
Ours	10 / 10 / 7	10 / 9 / 10	8 / 10 / 9	10 / 10 / 9	10 / 10 / 10	10 / 10 / 10	95.6

Four additional tasks · Table III

Method	OpenOven	PickAndPlaceAndHugContainer	OpenFaucet	OpenTrashCan	Mean (%)
Ψ₀	7 / 5 / 4	7 / 6 / 3	3 / 3 / 4	7 / 9 / 9	55.8
DiT4DiT	10 / 10 / 9	7 / 8 / 8	5 / 6 / 7	6 / 7 / 5	73.3
Fast-WAM	1 / 1 / 0	0 / 0 / 0	2 / 1 / 0	2 / 2 / 1	8.3
VLA-JEPA	8 / 9 / 7	8 / 8 / 7	6 / 2 / 4	4 / 3 / 4	58.3
Ours	9 / 10 / 10	9 / 9 / 8	5 / 7 / 5	8 / 8 / 9	80.8

Table entries show successes at L0 / L1 / L2, out of ten trials each. DiT4DiT, Fast-WAM, VLA-JEPA and Ours are evaluated by us; the other baselines are reported in SIMPLE v1. Mobile P&P denotes LocomotionPickBetweenTables; Grasp denotes TabletopGrasp. BendPick and Grasp use motion-planning demonstrations.

02 · Real robot

Standard conditions

Unitree G1 · Seven tasks

Success over seven real-robot tasks, with ten episodes per task. We train and evaluate GR00T N1.7, Ψ₀ and DiT4DiT for the real-robot comparison.

Ψ₀: 32.9% mean; DiT4DiT: 62.9% mean; GR00T N1.7: 70.0% mean; Ours: 85.7% mean — Download SVG ↓

Per-task counts · Table IV

Task	Ψ₀	DiT4DiT	GR00T N1.7	Ours
Dispose of Trash	2/10	4/10	9/10	6/10
Collect Toy with Cart	6/10	8/10	6/10	6/10
Water Plant	6/10	8/10	8/10	10/10
Pack and Stow Power Strip	4/10	6/10	8/10	8/10
Load Washer	2/10	8/10	2/10	10/10
Retrieve Shirt from Drawer	2/10	4/10	8/10	10/10
Place Basket on Table	1/10	6/10	8/10	10/10
Mean (%)	32.9	62.9	70.0	85.7

03 · Real robot

Unseen object configurations

Unitree G1 · Seven tasks

Objects, positions and orientations change; demonstration-derived subgoal images and their order stay the same.

Ψ₀: 21.4% mean; DiT4DiT: 38.6% mean; GR00T N1.7: 48.6% mean; Ours: 65.7% mean — Download SVG ↓

Per-task counts · Table IV

Task	Ψ₀	DiT4DiT	GR00T N1.7	Ours
Dispose of Trash	2/10	1/10	4/10	4/10
Collect Toy with Cart	3/10	4/10	4/10	4/10
Water Plant	0/10	5/10	5/10	8/10
Pack and Stow Power Strip	3/10	5/10	5/10	6/10
Load Washer	3/10	4/10	6/10	7/10
Retrieve Shirt from Drawer	2/10	3/10	4/10	8/10
Place Basket on Table	2/10	5/10	6/10	9/10
Mean (%)	21.4	38.6	48.6	65.7

Data efficiency

Learning from fewer demonstrations

With 25, 50 and 100 demonstrations per task, our method achieves 48.0%, 52.7%, 84.0%, compared with 33.3%, 41.3%, 56.7% for Ψ₀.

Evaluated on OpenOven, PickAndPlaceAndHugContainer, OpenFaucet, OpenTrashCan and Mobile P&P.

Data efficiency: Ours 48.0%, 52.7%, 84.0%; Ψ₀ 33.3%, 41.3%, 56.7%. — Download SVG ↓

Ablations

Prediction and temporal consistency

The endpoint formulation encodes current and future observations separately. The feature-shift variant uses joint clip encoding and pools multiple predicted blocks for scoring.

Prediction formulation · Table V

Configuration	Six main (%)	Four additional (%)	Overall (%)
Feature-shift variant	91.7	73.3	84.3
Endpoint (Ours)	95.6	80.8	89.7

Component ablations use the four additional tasks, on which the full model has lower average success than on the six main tasks: 120 episodes per configuration. Within each ablation, demonstration data, training budgets, initial conditions and subgoals are shared.

Action consistency regularization · Table VI

Both models use visual-only scoring. ACR improves success from 55.0% to 65.8%.

Configuration	OpenOven	PickAndPlaceAndHugContainer	OpenFaucet	OpenTrashCan	Mean (%)
w/o ACR	7 / 7 / 6	7 / 6 / 6	3 / 4 / 3	6 / 6 / 5	55.0
w/ ACR	8 / 8 / 8	8 / 7 / 7	4 / 5 / 4	7 / 6 / 7	65.8

Temporal prior · Table VII

The joint prior improves success from 65.8% to 80.8%. Smoothness penalties and a factorized prior provide smaller gains.

Configuration	OpenOven	PickAndPlaceAndHugContainer	OpenFaucet	OpenTrashCan	Mean (%)
Visual only	8 / 8 / 8	8 / 7 / 7	4 / 5 / 4	7 / 6 / 7	65.8
1st-order smoothness	8 / 8 / 8	8 / 8 / 7	4 / 5 / 4	7 / 7 / 7	67.5
2nd-order smoothness	8 / 9 / 8	8 / 7 / 8	4 / 6 / 5	7 / 7 / 7	70.0
Factorized prior	8 / 9 / 9	8 / 8 / 8	4 / 6 / 4	8 / 7 / 7	71.7
Joint prior (Ours)	9 / 10 / 10	9 / 9 / 8	5 / 7 / 5	8 / 8 / 9	80.8

Deployment

Inference cost and deployment

Our task-success evaluations use the RTX 4090D listed in Table I. In an additional onboard deployment test, our method runs on the robot’s Jetson Orin NX (16 GB), averaging 12.84 GB system memory and 0.73 Hz across seven tasks.

Inference cost on SIMPLE · Table VIII

Method	GPU memory (GB)	Trainable params. (B)	Inference (Hz)
GR00T N1.6	6.8	1.62	15.46
DiT4DiT	21	2.35	3.8
Fast-WAM	28	6.7	5.2
Ours	7.2	3.0	9.1

Measured on an RTX 4090D. Hz denotes command chunks per second; timing for Ours includes sequence generation, visual prediction and prior scoring.

Changes in the scene

Qualitative results and generalization

View full figure ↗

Qualitative results for Water Plant on the left and Load Washer on the right, showing object state, grasping hand and task execution. — **Qualitative results: Water Plant (left) and Load Washer (right).** The robot uses its right hand in standard trials and its left hand with a reversed watering-can spout or a T-shirt placed on its left, completing both tasks. The left-hand T-shirt grasp does not appear in the demonstrations. Full figure ↗

Different objects, placements, orientations, and nearby distractors. Each image highlights one setting; select it to open the corresponding recording.

Implementation

Code

Training and inference code.

Coming soon.

Research data

Humanoid loco-manipulation dataset

2,600 demonstrations across seven tasks on Unitree G1.

View dataset

Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight