Recorded MimicX tennis motion rendered on court; successive poses arranged from left to right

MimicX: Policy-in-the-Loop Supervision Refinement
for Video-Driven Humanoid Motion Tracking

Shuaijun LiuChenglong ZhangXuhao LiuFeiyang YouYifan LiaoShuyang HaoChaozhe ZhangChengyu WuZhen SunNingxin Su*

The Hong Kong University of Science and Technology (Guangzhou)

* Corresponding author: Ningxin Su

Motions & scenes

01 / 07
Football Juggling Foot-ball contact · 4× slow motion
Human and G1 parkour motion across an outdoor collision scene
Parkour Motion sequence
Track Run: source frame, human reconstruction and G1 reference evolution
Track Run Source to robot reference
Stair Ascent: successive human and G1 stages in the source scene
Stair Ascent Source to robot reference
Platform Jump: reconstructed and retargeted phases from takeoff to landing
Platform Jump Source to robot reference
Forest Traversal: human observation, recovered body and articulated G1 reference
Forest Traversal Source to robot reference
Tennis Swing Registered policy motion

Recorded execution

Original input videos and recorded policies share the source timeline and a stable source-anchored camera. BeyondMimic and SONIC show median-error recordings from their valid complete runs; BeyondMimic retains its best verified checkpoint. MimicX shows its previously selected best recording. Blue references denote Fixed Reference, purple BeyondMimic, green SONIC and coral MimicX; solid robots are the actual policies. SONIC recordings have no startup support during playback. MimicX includes additional policy continuation; these selected demonstrations are not an equal-budget statistical comparison.

Tennis Swing

Robot only
Human inputTennis Swing
Fixed ReferenceBest valid recorded rollout
BeyondMimicRecorded rollout
SONICReleased policy
MimicXWith reference overlay

Football Juggling

Robot only
Human inputFootball Juggling
Fixed ReferenceBest valid recorded rollout
BeyondMimicRecorded rollout
SONICReleased policy
MimicXWith reference overlay

Dance Sequence

Robot only
Human inputDance Sequence
Fixed ReferenceBest valid recorded rollout
BeyondMimicRecorded rollout
SONICReleased policy
MimicXWith reference overlay

Kung Fu

Robot only
Human inputKung Fu
Fixed ReferenceBest valid recorded rollout
BeyondMimicRecorded rollout
SONICReleased policy
MimicXWith reference overlay

Execution in collision scenes

Six human inputs and robot executions in task-equivalent collision scenes, each with its own timing. Parkour combines MimicX with MPC; the other clips use their selected policies. These demonstrations and the earlier policy recording details are separate from the controlled paper comparisons below.

Track Running

Robot only
Human inputSource excerpt
MimicX97 steps; reference overlay

Accepted world-space refinement; the full 97-step reference is preserved.

Stair Ascent

Robot only
Human inputSource excerpt
MimicX449 steps; reference overlay

The verified complete policy is retained for the full 449-step ascent.

Forest Traversal

Robot only
Human inputSource excerpt
MimicX404 steps; reference overlay

Accepted world-space refinement; the full 404-step traversal is preserved.

Platform Jump

Robot only
Human inputSource excerpt
MimicX58 steps; reference overlay

The source excerpt is 3.54 s; the selected jump reference and policy span 1.16 s. The complete verified jump policy is retained.

Parkour

Full clip
Human inputSource excerpt
MimicX + MPC269 steps; joint control

MimicX + MPC, locally retimed 5.38 s: robot joint control in a diagnostic control demonstration.

Stair Descent

Robot only
Human inputSource excerpt
MimicX240 steps; reference overlay

Complete descent and lower landing, verified on three additional reset seeds. Recording details.

From human video to robot execution

MimicX turns a human demonstration into a humanoid skill, then uses the robot's execution to refine its supervision. Motion reconstruction and retargeting establish a reference; rollout feedback guides what the policy learns next.

  1. Paper-selected original tennis video frameInput videoHuman demonstration
  2. Paper-selected human motion reconstructionHuman reconstructionGVHMR / SMPL-X
  3. Paper-selected learned G1 policy in simulationPolicy in simulationG1 motion tracking
  4. The same policy motion rendered in the reconstructed tennis sceneReconstructed sceneRegistered policy motion

Execution becomes supervision

A registered motion reference establishes the task. Policy feedback determines how that task is practiced and rewarded.

MimicX method: rollout diagnosis conditions training objectives and failure-window practice, followed by verified policy continuation
Policy-in-the-loop supervision refinement. Rollout errors localize difficult motion intervals and affected body regions; bounded candidates coordinate tracking objectives and practice.

One refinement pass

  1. Read the rollout

    Localize where execution departs from the demonstration.

    When
    Failure windows
    Where
    Root, limbs and contacts
    What
    Pose and dynamics errors

  2. Change the supervision

    Coordinate task-aware tracking objectives and targeted practice; continue from the current policy.

  3. Keep verified progress

    Repeated execution decides the update. Completion, failure horizon and tracking quality take priority over reward.

The selected policy starts the next pass.

Tracking quality and execution

Direct comparisons evaluate learned policies on the same demonstrated motion. Controlled continuation and collision-scene experiments examine how refinement changes execution.

Tennis body error: MimicX 5.54 cm, BeyondMimic MjLab 13.65 cm, Fixed Reference 15.14 cm and SONIC 23.60 cm
Same motion, closer trackingTennis Swing, 518 frames at 50 Hz. Points show three runs; diamonds and bars show means and sample SD. Root-local body error is 59.4% lower than BeyondMimic (MjLab).
Body and root p95 errors decrease across five collision-scene tasks
Refinement across collision scenesBodyRootError reductions over matched pre-failure prefixes. One training seed, three evaluation seeds per task; equal continuation budgets.
First-failure horizon for all twelve paired task and continuation-seed trials
Longer executionFixed ReferenceMimicXAll twelve matched task-seed pairs. Each point reports the worst repeat; 801 denotes no failure within the 800-step evaluation.
Five feedback workload timings per executor, with median markers
Shorter feedback cyclesFive fixed-policy workloads per executor. Every dot is a measured wall time; ticks show medians. HLoop is 1.945 times faster than sequential execution.
All 3000 unsmoothed body-error measurements across four supervision methods and three seeds
How the policy learnsFixed ReferenceFailure CurriculumTask-Aware RefinementMimicXTennis continuation, 250 iterations per run. Thin traces show individual seeds, bold traces their means; all 3,000 logged measurements are shown without smoothing. The vertical axis is logarithmic.

Direct comparison: common reference and evaluation clock, three continuation seeds for trained trackers and three executions of released SONIC. BeyondMimic uses its MjLab implementation with 250 continuation iterations; MimicX uses the frozen verified output. These execution results are separate from equal-budget core ablations.

Four-task comparison

MimicX
Means over three continuation seeds. Body error is the temporal mean of aligned worst-body distance; horizon is the worst first-failure step across repeats, with H + 1 for an unfailed rollout.
TaskMethodStrict successHorizon, stepsBody error
Tennis SwingFixed Reference11.1%322.00.241 m
Tennis SwingMimicX100.0%801.00.157 m
Football JugglingFixed Reference0.0%53.70.286 m
Football JugglingMimicX0.0%427.30.244 m
Dance SequenceFixed Reference0.0%193.30.270 m
Dance SequenceMimicX0.0%341.30.244 m
Kung Fu SequenceFixed Reference0.0%97.70.249 m
Kung Fu SequenceMimicX0.0%196.00.141 m
Numerical data & evaluation protocols

All core tasks and component methods are retained in the underlying records. The selected Dance and Kung Fu outputs retain the protected incumbent policy.

Metric definitionsData provenanceHistorical adapter comparison

Faster feedback. The same decision.

MimicX-HLoop overlaps GPU rollout and CPU diagnosis along a dependency graph. Selection uses all evaluation evidence, independent of execution order.

Execution-priority acceptance and heterogeneous scheduling preserve the selected policy
Candidate policies are repeatedly evaluated against the current policy. Selection prioritizes task completion, the failure frontier and tracking guards: accepted candidates start the next refinement pass, while rejected candidates leave the current policy unchanged. HLoop overlaps independent GPU rollouts and CPU diagnosis as their dependencies are satisfied, then assembles the complete evidence before applying the same selection rule.

One feedback workload

Eight rollouts, eight CPU diagnoses and one report-selection step: the same four fixed policies, evaluated with two seeds.

Sequential
Complete each job before starting the next.
Bulk-synchronous
Parallel jobs within each stage; barriers between stages.
MimicX-HLoop
Start each CPU diagnosis when its rollout finishes, overlapping independent jobs.

Feedback wall time

Sequential241.221 s
241.221 seconds
Bulk-synchronous122.258 s
122.258 seconds
MimicX-HLoop124.017 s
124.017 seconds

Selection waits for the complete evidence. All five repetitions preserve the sequential outcome.

Medians of five runs; two GPUs and CPU workers. Fixed checkpoints and zero PPO updates isolate feedback scheduling from policy learning.

HLoop protocol