Sim2Claw / physical-to-simulation replay
Reality on the left. The simulator has to earn the right.
Sim2Claw turns physical SO-101 recordings into synchronized, evaluator-scored simulator evidence. The goal is not a plausible robot video. It is a replay whose camera, motion, contact, and consequences survive independent checks.
The comparison bay
One held-out episode, frame for frame.
The physical C922 recording and frozen renderer-native candidate share a 5 fps evaluation timeline. No physical pixels are embedded in the simulated side.
This is the strongest honest visual result, not “nearly identical dynamics.” The evaluator passed its frozen numeric visual gates, then a human-readable sanity audit found that the board/background dominated the score while the robot and outside-board scene still differed materially.
Two episodes · 246 frames · visual metric only
One outcome-informed object-dynamics episode · quarantined
No simulator promotion or physical transfer
What succeeded
Three real-to-sim wins worth showing.
Each is useful on its own. None is allowed to quietly inherit the authority of the next proof class.
Synchronized divergence footage
A 531-sample physical trajectory, simulator trace, and C922 recording can be viewed on one bound timeline. The resulting film makes the first contact and object-consequence mismatch inspectable instead of hiding it in an aggregate score.
- Timeline
- 20 fps / 26.55 s
- Actions changed
- No
Held-out renderer metric
A frozen camera, workcell, mesh renderer, and monotone response passed six of six numeric visual gates plus seven of seven integrity gates on two evaluator-held-out episodes.
- Frames
- 246
- Mean similarity
- 0.84445
Natural object-dynamics endpoint
With the robot driven through an unchanged 531-row measured-state trajectory, the simulated pawn evolved under natural dynamics and ended upright within 4.26 mm of the physical destination. The result reproduced without a refit, and all seven terminal outcome gates passed.
- Final tilt
- 0.003864°
- Other-piece motion
- 0.000041 mm
What did not close
The footage is closer. The dynamics are not yet the same.
The best result remains useful precisely because the evaluator records where it fails.
Robot appearance, operator/cable structure, and the scene outside the board remain visibly different.
No bilateral named-jaw contact; pawn motion and support loss arrive early; sample-260 tilt is 26.1°.
The 4.26 mm endpoint was selected on the same episode. It is not held-out calibration or predictive physics.
Twin fidelity remains 0/6 and strict task closure 0/11. Training, promotion, and transfer stay false.
Project brief
A proof loop for embodied systems.
The problem
A simulation can look persuasive while disagreeing with the physical robot at the exact moment contact matters. Commands, measured motion, rendered pixels, object consequences, and physical authority are different claims.
The system
Sim2Claw binds SO-101 recordings, six-joint state and action streams, MuJoCo replay, camera registration, browser inspection tools, frozen splits, and independent evaluator receipts into one traceable workflow.
The contribution
The project makes failure legible. Exact action hashes prevent helpful edits; held-out evaluators prevent self-promotion; terminal negatives preserve the next experiment instead of being rewritten as demos.
- 01Record
Physical video, timestamps, measured state, requested actions.
- 02Bind
Hash episode identity, action tensors, runtime, and evaluator contract.
- 03Replay
Keep actions fixed while isolating geometry, timing, and contact gaps.
- 04Evaluate
Score frozen consequences on a separately owned CPU/fp32 path.
- 05Decide
Promote, quarantine, or stop with a machine-readable receipt.
Evidence ledger
The result and its boundary travel together.
| Evidence | Observed result | What it proves | What it does not |
|---|---|---|---|
| OR26 · synchronized film | 531 samples on one registered timeline | Real/sim divergence can be inspected frame by frame | Physics success or metric camera calibration |
| OR50 · dynamics endpoint | 7/7 terminal gates; 4.26 mm center error | A same-episode natural object-dynamics outcome can be reproduced exactly | Event parity, held-out calibration, or transfer |
| OR91 · held-out visual | 6/6 numeric and 7/7 integrity gates | A frozen renderer clears its preregistered visual metric | A semantic “same video” match |
| OR133C · stop boundary | Residual identity unidentifiable from retained evidence | The current corpus cannot resolve the remaining actor/cable residual | A guessed fit or retrospective fidelity improvement |
Next credible step: a fresh capture with independent actor/cable observability, metric camera inputs, and rigid jaw-bound landmarks—then a preregistered held-out dynamics comparison.
Inspect the current stop-boundary closeout