A real scene.Rebuilt in 3D.
A robot, a chessboard, and sparse observations. Reconstructed in MuJoCo, then replayed against the real episode to make the differences inspectable.
- Reconstructed
- SO-101 robot & chessboard
- Grounded in
- Camera, joint state & telemetry
- Made inspectable
- 531 measured-state samples
Synchronized replay
The same episode, side by side.
26.55 seconds · one measured-state clock
The useful comparison: the arm path remains recognizable.
The simulated pawn moves 15 samples early, tips 102.1°, and finishes 33.946 mm from D2. The first major mismatch is at grasp and load—not the gross arm motion.
Retained diagnostic result
Upright, but still not transferred.
Moving the simulated start pose to the exact center of D1 kept the pawn upright at 0.0027° final tilt and reduced planar error—but it still ended 45.856 mm from D2. Two evaluator gates improved. Task success remains no. This was a diagnostic prior, not a recovered physical calibration.
Closer view
Why the grasp still looks different.
Callback-aligned display · not camera-calibrated
The saved software clock does not explain the visible delta. In simulation, the named-jaw midpoint closes 34.482 mm from exact D1, while first broad contact comes from another gripper mesh and the nearest named jaw remains 4.523 mm clear.
That makes gripper geometry and contact representation the leading retained-data explanation—not proof of an early physical close.
The project, briefly
Make real-to-sim gaps measurable.
Sim2Claw reconstructed a physical SO-101 robot and chessboard environment from limited robot, camera, and telemetry data. The measured robot state can be replayed in that scene, making object behavior easy to compare against the physical episode. The reconstruction is useful because its remaining gaps can be located and measured without quietly changing the robot policy.
Methods, timing correction, and receipts
The main pair shares 531 measured state samples and can be read as synchronized. The wrist pair cannot: its physical and simulated cameras are not calibrated. For this display, physical D405 time is simulation time + 6.475 s—the recorded 8.499 s action-start offset minus 2.025 s before the first written D405 frame. This corrects presentation chronology only.
Exact D1 centering is an owner-declared task prior, not a recovered physical calibration. The replay still fails the destination-center gate, so none of this proves transfer or general physics fidelity.