xBerry Robotic Arm Project Week 4: sim-to-real transfer – 80% in simulation, 50% in reality and what 10 demonstrations fixed
We trained a robot policy in simulation and scored 80%. On the physical SO-101 arm, the same policy scored 50%. Then we fine-tuned on 10 real demonstrations and reached 65%, but dropped back to 75% in simulation. Improving in one world has a cost in the other.
TL;DR
Week 4 was our first full sim-to-real transfer experiment using IsaacSim and IsaacLab inside the LeRobot ecosystem. A policy trained purely on simulated data scored 80% in simulation and 50% on the physical robot, a 30-point reality gap. Fine-tuning on 10 real demonstrations raised the physical score to 65% while reducing simulation performance to 75%. The data suggests that the optimal strategy is co-training on both sources, not choosing one over the other.
From real data to simulation: why we made the switch
What is Sim-to-real transfer (sim2real)? This is a training strategy where a robot policy is trained inside a physics simulator and then deployed on physical hardware, reducing the cost and risk of training directly on a real robot.
What is Digital twin? It is a virtual replica of a physical robot and its environment, modelled to match real kinematics, dynamics, and sensor geometry closely enough that a policy trained in the virtual copy has a reasonable chance of working on the physical original.
In Week 3, our best physical policy reached 50% on the Pick, Lift and Place task using RGB-only input. Collecting more physical demonstrations is the obvious next step, but it is also the slow and expensive one: every episode requires resetting the workspace manually, there is mechanical wear on the arm, and episodes cannot be parallelised. Simulation removes all three constraints.
If you want to read more about our activities in week 3, check out xBerry LeRobot Week 3: RGB vs RGBD – 60 trials, one surprising winner.
For our digital twin we used IsaacSim: NVIDIA’s physics simulator for robotics, running inside IsaacLab: a framework built on IsaacSim that integrates with the LeRobot training pipeline. The task was the same Pick, Lift and Place scenario, with a ball and bowl replacing the object-and-box setup from earlier weeks.
The full Week 4 pipeline was: collect dataset in simulation, train ACT policy, evaluate in simulation, transfer policy 1:1 to physical robot, evaluate on hardware, fine-tune on a small sample of real demonstrations, evaluate both environments again.
Getting the simulation running
IsaacSim requires a specific combination of CUDA versions, NVIDIA drivers, and Omniverse libraries. Rather than installing these directly on the host machine (which creates fragile, hard-to-replicate configurations), we ran the entire environment in a Docker container:
xhost + docker run --name teleop -it --privileged --gpus all \ -e "ACCEPT_EULA=Y" -e "PRIVACY_CONSENT=Y" \ -e DISPLAY --rm --network=host \ -v /dev:/dev \ -v /run/udev:/run/udev:ro \ -v $HOME/.Xauthority:/root/.Xauthority \ -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \ -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \ -v ~/.cache/huggingface/lerobot/calibration:/root/.cache/huggingface/lerobot/calibration \ -v ~/Sim-to-Real-SO-101-Workshop:/workspace/Sim-to-Real-SO-101-Workshop \ teleop-docker:latest
The key flags: –gpus all gives the container full GPU access for physics rendering, –network=host shares the host network so the container can communicate with USB-connected hardware, -v /dev:/dev mounts physical devices including the robot’s USB ports, and the huggingface/lerobot/calibration volume mount ensures robot calibration data survives container restarts.
Before collecting any demonstrations, we calibrated both arms and ran motion tests: the physical leader arm drove the follower arm inside the simulation, and we verified that joint positions matched before recording any episodes.
Scene matching and domain randomization
For sim2real to have any chance of working, the simulation must look sufficiently similar to reality. We targeted a visual difference below 10%, measured by comparing camera frames between the physical setup and the simulated render. This required matching the SO-101 joint chain and ranges of motion, the geometry and material properties of the ball and bowl, and the position, angle, and field of view of all cameras.
What is Domain randomization? It is a technique that randomises simulation parameters at the start of each episode (object position, lighting, camera angle) to artificially broaden the training distribution so that real-world conditions fall within it rather than outside it.
Visual matching alone is not enough. A policy trained on one fixed scene learns a very narrow data distribution.
We randomised lighting at every episode reset using a function that samples a random exposure level, colour temperature, and HDRI texture from a library of indoor environment maps:
def randomize_sky_light(env, env_ids, exposure_range, temperature_range, textures_root, asset_cfg=None):
exposure = math_utils.sample_uniform(*exposure_range, (1,), device="cpu").item()
temperature = math_utils.sample_uniform(*temperature_range, (1,), device="cpu").item()
textures = glob.glob(os.path.join(textures_root, "*.exr"))
texture = textures[torch.randint(0, len(textures), (1,)).item()]
prim.GetAttribute("inputs:exposure").Set(exposure)
prim.GetAttribute("inputs:colorTemperature").Set(temperature)
prim.GetAttribute("inputs:texture:file").Set(Sdf.AssetPath(texture))
Ball and bowl positions were randomised similarly, with a small probability of placing the ball already inside the bowl to simulate partially completed episodes.
The results: a 30-point gap, and how 10 demonstrations closed half of it
With scene matching and domain randomization in place, we trained the ACT policy on simulated data and evaluated it in both environments.
Stage 1: policy trained on simulation data only
| Evaluation environment | Successes | Rate |
|---|---|---|
| Simulation (IsaacLab) | 16/20 | 80% |
| Physical robot (1:1 transfer) | 10/20 | 50% |
Reality gap: the performance difference between simulation and physical hardware that arises because simulation never perfectly replicates real-world friction, lighting variation, and sensor noise. Even with careful scene matching, the policy encountered a data distribution on the real robot that differed enough from its training distribution to drop 30 percentage points.
Stage 2: fine-tuning on 10 real demonstrations
| Evaluation environment | Before fine-tuning | After fine-tuning |
|---|---|---|
| Simulation (IsaacLab) | 80% (16/20) | 75% (15/20) |
| Physical robot | 50% (10/20) | 65% (13/20) |
What is Fine-tuning? It is continuing training of an already-trained model on a small new dataset, allowing it to adapt to new conditions without discarding what it learned from the original data.
Adding just 10 real episodes raised physical performance by 15 percentage points. It also reduced simulation performance by 5 percentage points. The most likely explanation: the real data pulled the model’s weights toward a different distribution (different lighting, different grasp dynamics), which improved generalisation to the physical world at the cost of some precision in the idealised simulation conditions.
Why this matters: Sim2real is not a one-way transfer. A model must find a compromise between two data distributions. Fine-tuning on real data closes part of the reality gap, but the improvement in one world comes at a measurable cost in the other. The optimal strategy is co-training: a carefully balanced mix of simulated and real data, not a sequential “simulate first, then switch.”
Closing the reality gap further
Domain randomization reduces the gap by broadening the training distribution, but dedicated methods exist for measuring and correcting it directly.
SAGE estimates and corrects the mismatch in dynamic parameters (joint friction, inertia) between the simulation model and the real robot’s physical behaviour.
GapONet trains a separate network that models the gap itself: the difference between what the simulation predicts and what the robot actually does, allowing policy correction without full retraining.
Cosmos generates additional synthetic data variants from existing collected data, increasing training diversity without requiring new physical demonstrations.
What is next
Next week we are modifying the simulation scene to better match physical reality, and transitioning the policy from imitation learning to reinforcement learning (RL). The comparison of imitation learning and RL in the sim2real context is one of the open questions in robotic manipulation: RL can explore strategies that never appeared in human demonstrations, but it also requires much more careful reward design and is harder to stabilise across the sim-to-real boundary.
FAQ
What is sim-to-real transfer in robotics?
Sim-to-real transfer is the technique of training a robot policy inside a physics simulator and deploying it on a physical robot, reducing data collection costs and hardware wear. The challenge is that simulation never perfectly replicates physical reality, creating a performance gap called the reality gap.
What is the reality gap in robot learning?
The reality gap is the performance difference between a policy evaluated in simulation and the same policy running on a physical robot. It arises from differences in friction, lighting, sensor noise, and object dynamics that simulation cannot fully replicate.
What is domain randomization?
Domain randomization is a training technique that randomises simulation parameters (lighting, object positions, camera angles) at the start of each episode so that the policy learns a broad distribution of conditions rather than a narrow fixed scene, making it more likely to generalise to real-world variation.
What is IsaacSim?
IsaacSim is NVIDIA’s physics simulator for robotics, used to create high-fidelity virtual environments for training and testing robot policies. IsaacLab is a framework built on IsaacSim that integrates with robot learning pipelines such as LeRobot.
How many real demonstrations are needed for sim-to-real fine-tuning?
In our experiment, 10 real demonstrations added to a simulation-trained ACT policy raised physical performance by 15 percentage points (from 50% to 65%), suggesting that even a very small real-world dataset can partially close the reality gap without requiring hundreds of physical episodes.
What is co-training in sim-to-real robotics?
Co-training is a strategy that mixes simulated and real training data in a single training run, rather than training sequentially on simulation and then fine-tuning on real data. The goal is to find a policy that generalises to both distributions simultaneously, reducing the performance trade-off observed when optimising for one world at the expense of the other.
