ACT Phase Recap: What 4 Weeks of Robot Training ACTually Taught Us

We ran more than 100 physical evaluation trials, trained two neural networks from scratch, and discovered that the model with more sensor data performed worse. Four weeks of ACT imitation learning on the SO-101 robotic arm: here is what we found, what surprised us, and where we go next.

TL;DR

Over four weeks, we built a complete imitation learning pipeline for the SO-101 robotic arm using the LeRobot framework and the ACT algorithm. We collected 90 training episodes, trained RGB and RGBD policy variants, evaluated them across 60 physical trials, and then ran a full sim-to-real transfer experiment in IsaacSim. The headline results: RGB outperformed RGBD (50% vs 43.3%), a simulation policy scored 80% in IsaacSim but only 50% on the physical robot, and 10 real demonstrations closed half of that gap. The optimal strategy we identified is not simulation-first or real-data-first, but co-training on both simultaneously.

What this phase was about

In June 2026, we set up two SO-101 robotic arms in a Leader-Follower configuration and asked a straightforward question: can a robot learn to pick up a ball and drop it into a container just by watching a human do it?

Imitation learning (IL): a training approach where a robot policy learns from recorded human demonstrations, without explicit trajectory programming or reward functions. ACT (Action Chunking with Transformers): the imitation learning algorithm we used throughout this phase, which predicts entire sequences of future actions at once rather than computing one step at a time, reducing the compounding errors typical of classical step-by-step models.

The task we benchmarked was Pick, Lift and Place (PLP): grasp a ball from one of six workspace starting positions, lift it, and place it inside a container. Simple to describe, and demanding enough to expose every weakness in the pipeline.

You can check out here How we build an imitation learning setup.

Week 1: the hardware lesson came first

Before we collected a single useful training episode, a servo motor failed mid-recording. LeRobot interpreted the hardware error as a natural episode end and halted the session. We lost data.

The fix was straightforward in hindsight: restructure the pipeline so that each completed episode is saved and uploaded to HuggingFace immediately after it finishes, before the next begins. A session-level save assumes the session completes. Hardware does not respect that assumption.

Not just us: Sherry Chen documented a nearly identical pipeline breakdown on HuggingFace while training ACT on the same SO-101 hardware – camera disconnects, recalibration mishaps, and a gripper motor worn out by excessive force during teleoperation. Real-world data collection is messier than any tutorial suggests.

After building this per-episode checkpointing pipeline, we recorded 90 training episodes across 6 starting positions, 15 per zone, with intentional positional variation within each zone. One constraint held throughout: the environment stayed completely unchanged, same lighting, same camera positions, same background. Imitation learning models correlate visual input with joint actions, and environmental variation between episodes introduces noise the model cannot separate from signal.

We start episode recording by using this ▼
 

lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/tty.usbmodem585A0076841 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/tty.usbmodem58760431551 \
--teleop.id=my_awesome_leader_arm \
--display_data=true \
--dataset.repo_id=${HF_USER}/record-test \
--dataset.num_episodes=5 \
--dataset.single_task="Grab the black cube" \
--dataset.streaming_encoding=true \
# --dataset.vcodec=auto \
--dataset.encoder_threads=2

Want to know more? Read about our Week 1 of collecting 90 Training Episodes for Imitation Learning.

Week 2: two models, one open question

Training ran on a single NVIDIA RTX PRO 4000 Blackwell GPU for 50,000 steps. We trained two ACT policy variants: RGB-only (2.5 hours) and RGBD, which adds depth maps from two Intel RealSense cameras (5 hours, reflecting the larger per-sample payload). Both converged cleanly.

The hypothesis going into evaluation was that depth data would give the model a better sense of 3D space and improve grasp accuracy. It was a reasonable hypothesis. It was also wrong, or at least, conditionally wrong in ways we did not anticipate.

We start training by using this ▼

lerobot-record \
--robot.type=so100_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_robot \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
--display_data=true \
--dataset.repo_id=${HF_USER}/eval_act_your_dataset \
--dataset.num_episodes=10 \
--dataset.single_task="Your task description" \
--dataset.streaming_encoding=true \
--dataset.encoder_threads=2 \
# --dataset.vcodec=auto \
--policy.path=${HF_USER}/act_policy

Get to know more about Week 2 of Training ACT on RGB and depth data.

Week 3: depth won where sensors could see, and failed everywhere else

What is an Occlusion? It is a condition where an object falls outside a sensor’s reliable line of sight, causing noisy or incomplete depth readings. At position 5, the ball blended into the background rather than registering as a distinct surface at a measurable distance. A likely second factor was hand-eye calibration error, which causes the depth signal to introduce spatial noise rather than spatial information. An RGB model never encounters this noise, because it does not consume depth at all.

We ran 60 physical evaluation trials, 30 per model, 5 per starting position. The headline: RGB finished at 50.0%, RGBD at 43.3%. The per-position breakdown tells a more useful story.

At positions 0 and 2, both within the optimal field of view of the RealSense D435 camera, RGBD scored 100% and 80%. At position 5, at the workspace edge and at an unfavourable angle to both depth sensors, RGBD scored 20% while RGB scored 80%.

Why this matters: More sensors do not automatically produce a better policy. Each sensor contributes useful data only within a perceptual window defined by its physical placement and calibration quality. Outside that window, a sensor can degrade performance compared to a simpler model that ignores it entirely.

We evaluated RGBD by using this ▼
 

python scripts/eval_policy.py \
  --policy-path=xBerry/lerobot_act_policy \
  --repo-id=xBerry/eval_lerobot_policy_rollout \
  --root=./datasets/eval_lerobot_policy_rollout \
  --task="Pick the object and place it in the container" \
  --fps=15 \
  --num-episodes=10 \
  --episode-time-s=30 \
  --reset-time-s=15 \
  --robot-port=/dev/ttyUSB0 \
  --robot-id=white_arm \
  --robot-calibration-dir=./local_calibration/robots/so_follower \
  --resume

And evaluated RGB-only by using this ▼
 

python scripts/eval_policy.py \
  --policy-path=xBerry/lerobot_act_rgb_policy \
  --rgb-only \
  --repo-id=xBerry/eval_lerobot_policy_rollout \
  --root=./datasets/eval_lerobot_policy_rollout \
  --task="Pick the object and place it in the container" \
  --fps=15 \
  --num-episodes=10 \
  --episode-time-s=30 \
  --reset-time-s=15 \
  --robot-port=/dev/ttyUSB0 \
  --robot-id=white_arm \
  --robot-calibration-dir=./local_calibration/robots/so_follower

Read more about Week 3 – RGB vs RGBD evaluation, depth sensor analysis.

Week 4: 80% in simulation, 50% in reality

Collecting more physical demonstrations is the slow, expensive path to improving performance: every episode requires manual workspace reset, there is mechanical wear on the arm, and episodes cannot run in parallel. Simulation removes all three constraints.

We built a digital twin of the SO-101 setup in IsaacSim (NVIDIA’s physics simulator for robotics), matched the scene to within 10% visual difference, and applied domain randomisation: a technique that randomises simulation parameters at every episode reset, including lighting, object positions, and camera parameters, to broaden the training distribution so that real-world conditions fall within it rather than outside it. A policy trained purely on simulated data scored 80% in IsaacSim and 50% on the physical robot, a 30-point reality gap.

What is Fine-tuning?It is a continuing training of an already-trained model on a small new dataset, allowing it to adapt to new conditions without discarding prior learning.

Fine-tuning on just 10 real demonstrations raised physical performance to 65%, but reduced simulation performance from 80% to 75%. The real data pulled the model’s weights toward a different distribution: improved generalisation to the physical world at a measurable cost in idealised simulation conditions. The data points toward co-training on a carefully mixed sim and real dataset as the better strategy, rather than treating the two as sequential phases.

We ran the entire environment in a Docker container using this command ▼
 

xhost +
docker run --name teleop -it --privileged --gpus all \
  -e "ACCEPT_EULA=Y" -e "PRIVACY_CONSENT=Y" \
  -e DISPLAY --rm --network=host \
  -v /dev:/dev \
  -v /run/udev:/run/udev:ro \
  -v $HOME/.Xauthority:/root/.Xauthority \
  -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
  -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
  -v ~/.cache/huggingface/lerobot/calibration:/root/.cache/huggingface/lerobot/calibration \
  -v ~/Sim-to-Real-SO-101-Workshop:/workspace/Sim-to-Real-SO-101-Workshop \
  teleop-docker:latest

If you want to know more you can check how we did on Week 4 – sim-to-real transfer.

The moment nobody anticipated: when the ball escaped

What is the counter-intuitive fix? It is deliberately record failure during data collection. Push the ball, let it roll away, drop it outside the arm’s reach, and then steer the robot back to the starting position. These recovery episodes teach the model that ball-escaping is a recognisable state with a correct response: return to home and try again. A robot that recovers from a failed grasp is dramatically more reliable than one that thrashes.

One finding from the final evaluation report deserves its own section: Out-of-Distribution (OOD) failure.

What is Out-of-Distribution (OOD)? It is a condition where a model encounters a state during evaluation that never appeared in its training data. In imitation learning, OOD inputs cause policy breakdown, not graceful degradation, but chaotic, unpredictable joint movement.

We recorded only successful demonstrations – the ball was picked up, lifted, and placed correctly in every training episode. When the ball slipped from the gripper mid-episode during evaluation, the model had no learned response. It had never seen a slipping ball. The arm swung. It struck the table. The movements were entirely random.

Why this matters: An imitation learning policy is only as robust as the distribution it was trained on. Any state that never appeared in training will cause complete breakdown. Recovery demonstrations are not optional polish, they are a baseline requirement for deployable physical AI systems.

What would we do differently?

Looking back across four weeks, three lessons stand out.

Observation selection matters more than observation count.

We confirmed this twice: RGBD hurt performance at edge positions, and reducing the camera count in the simulated setup improved sim2real transfer (the robot was no longer training on observations it could not see during physical deployment). The right sensors, correctly calibrated and positioned, outperform a richer but noisier sensor suite.

Domain randomisation should cover mechanical properties, not just visual ones.

In Week 4 we randomised lighting and object positions. What we missed: servo friction, joint inertia, and material resistance. These dynamics-level differences between simulation and hardware compound across the full trajectory. Broader randomisation, including physical properties, would have produced a better initial transfer without additional real data.

System identification should precede domain randomisation.

Documenting every physical parameter of the robot before configuring the simulator, joint ranges, servo torque curves, camera intrinsics, reduces the baseline sim2real gap before randomisation begins. A well-characterised simulation paired with broad randomisation is more powerful than either approach alone.

What is next: reinforcement learning

What is reinforcement learning (RL)? It is a training paradigm where an agent learns an optimal policy through trial-and-error interaction with an environment, maximising a cumulative reward signal without needing labelled demonstration data.

The next phase of this project moves from imitation learning to reinforcement learning (RL).

Where ACT is bounded by what a human demonstrated, RL can explore strategies that never appeared in any recording. The trade-off is reward design: specifying what “good” looks like in a physical manipulation task requires careful engineering. We will compare off-policy algorithms (SAC, TD3) and on-policy algorithms (PPO) on the same Pick, Lift and Place task, across the sim2real boundary, to determine which algorithm family transfers better from IsaacSim to the physical SO-101.

We will also explore Cosmos3 by NVIDIA for dataset augmentation: generating photorealistic video variants from simulation outputs to reduce the domain gap without requiring additional physical data collection.

FAQ

What is ACT (Action Chunking with Transformers)? ACT is an imitation learning algorithm that uses a Transformer architecture to predict entire sequences of future robot joint positions simultaneously, rather than computing one action at a time. This approach reduces compounding prediction errors in long manipulation sequences.

Why did RGB outperform RGBD overall in the SO-101 evaluation? The RGBD model outperformed RGB only at positions where the RealSense depth sensors had a clear line of sight to the ball (positions 0 and 2, scoring 100% and 80%). At edge positions with occlusion or unfavourable sensor geometry, depth data introduced noise that the RGB model never encountered, pulling RGBD’s overall rate to 43.3% versus RGB’s 50.0%.

What is the reality gap in sim-to-real transfer? The reality gap is the performance difference between a policy evaluated in simulation and the same policy deployed on a physical robot. In our experiment the gap was 30 percentage points: 80% in IsaacSim and 50% on the physical SO-101, arising from differences in friction, lighting, and sensor noise that simulation could not fully replicate.

How many real demonstrations are needed to partially close the sim-to-real gap? In our experiment, 10 real demonstration episodes added to a simulation-trained ACT policy raised physical performance by 15 percentage points (from 50% to 65%), at a cost of 5 percentage points in simulation (from 80% to 75%). A co-training strategy mixing sim and real data from the start is expected to produce better results than sequential fine-tuning.

What is Out-of-Distribution failure in imitation learning? Out-of-Distribution (OOD) failure occurs when a robot encounters a state during evaluation that never appeared in its training data. In imitation learning, OOD inputs cause complete policy breakdown: the model has no learned response and outputs chaotic, unpredictable joint movements. Including deliberate failure and recovery scenarios in training data is the standard mitigation.

What is domain randomisation in robot learning? Domain randomisation is a training technique that randomises simulation parameters at every episode reset, including lighting, object positions, and camera parameters, so that the policy learns to handle a broad range of conditions rather than a single fixed scene. A policy trained with domain randomisation is more likely to transfer to physical hardware because real-world variation falls within its training distribution.