xBerry LeRobot Week 3: RGB vs RGBD – 60 trials, one surprising winner
We ran 60 physical evaluation trials comparing RGB and RGBD to test whether adding depth data to our ACT policy would improve performance. At position 5, the depth-trained model succeeded once in 5 attempts. The RGB-only model succeeded 4 times out of 5.
TL;DR
We compared two ACT policies on the SO-101 follower arm: one trained on 4 RGB cameras, one trained on the same cameras plus 2 RealSense depth streams. Over 60 trials across 6 workspace positions, RGB finished at 50.0% and RGBD at 43.3%. Depth helped significantly in zones where the sensors had a clear line of sight, and actively hurt performance in zones where the sensors could not reliably read the object. The takeaway: depth is a conditional improvement, not an automatic one.
The assumption we were testing
What is RGBD? It’s a camera input format that combines standard RGB colour data with a depth channel from a sensor such as the Intel RealSense, giving the model a spatial distance estimate for every pixel.
After training both model variants in Week 2, we were ready to find out whether the extra sensor data actually helped the robot. The intuition behind RGBD is compelling: a depth stream gives the model a sense of three-dimensional space rather than just flat colour, which should help it judge grip distance and approach angle more accurately. Our setup used two Intel RealSense cameras (D435 and D455) feeding depth maps alongside the standard colour feed from all 4 cameras.
The question was simple: would the richer input produce a better manipulation policy?
Running evaluation on physical hardware
Given that both models were trained and saved to HuggingFace Hub, the next step was physical evaluation on the SO-101 arm. We used eval_policy.py, the LeRobot evaluation script that loads a trained policy, runs it directly on the robot, records a success or failure for each episode, and saves results to a dataset.
eval_policy.py: the LeRobot script for deploying a trained policy onto a physical robot, recording outcomes per episode, and uploading results to a HuggingFace dataset for analysis.
We ran two evaluation sessions: one for the RGBD model and one for the RGB-only model. The RGBD variant loads all 6 camera streams (4 colour + 2 depth) including a depth filter pipeline (decimation, disparity, spatial, temporal, and hole-filling filters). The RGB-only variant uses the —rgb-only flag, which starts the RealSense sensors in colour mode only, skips depth pipeline initialisation, and reduces session startup time.
RGBD evaluation command:
python scripts/eval_policy.py \ --policy-path=xBerry/lerobot_act_policy \ --repo-id=xBerry/eval_lerobot_policy_rollout \ --root=./datasets/eval_lerobot_policy_rollout \ --task="Pick the object and place it in the container" \ --fps=15 \ --num-episodes=10 \ --episode-time-s=30 \ --reset-time-s=15 \ --robot-port=/dev/ttyUSB0 \ --robot-id=white_arm \ --robot-calibration-dir=./local_calibration/robots/so_follower \ --resume
RGB-only evaluation command:
python scripts/eval_policy.py \ --policy-path=xBerry/lerobot_act_rgb_policy \ --rgb-only \ --repo-id=xBerry/eval_lerobot_policy_rollout \ --root=./datasets/eval_lerobot_policy_rollout \ --task="Pick the object and place it in the container" \ --fps=15 \ --num-episodes=10 \ --episode-time-s=30 \ --reset-time-s=15 \ --robot-port=/dev/ttyUSB0 \ --robot-id=white_arm \ --robot-calibration-dir=./local_calibration/robots/so_follower
The –resume flag, used only in the RGBD session, lets the script detect an existing local dataset and continue from the last saved episode after an interruption.
A successful trial was defined as the robot picking up the ball and placing it inside the container within the 30-second episode window.
Each starting position was tested 5 times, for 30 trials per model variant, totalling 60 trials.
Check out also: How we trained ACT on RGB and depth data.
Results: depth wins in some zones, loses badly in others
The headline numbers say RGB won. But the per-position breakdown reveals a much more interesting story.
| Starting position | RGB successes | RGB rate | RGB-D successes | RGB-D rate |
|---|---|---|---|---|
| Index 0 | 2/5 | 40% | 5/5 | 100% |
| Index 1 | 2/5 | 40% | 1/5 | 20% |
| Index 2 | 3/5 | 60% | 4/5 | 80% |
| Index 3 | 1/5 | 20% | 1/5 | 20% |
| Index 4 | 3/5 | 60% | 1/5 | 20% |
| Index 5 | 4/5 | 80% | 1/5 | 20% |
| Total | 15/30 | 50.0% | 13/30 | 43.3% |
Positions 0 and 2 sit directly within the optimal field of view of Camera 2 (RealSense D435). At those positions, RGBD scored 100% and 80% respectively, while RGB scored 40% and 60%. When the depth sensor has a clear, unobstructed line of sight to the object, its distance estimates are clean and the model uses them effectively.
Position 5 is the edge of the workspace, far from Camera 2 and at an unfavourable angle to Camera 3. There, RGBD scored 20% and RGB scored 80%. The depth model failed four times as often.
Why depth backfired at the workspace edge?
Occlusion: a condition where an object falls outside a sensor’s direct line of sight, either blocked by another object or placed at an angle where the sensor’s beam cannot reach it effectively.
Position 5 put the ball near the boundary of what the RealSense sensors can read reliably. At that geometry, the depth image is noisy or incomplete: the ball blends into the background rather than registering as a distinct surface at a measurable distance.
A second likely factor is hand-eye calibration. Hand-eye calibration: the process of aligning the coordinate frame of a camera with the coordinate frame of a robot’s end-effector, so that “what the camera sees at these pixel coordinates” maps correctly to “where the robot’s gripper should move in 3D space.” An error in this calibration means the depth signal introduces spatial noise rather than spatial signal. An RGB-only model never encounters this noise, because it does not consume depth at all.
Why this matters: Adding more sensors to a robot does not automatically produce a better policy. Each sensor contributes useful data only within a specific perceptual window defined by its physical placement and calibration. Outside that window, the sensor can introduce noise that a simpler model would never encounter.
What this means for sensor design?
The RGB model generalised more evenly across all positions precisely because it had no localised failure mode. A colour camera covers the full workspace with roughly uniform quality. A depth sensor has a geometry-dependent coverage pattern, and any zone that falls outside optimal range becomes a liability for a model trained to trust depth data.
Two directions are worth exploring from here. The first is selective depth masking: disabling depth input dynamically for episodes where the starting position falls outside the reliable sensor zone, so the policy falls back to RGB behaviour at the edges. The second is targeted data collection: gathering additional training episodes specifically from edge positions and training a dedicated policy (or a residual policy layer) on depth data from those zones, rather than applying a single model uniformly across the workspace.
What is next?
What is IsaacSim? It is a NVIDIA’s physics simulator for robotics, which lets teams train robot policies in a virtual environment before transferring them to physical hardware.
Next week we are moving to simulation. We plan to train a policy in IsaacSim and then attempt sim-to-real transfer onto the physical SO-101 arm. The open question is whether training in simulation can close the performance gap on edge positions where real sensor coverage is imperfect, and how much of what the robot learned from physical demonstrations survives the transition to a virtual training environment.
FAQ
What is eval_policy.py in LeRobot?
eval_policy.py is the LeRobot evaluation script that runs a trained policy directly on a physical robot, records success or failure per episode, and saves results to a HuggingFace dataset for analysis.
Why did the RGBD model perform worse overall than RGB?
The RGBD model outperformed RGB only in workspace positions where the RealSense sensors had a clear line of sight to the object (positions 0 and 2). In edge zones where depth readings were noisy or incomplete, RGBD scored significantly lower, bringing its overall rate to 43.3% versus RGB’s 50.0%.
What is hand-eye calibration in robotics?
Hand-eye calibration is the process of precisely aligning the coordinate frame of a camera with the coordinate frame of a robot’s end-effector, so that the robot can accurately translate what the camera sees into physical movement targets.
What is sim-to-real transfer?
Sim-to-real transfer is the technique of training a robot policy inside a physics simulator and then deploying that policy on a physical robot, reducing the cost and risk of training directly on hardware.
Does depth data always improve robot manipulation policies?
Depth data improves manipulation policies only in workspace zones where the depth sensor has reliable line of sight to the object. In zones outside optimal sensor geometry, depth data can introduce noise that reduces performance compared to simpler RGB-only models.
What is the –rgb-only flag in LeRobot eval_policy.py?
The –rgb-only flag in LeRobot’s eval_policy.py tells the script to initialise RealSense cameras in colour mode only, skipping the depth pipeline. This reduces startup time and loads only the 4 RGB streams used during RGB model training, ensuring the evaluation matches the model’s training input exactly.
