xBerry LeRobot Week 2: Training ACT on RGB and depth data

When we set up training for our SO-101 robot, we had one open question: does giving the model a sense of depth actually make it better at manipulation, or does it just double the training time for nothing? Week 2 was where we found out.

TL;DR – Two ACT models, 50,000 steps, zero problems

We ran ACT training for 50,000 steps using the LeRobot framework on a single NVIDIA RTX PRO 4000 Blackwell GPU. 2 models were trained: RGB-only (2.5 hours) and RGB+depth (5 hours).

Why we trained 2 ACT models: RGB vs. RGB+depth

What is ACT (Action Chunking with Transformers)? It is an imitation learning algorithm that predicts entire sequences of future robot joint positions simultaneously using a Transformer architecture, rather than one action at a time.

Before launching ACT training, we defined 2 distinct model variants to run a controlled comparison. The first model trains exclusively on color (RGB) images. The second model receives both color images and depth maps: per-pixel distance measurements that give the model a 3D spatial understanding of the scene. By training both variants on the same dataset with identical hyperparameters, we isolate the contribution of depth information to robot performance.

Datasets used for ACT training:

• RGB-only: xBerry/lerobot_act_rgb
• RGB+depth: xBerry/lerobot_act

How to start training:

lerobot-record \
--robot.type=so100_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_robot \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
--display_data=true \
--dataset.repo_id=${HF_USER}/eval_act_your_dataset \
--dataset.num_episodes=10 \
--dataset.single_task="Your task description" \
--dataset.streaming_encoding=true \
--dataset.encoder_threads=2 \
# --dataset.vcodec=auto \
--policy.path=${HF_USER}/act_policy

Why adding depth is important: Adding depth as an input modality is a common design choice in robot manipulation, but it roughly doubles training time. Measuring the actual performance delta tells us whether that overhead is justified for pick-and-place tasks — that comparison is the core question of Week 3 evaluation.

ACT training setup and hyperparameters

Moving on to the configuration: The visual backbone was ResNet-18 – a compact 18-layer convolutional neural network that extracts visual features from camera frames and feeds them into the ACT transformer.

We optimized the data using the AdamW algorithm – an adaptive gradient descent algorithm with distributed weight regularization that stabilizes training on noisy demo data.

The parameters are presented in the table below:

As you can see in the image, both learning curves converged cleanly to their minimums. The RGB model completed training in 2.5 hours, while the RGB+depth model required 5 hours due to the larger data payload per sample.

Hardware and DataLoader configuration

With the hyperparameters locked, how fast training actually runs comes down to the hardware. The entire training ran on a single in-office xBerry workstation, sized to eliminate CPU-GPU data transfer bottlenecks.

Below are the specifications in the table:

What is DataLoader? It is a tool that helps you efficiently load and pre-process data from a dataset and prepare it for input into a machine learning model during training.

In addition, we also configured DataLoader so that the GPU was constantly fed with data throughout the 5-hour training session:

• num_workers = 12 – 12 parallel data loading processes running concurrently,
• prefetch_factor = 4 – each worker pre-fetches 4 batches in advance,
• persistent_workers = true – worker processes stay alive between epochs, eliminating per-epoch restart overhead.

How to configure DataLoader:

1. Creation of LeRobot’s dataset:

LeRobotDataset.create(
    args.repo_id,
    args.fps,
    root=args.root,
    robot_type=robot.name,
    features=dataset_features,
    use_videos=True,
    image_writer_processes=args.image_writer_processes,
    image_writer_threads=args.image_writer_threads,
    batch_encoding_size=args.video_encoding_batch_size,
    vcodec=args.vcodec,
    streaming_encoding=args.streaming_encoding,
    encoder_queue_maxsize=args.encoder_queue_maxsize,
    encoder_threads=args.encoder_threads,
)

2. Training loop data loading configuration:

lerobot-train \
--dataset.repo_id=$(HF_USER)/$$ds \
--dataset.root=$(HOME)/robotic_arm/datasets/$$ds \
--policy.type=act \
--output_dir=$(CURDIR)/outputs/train/$${ds}_policy \
--job_name=$${ds}_policy \
--policy.device=$(TRAIN_DEVICE) \
--policy.repo_id=$(HF_USER)/$${ds}_policy \
--policy.push_to_hub=true \
--policy.vision_backbone=resnet18 \
--policy.pretrained_backbone_weights=ResNet18_Weights.IMAGENET1K_V1 \
--policy.chunk_size=16 \
--policy.n_action_steps=16 \
--steps=$(TRAIN_STEPS) \
--batch_size=8 \
--num_workers=12 \
--prefetch_factor=4 \
--persistent_workers=true \
--save_freq=25000 \
--eval_freq=25000 \
--log_freq=100 \
--wandb.enable=$(WANDB_ENABLE); \

Why DataLoader configuration is important: DataLoader configuration is one of the most overlooked bottlenecks in vision-based robot training pipelines. With 12 workers and prefetch_factor=4, CPU data preparation stays ahead of GPU consumption – especially important for the RGB+depth run, which processes a larger per-sample payload.

Real-time monitoring with WandB

With training running for up to 5 hours, we needed a way to confirm convergence without stopping the process. We logged progress in real time to WandB (Weights & Biases), an experiment tracking platform that records loss curves, metrics, and system statistics throughout model training.

The LeRobot framework integrates with WandB natively, pushing metrics to the dashboard every 100 steps. This allowed the team to confirm convergence from the xBerry office without interrupting the training process, both loss curves were trending cleanly downward by the 10,000-step mark.

Both trained models are now publicly available on HuggingFace:

• RGB-only policy: xBerry/lerobot_act_rgb_policy
• RGB+depth policy: xBerry/lerobot_act_policy

If you want to check how we started with imitation learning in our company – see How We Built an Imitation Learning Setup.

What’s next: evaluation on the physical robot

Both models are trained, converged, and published. Week 3 will focus on deploying both ACT models to the physical SO-101 arm and running structured evaluation, executing the pick-and-place task and recording success rates across both input configurations.

FAQ

What is ACT (Action Chunking with Transformers)?

ACT is an imitation learning algorithm for robot manipulation that uses a Transformer architecture to predict entire sequences of future joint positions simultaneously, rather than one action per step. This approach improves temporal consistency in robot motion and reduces compounding prediction errors in long manipulation sequences.

What is the difference between RGB-only and RGB+depth training?

An RGB-only model receives standard color camera images as input. An RGB+depth model additionally receives depth maps – per-pixel distance measurements – which provide spatial awareness of object positions in 3D space. The RGB+depth model takes roughly twice as long to train.

How long does ACT model training take with LeRobot?

With 50,000 training steps and batch size 8, the RGB-only ACT model trained in 2.5 hours on an NVIDIA RTX PRO 4000 Blackwell GPU. The RGB+depth model required 5 hours due to the larger per-sample data payload.

What hardware is needed to train a LeRobot ACT model?

The xBerry team trained ACT on a single NVIDIA RTX PRO 4000 Blackwell GPU with 24 GB VRAM and 8,960 CUDA cores, paired with a 20-core CPU and 64 GB RAM. No cloud compute was required for the full 50,000-step training run.

What is WandB and why use it in robot training?

WandB (Weights & Biases) is an experiment tracking platform that logs training loss, metrics, and hardware stats in real time during model training. The LeRobot framework integrates natively with WandB, enabling live convergence monitoring without stopping the training loop.