xBerry LeRobot Week 1: Collecting 90 Training Episodes for Imitation Learning

Collecting LeRobot training data for imitation learning sounds like a solved problem — until a servo fails mid-session and the recording evaporates. Here is exactly how we gathered 90 demonstration episodes for our ACT model on the SO-101 arm, what broke, and the pipeline that saved everything.

TL;DR – LeRobot Training Data Collection: Week 1 Results

We completed Week 1 of our internal R&D robotics project by collecting 90 training episodes for a Pick, Lift and Place task. Data was captured from 4 RGB cameras and 2 depth maps across 6 starting positions, 15 episodes each. A servo fault and intermittent communication drops forced us to build an auto-save pipeline with per-episode HuggingFace upload, and the result was more robust than any session-level approach could have been.

How Many Training Episodes Does an Imitation Learning Policy Need?

What is Imitation Learning? It is a robot training approach where the model learns from human-controlled demonstrations rather than from explicit programmed trajectories.

The number of demonstration sequences required for a functional imitation learning policy depends on task complexity. For a Pick, Lift and Place (PLP) task – in which a robot grasps a target object, lifts it and places it in a designated container – the typical range is 60–200 episodes.

The exact number scales with several independent factors:

• Number of starting positions – more spatial zones means more variation the model must generalise across.
• Number of objects – each additional object introduces new grasp configurations. Adding one more object requires recording k additional demonstration episodes.
• Number of variable factors – box position, object colour, surface texture and similar environmental variables each multiply combinatorial complexity. Every new variable added to the setup requires k more demonstrations.

What are the consequences of too few episodes? Simple. The policy under-learns and fails to generalize to unseen positions. However, too many repetitions of the same trajectory cause the policy to overlearn and memorize the path instead of understanding the task.

Why this matters: Defining the variable matrix before recording begins saves hours of re-training. A miscalculated episode count in either direction is expensive to fix after data collection is complete.

Our LeRobot Training Data Collection Setup: 1 Ball, 6 Zones, 90 Episodes Total

For Week 1 we chose a deliberately constrained setup to establish a clean baseline before adding complexity:

• 1 object – a single green ball.
• 1 box – the placement target.
• 6 starting positions – 6 spatial zones from which the ball was placed before each episode began.

We recorded 15 episodes per starting position, yielding 90 total episodes in the training dataset.

Within each starting zone, ball placement was intentionally varied – corners, centre, left edge, right edge, and intermediate positions. This in-zone positional variation is critical: a model trained only on centre-placed objects will fail when the ball lands even a few centimetres off-centre. The model needs to learn the task, not the coordinate.

Critical constraint – environment consistency: All 90 episodes were recorded in a single, unchanged environment. Lighting conditions, camera positions, table surface, and background remained fixed throughout the entire recording session. This is not a preference – it is a hard requirement for imitation learning. Environmental variation between episodes introduces noise that the model cannot distinguish from meaningful signal.

Recording Protocol: 30-Second Episodes, 4 Cameras, 2 Depth Maps

How to start episode recording:

lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/tty.usbmodem585A0076841 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/tty.usbmodem58760431551 \
--teleop.id=my_awesome_leader_arm \
--display_data=true \
--dataset.repo_id=${HF_USER}/record-test \
--dataset.num_episodes=5 \
--dataset.single_task="Grab the black cube" \
--dataset.streaming_encoding=true \
# --dataset.vcodec=auto \
--dataset.encoder_threads=2

Each demonstration episode followed a fixed protocol:

• Episode duration: ~30 seconds – sufficient to complete the full Pick, Lift and Place sequence at a controlled pace.
• Reset period: 10 seconds between episodes to restore the environment: moving the ball from the final resting point back to the next starting position.
• Visual data: 4 RGB cameras providing standard colour video streams recorded simultaneously.
• Depth data: 2 depth maps from Intel RealSense cameras, capturing 3D spatial information alongside the colour frames.

The SO-101 arm operates in Leader–Follower configuration – the human operator physically manipulates the leader arm by hand, and the follower arm mirrors every joint movement in real time.

 

In the same time, the LeRobot framework – the open-source Hugging Face library that handles multi-camera synchronisation, dataset structuring, and policy training – records all camera streams and joint position data synchronously, producing a multi-modal dataset ready for training.

 

The setup worked, until the hardware didn’t.

What Broke: 2 Hardware Problems That Stopped Recording

We encountered 2 distinct problems during the recording sessions.

Problem 1 – Servo fault on the end effector. The servo motor controlling the gripper (end effector – the terminal joint that makes contact with objects) on the SO-101 reported an error signal to the controller during a live recording. LeRobot interpreted this servo error as a natural episode-end signal and halted recording automatically. The session terminated mid-episode without warning.

Problem 2 – Communication interruptions. Intermittent delays in the communication layer between the arm controller and the recording workstation caused LeRobot to similarly abort the session – treating the communication delay as an error condition.

How We Solved It: Auto-Save Pipeline with Per-Episode HuggingFace Upload

Instead of treating the entire recording session as a single monolithic operation, we restructured the pipeline to save each completed episode individually – at the moment it finishes, before moving to the next one. Each saved episode is uploaded directly to HuggingFace – the machine learning platform hosting our dataset – immediately after the 30-second recording window closes.

How to resume recording or add additional episodes:

lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/tty.usbmodem585A0076841 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
--teleop.type=so101_leader \
--teleop.port=/dev/tty.usbmodem58760431551 \
--teleop.id=my_awesome_leader_arm \
--display_data=true \
--dataset.repo_id=${HF_USER}/record-test \
--dataset.num_episodes=5 \
--dataset.single_task="Grab the black cube" \
--dataset.streaming_encoding=true \
# --dataset.vcodec=auto \
--dataset.encoder_threads=2
--resume=true # wznowienie nagrywania

This architecture resolved both problems simultaneously:

• A servo fault or communication drop after episode N leaves episodes 1 through N-1 fully intact on HuggingFace. No data is lost.
• Recording can be resumed after any interruption without restructuring the dataset.
• Adding further episodes later – if 90 proves insufficient for training convergence – requires only continuing the existing dataset on the same HuggingFace repository.

The faulty servo was replaced with a new unit. The SO-101 is a modular robot – each joint uses a standardised servo motor module that can be removed and replaced in minutes without disassembling the rest of the arm. The swap required no full-arm recalibration, and recording resumed quickly.

Why this matters: Building resumable, episode-level checkpoint pipelines from day one is not overengineering – it is the minimum viable reliability standard for physical AI systems. Hardware will fail. The pipeline must be indifferent to it.

Key Lesson: Episode-level checkpointing is non-negotiable

The most transferable insight from Week 1 has nothing to do with robot arms specifically: physical AI systems require episode-level checkpointing with cloud backup from day one.

Session-level saves assume the session completes. Hardware does not respect that assumption. Any pipeline that can lose an hour of recordings because of a single dropped connection is not a pipeline, it is a gamble.

What’s Next: Training the ACT Model on 90 Episodes

What is ACT? It is a Transformer-based imitation learning algorithm that predicts entire future action sequences simultaneously rather than computing one action step at a time.

With 90 episodes uploaded to HuggingFace, Week 2 moves to model training. The next step is writing the training pipeline to teach the SO-101 arm to execute Pick, Lift and Place autonomously using ACT (Action Chunking with Transformers). This chunk-based prediction approach reduces the compounding error accumulation that plagues step-by-step methods in long manipulation sequences.

If you’re interested in our internal robotic project and would like to start one on your own – don’t hesitate and Contact us right now!
 
 

FAQ

What is imitation learning in robotics?

Imitation learning is a robot training approach in which a policy – the model that controls the robot — learns by observing human-controlled demonstrations rather than being programmed with explicit motion trajectories or reward-based reinforcement signals.

How many training episodes are needed for a Pick, Lift and Place task?

A Pick, Lift and Place imitation learning task typically requires between 60 and 200 demonstration episodes, depending on the number of starting positions, objects involved, and other variable factors in the setup. Each new variable added to the task requires recording additional k demonstration episodes.

What is the ACT algorithm used in LeRobot?

ACT (Action Chunking with Transformers) is an imitation learning algorithm that uses a Transformer architecture to predict entire chunks of future robot actions simultaneously rather than one step at a time, reducing the compounding prediction errors typical of step-by-step methods in long manipulation sequences.

What is the SO-101 robotic arm?

The SO-101 is a modular 6-degree-of-freedom robotic manipulator designed for imitation learning research. The SO-101 operates in Leader–Follower configuration, where a human operator physically controls one arm and a second arm mirrors the joint movements in real time for training data recording.

Why must all training episodes be recorded in the same environment?

Imitation learning models learn from the statistical correlation between visual inputs and robot joint actions. If the environment changes between episodes – different lighting, camera angles, or background elements – the model cannot reliably map what it sees to the correct actions, introducing noise that degrades policy quality.

What is LeRobot and how does it handle data collection?

LeRobot is an open-source robotics research framework developed by Hugging Face that manages synchronised multi-camera data recording, dataset structuring and compression, neural network policy training, and model evaluation. During recording, LeRobot monitors hardware state signals and will abort a session if it detects an error condition – making robust pipeline design essential.

If you want to be updated about our progress with LeRobot project, follow us on Linkedin – xBerry R&D House.