We’re entering the RL Phase: Measure First, Simulate Second
The 30-percentage-point gap between simulation and reality we found in our ACT phase had a precise cause: the simulation was built on assumptions, not measurements. Week 1 of reinforcement learning did not start with training. It started with a tape measure, a telemetry logger, and six servos that needed to be understood before they could be modelled.
TL;DR
We kicked off the reinforcement learning phase of the xBerry Robotic Arm project by rebuilding the simulation from scratch – this time with measured parameters instead of defaults. Using IsaacLab, we ported the full scene (SO-101 arm, black bowl, yellow ball, white table), calibrated two cameras with physically measured FOV and distance values, and ran system identification on all 6 STS3215 servos to extract friction, torque limits, stiffness, and damping. The result is a simulation that matches the physical setup at approximately 90%. The environment is ready for RL training.
From imitation to reinforcement: a different kind of learning
In the ACT phase, we taught the SO-101 arm to pick up a ball by showing it 90 human demonstrations. The robot learned to copy what it saw. Reinforcement learning (RL) takes a different approach: instead of demonstrations, the agent receives a reward signal for every action it takes and discovers effective behaviour through trial and error. There are no human trajectories to imitate – only a task definition, an observation space, and a reward function.
The practical consequence is that RL requires far more simulation time. Where ACT needed 90 real recordings, RL needs hundreds of thousands of simulated episodes to converge on a policy. A simulation that diverges from the physical robot by 30 percentage points – as ours did in the ACT phase – would produce a policy that fails immediately on hardware. Getting the simulation right is not a preliminary step. It is the work.
Building the IsaacLab scene
IsaacLab – NVIDIA’s open-source framework for training robot learning agents in simulation, built on top of IsaacSim. It provides structured environments for manipulation, locomotion, and control tasks, with built-in support for physics-accurate simulation and domain randomisation.
We ported the full Pick, Lift and Place scene into IsaacLab: the white SO-101 arm, a black bowl, a yellow ball, and a white table — all as USD assets. Physics validation confirmed correct robot–ball–bowl interaction: the ball rolls, bounces, and rests realistically against the bowl. The camera setup mirrors the physical lab exactly: an overhead OVER camera at approximately 134 cm above the workspace and a close-range EGO camera mounted near the gripper at 28 cm.
Both cameras were configured from physical measurements, not simulator defaults:
| Camera | Distance | FOV | Resolution |
|---|---|---|---|
| OVER | ~134 cm | 30.9° | 640 × 480 |
| EGO | ~28 cm | 67.3° | 1920 × 1080 |
# OVER camera CameraCfg(width=640, height=480, horizontal_fov=30.9) # EGO camera CameraCfg(width=1920, height=1080, horizontal_fov=67.3)
The EGO camera – mounted on the arm itself – gives the model a first-person view of the gripper approaching the target. Its wide 67.3° field of view captures the hand-to-object relationship at close range, which is the critical frame for grasp decisions.
System identification: measuring instead of guessing
System identification (SysID) is the process of measuring a physical system’s dynamic properties and using those measurements to configure a model. In robot simulation, it means extracting the real servo parameters – friction, torque limits, position stiffness, velocity damping and encoding them directly into the physics engine rather than using manufacturer defaults or estimates.
This is where the ACT phase reality gap was rooted. The IsaacSim setup used approximated servo parameters. The physical robot behaved differently at every joint because real servos have unique friction profiles, gear ratios, and current limits that only telemetry reveals.
We ran telemetry logging on all 5 active servos of the SO-101 (model STS3215) across a 50-second motion sequence, recording position, velocity, and load at every timestep. The resulting curves revealed each joint’s individual friction signature, velocity saturation point, and load distribution – data that cannot be read from any datasheet.
Those measurements fed directly into the SO101_CFG configuration in IsaacLab. Each joint now has a physically-grounded stiffness (Kp) and damping (Kd) value derived from telemetry, not assumed:
SO101_CFG = ArticulationCfg(
actuators={
# Gear 1/191 — 34.4 N·m
"rotation": ImplicitActuatorCfg(
joint_names_expr=["Rotation"],
effort_limit_sim=30,
stiffness=55,
damping=0.7,
),
# Gear 1/345 — 62.1 N·m (highest torque joint)
"pitch": ImplicitActuatorCfg(
joint_names_expr=["Pitch"],
effort_limit_sim=30,
stiffness=30,
damping=0.8,
),
# Gear 1/191 — 34.4 N·m
"elbow": ImplicitActuatorCfg(
joint_names_expr=["Elbow"],
effort_limit_sim=30,
stiffness=25,
damping=0.7,
),
# Gear 1/147 — 26.5 N·m
"wrist_pitch": ImplicitActuatorCfg(
joint_names_expr=["Wrist_Pitch"],
effort_limit_sim=30,
stiffness=12,
damping=0.5,
),
# Gear 1/147 — 26.5 N·m
"wrist_roll": ImplicitActuatorCfg(
joint_names_expr=["Wrist_Roll"],
effort_limit_sim=30,
stiffness=7,
damping=0.5,
),
# Gear 1/147 — 26.5 N·m
"gripper": ImplicitActuatorCfg(
joint_names_expr=["Jaw"],
effort_limit_sim=30,
stiffness=4,
damping=0.3,
),
},
)
The actuator model uses a PD controller — proportional-derivative control — to generate torque for each joint. The output torque τ follows the equation:
τ = Kp × (q_desired − q) + Kd × (q̇_desired − q̇)
where Kp (stiffness) drives the joint toward the target position and Kd (damping) suppresses the oscillations that high Kp values can cause. Every joint in the SO-101 has a different gear ratio and load profile, so each Kp/Kd pair was tuned individually from telemetry data. The highest stiffness (Kp = 55) belongs to the Rotation joint; the gripper uses the lowest (Kp = 4) to avoid damaging objects on contact.
Why this matters: A simulation that uses measured servo parameters will produce a policy that transfers better to hardware. The sim2real gap is not primarily a visual problem – it is a dynamics problem. Get the physics right before worrying about the pixels.
What is next: defining the RL task
The environment is built and validated at approximately 90% visual match to the physical setup. Week 2 will focus on defining the three core components of the RL task: the action space (what joint commands the agent can issue), the observation space (what the agent sees — camera frames, joint positions, gripper state), and the reward function (how the agent learns that placing the ball in the bowl is the goal). These three design choices will determine everything about the quality of the policy that emerges from training.
FAQ
What is reinforcement learning and how does it differ from imitation learning?
Reinforcement learning (RL) is a training paradigm where an agent learns an optimal policy through trial-and-error interaction with an environment, receiving a reward signal for each action taken. Unlike imitation learning, which requires human demonstrations, RL discovers effective behaviour independently, which allows it to find strategies that no human demonstration would cover.
What is IsaacLab?
IsaacLab is NVIDIA’s open-source framework for robot learning in simulation, built on IsaacSim. It provides structured environments, domain randomisation tools, and physics-accurate simulation designed for training RL and imitation learning policies that transfer to physical hardware.
What is system identification in robotics?
System identification (SysID) is the process of measuring a physical system’s dynamic properties — friction, torque limits, gear ratios, inertia and using those measurements to configure a simulation model. Accurate SysID reduces the sim-to-real gap by ensuring the simulation physics match the behaviour of the physical hardware.
What is a PD controller in servo actuation?
A proportional-derivative (PD) controller generates joint torque as τ = Kp × (q_des − q) + Kd × (q̇_des − q̇). The stiffness term Kp drives the joint toward the target position; the damping term Kd suppresses oscillations caused by high Kp values. Each joint in the SO-101 has independently tuned Kp and Kd values based on physical telemetry measurements.
Why does camera FOV matter for sim2real transfer?
A camera configured with incorrect FOV produces images with different perspective distortion than the physical camera, causing the policy to receive observations in training that do not match deployment conditions. Physically measuring FOV and distance for each camera eliminates this source of sim2real error before training begins.
What will Week 2 cover? Week 2 defines the three components of the RL task environment: the action space, the observation space, and the reward function. These design choices determine the quality and transferability of the resulting policy.
