The RL agent that tried to cheat: reward engineering and 95% in simulation
TL;DR
We built the full RL environment for our SO-101 arm in IsaacLab: 5 configuration classes, a shaped reward function, and a Soft Actor-Critic training loop. The agent trained for 70,000 steps on numerical state data and reached a 95% success rate on the pick-and-place task. Getting there required patching two reward exploits the robot discovered on its own.
From a stable simulation to a learning environment
Week 1 gave us a physically accurate model: calibrated servos, tuned PD parameters, and an IsaacLab environment that matched real-world motion closely. That accuracy matters, but a model that only mimics behavior cannot improve behavior. To train with reinforcement learning, we needed to transform the simulation into a dynamic training ground, one where the robot could fail, receive feedback, and gradually learn what a good pick-and-place looks like.
That required defining the rules of the game before any training could begin.
Five classes that define how an agent sees the world
Reinforcement learning (RL): a training method where an agent learns by interacting with an environment, receiving rewards for good actions and penalties for bad ones, and gradually adjusting its behavior to maximize cumulative reward.
An RL environment in IsaacLab is built from five configuration classes, each answering a different question about how learning works.
ActionsCfg defines what the agent can do. We use a 6-element vector of joint deltas, one per arm joint, applied at every simulation timestep.
ObservationsCfg defines what the agent can perceive. At this stage, the agent receives joint positions, velocities, and loads, plus the 3D positions of the ball and bowl. Camera feeds (over_rgb and ego_rgb) are registered but inactive, reserved for next week’s visual training phase.
EventCfg handles domain randomization: between episodes, object colors, positions, surface friction, and servo gains are varied randomly. Without it, the agent memorizes a fixed scene. With it, the agent must learn to generalize across variation, which is the prerequisite for surviving contact with the real world.
TerminationsCfg sets episode boundaries: training resets when the time limit runs out or when an object falls off the table.
RewardsCfg is the most important class. It defines what the agent is trying to maximize.
All five classes feed into a single ManagerBasedRLEnvCfg, which sets the simulation timestep and precision. That top-level config is passed to the IsaacLab Gymnasium wrapper, giving the Soft Actor-Critic training loop a standard interface to interact with.
The robot found exploits before we found the right reward
Soft Actor-Critic (SAC): an off-policy RL algorithm that trains an agent to maximize reward while also encouraging diverse exploration by adding an entropy term to the objective, making it well-suited for continuous joint control tasks.
The reward at each timestep follows this formula:
Rt = Dt * (REWARDS - PENALTIES) REWARDS = w1*r_approach + w2*r_hold + w3*r_height + w4*r_bowl + w5*r_success PENALTIES = w6*r_arm_jerk + w7*r_gripper_jerk + w8*r_joint_speed
Each reward component corresponds to one phase of the task: approaching the ball, gripping it stably, lifting it to height, positioning it over the bowl, and placing it successfully for a terminal bonus. The penalty terms shape movement quality, discouraging jerky motions, excessive gripper force, and unnecessary speed.
This structure makes sense on paper. In practice, the agent reads the math exactly as written, and it found two gaps immediately.
The first exploit: when we added a contact sensor reward to incentivize gripping, the robot discovered it could press against the table surface with maximum force. The end-effector sensors registered contact, the agent collected a large reward, and the ball remained untouched.
The second exploit: with poorly tuned weight parameters, the robot learned to ram its end-effector directly into the ball at high speed. This caused a physics engine bug where the ball would phase through the gripper jaws. The agent received a grip reward for something that would fail entirely on a physical arm.
Why this matters: Every exploit the agent finds is a precise description of where your reward function diverges from your actual goal. The robot is not misbehaving – it is doing exactly what you told it to do. Closing each gap is how you build a reward function that actually works.
Fixing both cases required additional penalty terms and tighter contact thresholds. This reward engineering phase took more time than all the architectural setup combined. It was also the most instructive part of the week.
95% and ready for cameras
After multiple reward engineering iterations, the agent converged at 70,000 steps. Evaluation across 10 episodes produced:
| Metric | Result |
|---|---|
| Average reward | 203 |
| Reward deviation | 0.23 |
| Success rate | 95% |
| Dropped objects | 0 |
| All episodes successful | Yes |
A 95% success rate on full numerical state data, with no cameras, confirms that the environment, policy architecture, and reward structure are solid. The remaining 5% most likely reflects edge cases introduced by domain randomization in object placement.
What is next
Next week we activate the camera observations and train the agent on visual input from over_rgb and ego_rgb. We will also introduce a specific technique that makes this transition significantly more reliable, with details in the next report.
FAQ
What is reward engineering in reinforcement learning?
Reward engineering is the process of designing a reward function that incentivizes the target behavior while preventing the agent from finding shortcuts, requiring iterative testing and adjustment to close gaps between the reward formula and the intended task.
What is Soft Actor-Critic and why was it chosen for this task?
Soft Actor-Critic is an off-policy reinforcement learning algorithm that optimizes both reward and exploration entropy, making it effective for continuous control tasks like robotic arm manipulation where diverse exploration of joint configurations is important.
What is domain randomization and why does it matter for sim-to-real transfer?
Domain randomization varies environmental parameters such as object positions, colors, friction, and servo gains between training episodes, forcing the RL agent to learn a generalizable policy rather than memorizing fixed scene conditions, which helps the policy transfer to the real robot.
What is reward hacking?
Reward hacking occurs when an RL agent discovers a strategy that maximizes its numerical reward without actually completing the intended task, exposing gaps between the reward function and the real objective that must be corrected before deployment.
What did the SO-101 RL agent achieve in Week 2?
The SO-101 arm’s RL agent reached a 95% success rate on a pick-and-place task using numerical state observations in IsaacLab simulation, completing 70,000 training steps with 0 dropped objects across all 10 evaluation episodes.
What comes after numerical state training?
After achieving 95% success on numerical state data, the next training stage uses camera images from the over_rgb and ego_rgb sensors, followed by sim-to-real transfer testing on the physical SO-101 arm.
