Robotic manipulation demands a seamless integration of perception, planning, and control within dynamic environments. Addressing these challenges necessitates expertise in spatial algebra, kinematics, image recognition, and motion planning. Real-world scenarios add further complexity due to infinite variability.
To tackle these issues, reinforcement learning is employed, a tool allowing shifting the focus from solving individual mentioned problems to specifying a goal and a reward function.
Reinforcement learning can be categorized into online and offline approaches: Online learning consists of algorithms where the agent interacts with the environment and receives reward based on his actions, thus converging to an optimal based on his own experience. On the contrary, offline algorithms are algorithms where a dataset of interaction is collected before training phase and the policy is being generated without the agent being able to interact with the environment.
Our goal was to train a robot to perform a simple specific task, pushing a cube to a certain location, using both methods discussed. The first step was to set up a training environment for the robot, we chose to use Issac Gym, which is a physics simulator for reinforcement learning environments. We created an environment consisting of a Panda arm and a cube on a table. The agent was configured to control the arm by supplying velocity vector on the XY plane. On each episode the cube and target were randomly placed within the agent’s reach.
At the first phase, the online training phase, we tried several reward functions: rewards based on the distance of the cube to target, rewards including the end effector to the cube and various rewards taken from literature. Neither of these generated the desired policy. We added a feature to the simulator allowing using previous positions. Using this feature, we created a reward based on the cube velocity towards the target goal. This reward function generated the intended policy and the cube was pushed towards the goal.
After successfully finishing the task the generated policy, we analyzed the policy generated when subset of the observations was fed to the agent. As we suspected when limiting the observations supplied, the efficiency of the policy is degraded, and it takes more steps to solve the task.
The second phase of was the offline learning process, Issac Gym has no support for offline learning, so the first step was to add needed integrations to Issac Gym: collecting datasets, performing offline training and offline agent model evaluation. Our design followed the industry standards, and our implementation is PyTorch compatible, allowing us to test any offline reinforcement learning algorithm implemented using PyTorch. Using these features, we collected a sample dataset using the online policy generated at the first phase, trained an agent using CQL, an such algorithm and evaluated the generated policy using our offline model evaluation feature within Issac Gym. The generated policy did push the cube in the right direction in many of the cases but almost never reached the target.
Finally, we reviewed the results of both the online and offline generated policies. Based on the results the online policy is more successful in performing the task defined: both success rate wise and steps needed.
We suspect that using a dataset from multiple sources and with fine-grained samples, and by further tuning the hyper parameters it may be possible to match the quality of the policies. But more work is needed to confirm this hypothesis.