DDPG (Deep Deterministic Policy Gradient) is a well-known algorithm that uses Deep Reinforcement Learning techniques to find optimal policies for settings where the action space is continuous.
Recently, an extension to the DDPG algorithm for motion planning tasks was introduced, named DDPG-MP.
The DDPG-MP algorithm supplies two main improvements to DDPG that enable efficient exploration and learning, and therefore improves the performance of DDPG in many settings.
The authors of DDPG-MP demonstrate their algorithm using a robotic arm with a kinematic simulator – a choice of simulator which makes the model simpler, but which might not cover the optimal policy for the robot (in the real world), since it neglects the dynamics of the system during the simulation.
In this project we study the mentioned tradeoff in the DDPG-MP setting that was described in the paper, by evaluating DDPG-MP with a dynamics-based simulator. As expected, a dynamics-based simulator should improve the results of the algorithm since it is more similar to the real world, but on the other hand, it is much more complicated, therefore harder to implement.
The goal of our project was to Implement a semi dynamics-based environment, then run DDPG algorithm to get some baseline results and if possible, find conclusions relevant for further DDPG-MP research on dynamic-based simulation.
Our project is divided into two main phases:
- Evaluate how hard it is to fulfill DDPG-MP’s requirements in the “semi-dynamics” setting.
- Test the DDPG algorithm in the “semi-dynamics” setting.
We have reached several conclusions in our project:
- It is possible to create a semi-dynamical forward model that is known
- Using Kinematic simulator is not such a bad idea
- DDPG algorithm (vanilla\with HER) can converge with the dynamics-based simulator
- Using dynamics simulator makes the problem more complex and much harder to converge
- DDPG is very limited algorithm with many explorations’ problems
- Adding exploration schemes to the algorithm (such as HER), can improve its results and help solve harder tasks
- it is hard to reach optimal policy