DDPG (Deep Deterministic Policy Gradient) is a well-known algorithm that uses Deep
Reinforcement Learning techniques to find optimal policies for settings where the action space
is continuous (e.g. robotic arm manipulation). Recently, an extension to the DDPG algorithm for
motion planning tasks was introduced, named DDPG-MP (MP – Motion Planning). The DDPG-MP
algorithm supplies two main improvements to DDPG that enable efficient exploration and
learning, and therefore improves the performance of DDPG in many settings.
Our project focuses on the first improvement suggested by the DDPG-MP paper: model-based
actor update. This new approach replaces the standard actor’s loss (which is the Q-function) by
a bellman equation that is equivalent to the Q-function, using the environment’s known
forward model and a reward model that is learned from the environment.
We aim to improve the actor’s learning by using different techniques to improve the proposed
bellman loss function:
1. actor’s loss with different rollout lengths:
In the original DDPG-MP paper the authors assumed that it is not necessary to expand
the bellman loss beyond a single step lookahead. We evaluate this assumption by
comparing the agent’s performance using different rollout lengths.
2. Ensemble reward model for actor loss:
We require good reward estimation in order to use the bellman loss. One way to reduce
an estimator’s variance, while maintaining the same bias, is to use an ensemble of
estimators instead of a single estimator.
We will use an ensemble of reward models in order to achieve better predictions for the
reward in the actor’s loss, and evaluate the contribution of this improvement