Deep Reinforcement Learning (DRL) is a thriving field in recent years as many breakthroughs were achieved in video games played by AI. Most notable is the Deep Q-Learning Network (DQN) architecture by DeepMind (acquired by Google) that solved many Atari games. In this project we will build on a recent paper “Shallow Updates for Deep Reinforcement Learning” (written by researchers from our faculty) which combines deep learning methods with “shallow” ones. Mainly, we will focus on adapting and implementing least-square techniques to modern deep learning algorithms such as Distributed Deep Deterministic Policy Gradients (DDDPG) and test the performance on different simulators and games.