Deep reinforcement learning has shown tremendous success over the last decade, but a big whole is yet to be filled until such methods can reliably and safely be deployed into real-life machines. Robust reinforcement learning aims to ensure such safety but is computationally expensive. In this work we focused on implementing an algorithm which is both computationally cheap and has robust properties which can be implemented in the real world. We rely on previous work of tabular version of R2 algorithm and extend it to infinite state-space environments using deep learning. We test the algorithm empirically against robust algorithms and against the baseline tabular R2 and show that it is indeed computationally cheap as a vanilla deep learning algorithm, but more robust.