In this project we will examine a complementary approach to Hierarchical Reinforcement Learning, namely using context free grammars. This approach allows the agent to commit to specific temporal structures specified by a formal language over actions. By using this method, we expect to improve the performance and sample efficiency of the learning process of the agent. It will provide the ability to impose safety constraints on an agent. Furthermore, the approach we introduce is a simple way to incorporate prior knowledge into the agent in a non-restrictive manner.