Our project examines a new deep exploration method in reinforcement learning.
Reinforcement learning is an area of machine learning. It is about taking suitable action to maximize reward in a particular situation. There are two factors in the learning process: the agent and the environment. In reinforcement learning, the agent observes the current environment state, takes an action and gets a reward from the environment. The agent learns by observing the reward of its actions at each state.
In such cases, in model-free reinforcement learning, one of the biggest problem that can arise is the balance between exploration vs. exploitation.
The exploration part of the learning process is for the agent to familiarize himself with the environment. The agent is gathering data along the process, with no batch of initial information to learn from, as opposed to supervised learning. In other words, the problem in this type of learning process is that the data the agent gathered is affected by his actions and vice versa. Therefore, sometimes it is worth taking different actions, namely preforming exploration.
The exploitation part in the learning process is where the agent starts exploiting what he had learned. The agent chooses the actions that will maximize the reward from the environment.
The agent’s policy dictates the balance between exploration and exploitation in the learning process. Finding the best method for exploration is critical. Given an agent implemented by a certain algorithm, changing the exploration method can greatly improve the agent’s performance, while not changing the general algorithm.
The main challenge the agent faces is finding states that the agent had not visited, and quickly figuring out if they lead to high rewards. This is a hard task because the agent never truly knows if there are more states to explore, and whether or not the agent should spend resources on the exploration of those states.
Most of the current deep exploration approaches which target neural-network environments rely on assumptions and evaluations of the data the agent had seen and not directly on that data, e.g DORA, Pseudo-counts.
The Columbus algorithm was designed to target complex problems that required sophisticated exploration. Columbus implements a more accurate approach that combines deep exploration. The main advantage of our exploration method is that we use the data with minimal manipulations. This guarantees more accurate results.