In this project, we consider the problem of RL in continuous state spaces with sparse reward. In continuous state spaces tabular methods cannot be applied directly. Thus, approximation approaches, such as function approximation methods, are used. These approaches often lack in exploration compared to classical tabular RL algorithms and thus perform badly in sparse reward domains. A possible solution is by performing a discretization of the space.
We explored different discretization methods for different settings. Our goal was to find a method which balances computational complexity and accuracy. Upon a given discretized new space we acted according to R-max, which has optimal exploration.
We explored and present methods for three settings:
- Continuous space with known dimensions offline discretization
- Continuous space with known dimensions online discretization.
- Continuous space with unknown, possibly infinite, dimensions online discretization.