Typical algorithms for solving reinforcement learning (RL) problems, are built on an assumption of a stationary environment (modeled as a stationary MDP), meaning the agent is learning how to act in an environment in which the action chosen in each state is not time dependent. However, one can think of many everyday life problems that occur in non-stationary environments, which change over time. Such problems were discussed in former articles (link to article), that suggest taking the time as part of the current state, thus forcing artificial stationarity, and allowing to solve non-stationary problems with the help of well-known methods.
This project discusses problems in which the time dependency is periodic, tries to examine them from a different angle, and research a new method for solving non-stationary periodic problems without forcing them into a stationary model. Furthermore, the project shows that this method improves the results. The suggested solution relies on the DQN algorithm, and searches for periodicity through an estimation of Fourier series coefficients for different frequencies. The solution has been tested for several kinds of periodicity, which are reflected in different elements of the environment, and provided the best results in all of them.