Deep reinforcement learning for partially observable markov decision processes

Abstract

Deep reinforcement learning has recently gained popularity owing to its many successful real-world applications in robotics and games. Conventional reinforcement learning faces a substantial challenge in developing effective algorithms for high-dimensional environments. The use of deep learning as a function approximator in reinforcement learning is a viable solution to overcome this challenge. Furthermore, in deep reinforcement learning, the environment is often thought to be fully observable, meaning that the agent can perceive the true state of the environment and so act appropriately in the current state. Most real-world problems are partially observable and the environmental models are unknown. Therefore, there is a significant need for reinforcement learning approaches to solve these problems, in which the agent perceives the state of the environment partially and noisily. Guided reinforcement learning methods solve this issue by providing additional state knowledge to reinforcement learning algorithms during the learning process, thereby allowing them to solve a partially observable Markov decision process (POMDP) more effectively. However, these guided approaches are relatively rare in the literature, and most existing approaches are model-based, which means that they require learning an appropriate model of the environment first. In this thesis, we present a novel model-free approach that combines the soft actor-critic method and supervised learning concept to solve real-world problems, formulating them as POMDPs. We evaluated our approach using modified partially observable MuJoCo tasks. In experiments performed on OpenAI Gym, an open-source simulation platform, our guided soft actor-critic approach outperformed other baseline algorithms, gaining 7∼20% more maximum average return on five partially observable tasks constructed based on continuous control problems and simulated in MuJoCo. To solve the autonomous driving problem, we focused on decision making under uncertainty, as a partially observable Markov decision process, using our guided soft actor-critic approach. A self-driving car was trained in a simulation environment, created using MATLAB/SIMULINK, for a scenario in which it encountered a pedestrian crossing the road. Experiments demonstrate that the agent exhibits desirable control behavior and performs close to the fully observable state under various uncertainty situations.