A generalized deep reinforcement learning based controller for heading keeping in waves

Abstract

Reinforcement Learning (RL) is a machine learning method where a learner (the agent) tries to maximize a reward by learning how to act under different environmental circumstances. The agent looks at the state of its environment (through the state vector), takes an action, and then gets a reward and the next state of its environment. The agent improves its action-taking strategy (policy) with every action it experiments with. RL methods have been used for many decision-making problems including control problems with promising results. Unlike many traditional control methods, a model-free RL doesn't need any environment dynamics to operate. This is especially beneficial for problems where the model dynamics are non-linear or not well-known. However, classical controllers are still the most used method of control for maritime applications. Heading-keeping is a maritime control problem where a controller's objective is to keep the heading (yaw) angle of a vehicle constant. Generally speaking, the industry standard is to use traditional feedback controllers such as PID for this problem. This study focuses on designing a generalized RL controller for the heading-keeping problem in waves. The study compares the designed RL controller to a traditional controller in terms of yaw error and rudder usage and observes that the designed RL-based controller performs better than the used traditional controller. The first iterations of the RL agent had many issues. Unlike traditional controllers, the RL agents don't inherently recognize that in an idealized environment they can deal with waves coming from 0 and 180 degrees with almost zero rudder usage. On top of that, the first few developed agents had problems with excessive rudder usage, steady-state error, and overshooting behavior. All of these problems have been solved in the final iteration of the RL agent. Instead of just explaining the final agent, the thesis starts off with a weak RL agent and explains how it can be improved iteratively. This way the thesis explains how one might approach the problem of developing an RL-based controller. The first section focuses on giving a rough summary of RL and the problem case, explains the purpose of the thesis, then talks about previous work over marine movement control in literature. Some detailed information about the used tools and simulation environment is also given here. The second section introduces LQR controllers and designs an LQR controller for the heading keeping problem. The third section explains RL in-depth to lay the foundation for the upcoming sections. The fourth section starts with a naively designed simple RL agent and iteratively improves it. In each iteration of development, the agent is compared to the designed LQR controller, its weaknesses are analyzed, and the improvements for the next iteration are determined. The fifth section summarizes the previous sections, explains the contributions of the thesis, and discusses possible future work.