A generalized deep reinforcement learning based controller for heading keeping in waves

dc.contributor.advisor Kınacı, Ömer
dc.contributor.author Beyazit, Afşin Baran
dc.contributor.authorID 508191229
dc.contributor.department Offshore Engineering
dc.date.accessioned 2024-05-13T07:24:00Z
dc.date.available 2024-05-13T07:24:00Z
dc.date.issued 2022-06-21
dc.description Thesis (M.Sc.) -- İstanbul Technical University, Graduate School, 2022
dc.description.abstract Reinforcement Learning (RL) is a machine learning method where a learner (the agent) tries to maximize a reward by learning how to act under different environmental circumstances. The agent looks at the state of its environment (through the state vector), takes an action, and then gets a reward and the next state of its environment. The agent improves its action-taking strategy (policy) with every action it experiments with. RL methods have been used for many decision-making problems including control problems with promising results. Unlike many traditional control methods, a model-free RL doesn't need any environment dynamics to operate. This is especially beneficial for problems where the model dynamics are non-linear or not well-known. However, classical controllers are still the most used method of control for maritime applications. Heading-keeping is a maritime control problem where a controller's objective is to keep the heading (yaw) angle of a vehicle constant. Generally speaking, the industry standard is to use traditional feedback controllers such as PID for this problem. This study focuses on designing a generalized RL controller for the heading-keeping problem in waves. The study compares the designed RL controller to a traditional controller in terms of yaw error and rudder usage and observes that the designed RL-based controller performs better than the used traditional controller. The first iterations of the RL agent had many issues. Unlike traditional controllers, the RL agents don't inherently recognize that in an idealized environment they can deal with waves coming from 0 and 180 degrees with almost zero rudder usage. On top of that, the first few developed agents had problems with excessive rudder usage, steady-state error, and overshooting behavior. All of these problems have been solved in the final iteration of the RL agent. Instead of just explaining the final agent, the thesis starts off with a weak RL agent and explains how it can be improved iteratively. This way the thesis explains how one might approach the problem of developing an RL-based controller. The first section focuses on giving a rough summary of RL and the problem case, explains the purpose of the thesis, then talks about previous work over marine movement control in literature. Some detailed information about the used tools and simulation environment is also given here. The second section introduces LQR controllers and designs an LQR controller for the heading keeping problem. The third section explains RL in-depth to lay the foundation for the upcoming sections. The fourth section starts with a naively designed simple RL agent and iteratively improves it. In each iteration of development, the agent is compared to the designed LQR controller, its weaknesses are analyzed, and the improvements for the next iteration are determined. The fifth section summarizes the previous sections, explains the contributions of the thesis, and discusses possible future work.
dc.description.degree M.Sc.
dc.identifier.uri http://hdl.handle.net/11527/24844
dc.language.iso en_US
dc.publisher Graduate School
dc.sdg.type Goal 9: Industry, Innovation and Infrastructure
dc.subject maritime
dc.subject denizcilik
dc.subject control problems
dc.subject kontrol problemleri
dc.subject machine learning
dc.subject makine öğrenmesi
dc.subject reinforced learning
dc.subject takviyeli öğrenme
dc.title A generalized deep reinforcement learning based controller for heading keeping in waves
dc.title.alternative Dalgalı ortamda yön tutma problemi için geliştirilmiş derin takviyeli öğrenme tabanlı bir kontrolcü
dc.type Master Thesis
Dosyalar
Orijinal seri
Şimdi gösteriliyor 1 - 1 / 1
thumbnail.default.alt
Ad:
508191229.pdf
Boyut:
2.09 MB
Format:
Adobe Portable Document Format
Açıklama
Lisanslı seri
Şimdi gösteriliyor 1 - 1 / 1
thumbnail.default.placeholder
Ad:
license.txt
Boyut:
1.58 KB
Format:
Item-specific license agreed upon to submission
Açıklama