Deep learning-based behavior analysis of seafarers

Abstract

Human error (HE) in maritime accidents is the focus of many researches. Researchers develop many approaches to mitigate it. Apart from the approaches introduced in the literature, a new approach is proposed in this thesis. The idea is that errors are hidden in human behaviours. If the behaviours causing marine accidents can be detected, relevant warnings and improvements can be arranged to eliminate those behaviours. In this context, this thesis aims to develop a deep learning-based algorithm to analyze the behaviour of seafarers. While the frequency of maritime accidents has been in decline thanks to the studies, one single incident can have catastrophic and long-term consequences for the marine environment. Especially collisions and groundings have the potential to cause those catastrophic results for the marine environment, so the thesis is delimited to eliminate the cause of collisions and groundings. According to the literature, errors causing collisions and groundings have occurred on the bridge where the main actor involved has been the watchkeeping personnel (WP). To validate the literature and find the main reasons causing collisions and groundings, a totally of 94 maritime incident reports on collisions and groundings are obtained from the UK's Marine Accident Investigation Branch, the Transportation Safety Board of Canada and the National Transportation Safety Board of the United States of America. TRACEr-MAR taxonomy is utilized on those incident reports to find root causes of the human errors causing collisions and groundings. Results show that 74 % of the errors are directly related to the watchkeeping behaviour of the WPs. Monitoring and assessing the behaviours of WPs all the time during navigation watch has the potential to mitigate those errors. An alerting algorithm can be adjusted to warn the master or assigned officer based on behaviours causing errors gathered from monitoring results. Besides, the assessment system encourages the WPs to keep standard watch because of knowing that they are continuously being monitored and evaluated. In this thesis, a Bridge Navigation Watch Monitoring System (BNWMS) is suggested to achieve those monitoring and assessment tasks. The proposed architecture for BNWMS enables to train a model that continuously analyses the behaviour of WPs. Motion heatmap of 3D body poses over a specific time interval is suggested as an input. 2D body poses belonging to the same person are estimated from multiple camera views by using a deep learning-based pose estimation algorithm. Those estimated 2D poses are projected into 3D space by utilizing multiple-view computer vision techniques. Finally, the obtained 3D poses are plotted on the bird's-eye view bridge plan to calculate a heatmap of body motions capturing temporal as well as spatial information. After validating the proposed vision-based approach in the pilot study, the multi-view video camera system is established on an actual bridge of a commercial bulk carrier by Veysel GOKCEK to collect relevant data. 14400 motion heatmaps, each of them presenting unique 12 minutes during navigation watch, are generated from collected data. Watchkeeping behaviours of the WPs based on generated heatmaps are classified as "Not Acceptable", "Below Standard", and "Standard". Training of models is conducted by using labelled 14400 motion heatmaps. Design of 6 custom CNNs and fine-tuning of 4 pre-trained CNNs are carried out to compare different CNN architectures. Pre-trained models show a higher value than custom CNNs, owing to their pre-trained initial layers which boost feature extraction. Pre-trained VGG16 model which has the highest accuracy of 0.96 among all models is utilized to predict instant monitoring and cumulative assessments of three navigation watch based on defined classes. Numerical scores are assigned to the classes, 0 points for "Not Acceptable", 50 points for "Below Standard", and 100 points for "Standard". Both instant monitoring and cumulative assessment using numerical scores are plotted on the graph to display the performance of the watches. While instant monitoring succeeds to show the momentary condition of the navigation watch, cumulative assessment achieves to separate watches based on their performance values. The BNWMS which is consist of both instant monitoring and cumulative assessment can also produce the numerical performance of navigation on a daily, weekly, monthly or a defined period basis. An alerting algorithm can be adjusted to warn the master or assigned officer when the instant monitoring or assessment value is under the threshold. Defining the relevant threshold value based on the condition of the voyage is the feature work including revision of maritime regulations, risk assessments and company procedures. This is the first research of deep learning-based behaviour analysis on WPs keeping watch on the ship's bridge. The developed BNWMS in the thesis has introduced two new approaches to the literature. One of them is explaining the behaviour of workers by a generation of their motion heatmaps on the 2D plan of the working area within a defined period. The second one is the instant and cumulative assessment of those heatmaps by deep learning-based artificial intelligence all the time. This research will be the basis for a series of other studies. Developed novel approaches will pave the way for behaviour analysis in environments other than ships such as factories that require working in a large area.