A deep learning based framework for identification of ship types using optical satellite images

Abstract

Today, monitoring of ship traffic in open and coastal seas is one of the primary and even indispensable activities of coastal countries for many reasons. Preventing activities such as illegal fishing, migration, smuggling, pollution of the seas, protection of underground resources and sea creatures, defensive reconnaissance, and surveillance activities can be listed as the main reasons for monitoring ship activity in the seas. Periodic monitoring of open and coastal seas using Earth observation satellites is a significant approach, providing fast and cost-effective results as well as large coverage extents. With this perspective, the thesis presents research about an effective and end to end ship monitoring approach via optical satellites. The use of deep learning (DL) techniques for the detection and classification of ships using optical satellite images, the creation of a ship database consisting of optical satellite images for the verification of the subject, and optical constellation modeling are the topics of this research. In the thesis, a comprehensive literature review is presented with the scope of gabs in the ship monitoring with using optical satellites. To mimic the real satellite image input, an optical satellite based image dataset – VHRShips - was formed. VHRShips consists of a total of 6312 images, 1000 without ships and 5312 with ships. There are a total of 11337 ships in the images. The database created includes 24 different types, and navy ships, one of these types, contain 11 different types. This dataset stands out with its rich ship database content and large sample size. Thesis reinterprets DL-based target detection and classification algorithms and proposes a flexible target detection and classification approach in a hierarchical design (HieD) that allows easy addition and removal of different algorithms. In addition, the detection and localization, recognition, and identification (DLRI) steps are staged and the outputs of the algorithm are detailed. In the phase of detecting, the presence of ships is verified in the images provided by the satellites. The determination of the positions of the ships in the images is carried out during the localization stage. The classification of the ships among the determined basic ship types is defined in the recognition stage. Lastly, the classification of navy ships, which is one of the main ship types, is handled during the identification stage. The feasibility of the developed approach has been demonstrated by the preliminary feasibility analysis for the covering of the Turkish surrounding seas with optical constellations. In this analysis, an optimization was carried out by using a software on how the satellite images required for the proposed method can be realized with a set of satellite design. It has been determined that 40 optical satellites are required to fully cover the selected sea area in 24 hours, and 100 optical satellites with the specified characteristics to be covered twice. The results of proposed method, HieD are presented in three formats which are; the individual stage performances, a comprehensive end-to-end evaluation and lastly a comparison with a well-known method, YOLOV4. The results of the thesis are very promising. F1 scores for detection, recognition, localization, and identification were respectively 99.17%, 94.20%, 84.08%, and 82.13% as a consequence of stage-by-stage optimization. The F1 scores for the same order after complete implementation of our suggested method were 99.17%, 93.43%, 74.00%, and 57.05%. End-to-end YOLOv4 produced F1-scores for DLRI of 99.17%, 86.59%, 68.87%, and 56.28%, in opposition. For the steps of localization, recognition, and identification, we outperformed YOLOv4 using HieD. In the thesis; it has been shown that the ship detection, localization, recognition and identification method developed in a hierarchical structure using a challenging data set containing images with different backgrounds in open and coastal seas in many different geographies, works successfully. The method is very open to development with the antecedent and intermediate methods to be added. The data set created in the thesis has been used with various detection and classification techniques, proving that there is no dataset that provides a limited opportunity for the study. VHRShips can serve as a standard for the developed approach as well as for further research into the use of deep learning and advancing geospatial AI applications in the maritime environment. Finally, the thesis presents a vision of the future along with a fair self-criticism.