Satellite images super resolution using generative adversarial networks

Abstract

The general broad definition of remote sensing is to observe an object and collect data regarding this object without actual contact. From a narrower perspective, it is the science that studies the earth and its atmosphere by gathering data from above the earth. Nowadays earth observation systems with their various sensors in multiple bands produce a huge amount of data that need to be processed and analyzed to get a final product in a certain discipline. Applications like monitoring the water resources, forest fire monitoring, soil type classifications are examples of remote sensing use in different fields of our modern life. Satellite imagery plays a pivotal role in remote sensing .they can be acquired by various types of sensors some of which are passive like optical sensors and some whıch are active like LIDAR and SAR. This study focuses on the satellite images in the visible portion of the spectrum. This type of satellite imagery can vary in resolution whether this resolution is spatial, spectral, temporal, or radiometric. The satellite imagery also can be categorized according to its spatial resolution into low, medium, and high-resolution images and each of them can be deployed in certain applications. Preprocessing these images is a critical stage that would affect the final product or the application that uses these images. High resolution is a desirable characteristic, yet it can be difficult to achieve financially and technically. However, image processing can offer a convenient software solution to this problem by super-resolution techniques. Hence, the importance of superresolution which is one of the preprocessing tasks that obtains high-resolution images is considered fundamental in lots of remote sensing applications. Super-resolution aims to obtain high-resolution images using low-resolution observation. Super-resolution is considered a classical image processing problem that is ill-posed due to the lack of a single unique solution. Thus, lots of algorithms and approaches were proposed over the years. This study gives a general review of the main significant types of super-resolution algorithms which can be divided into interpolation-based, reconstruction-based, and learning-based algorithms. The simplest methods are interpolation-based ones, nevertheless, the results lack high-frequency details. The second type is reconstruction-based methods which require a good prior choice to get better results. designing a good prior can be complex These methods can be complicated. The third category is example-based or learn-based methods which include learning the relationship between the low resolution and high-resolution images by exploiting datasets to learn from. Algorithms like sparse coding super-resolution and deep learning methods are learning-based methods. Super-resolution methods performance is usually evaluated by many metrics such as, peak signal to noise ratio PSNR, which is based on mean squared error, a pixel-wise metric thus, can be misleading, structural similarity index SSIM which is considered more accurate as it considers the structure of the image instead of the individual pixel value. Deep learning, which deploys deep neural networks in its algorithms, is a branch of machine learning which is, in turn, a subfield of artificial intelligence. It is widely used in image processing and computer vision problems, especially after the emergence of convolutional neural networks CNNs. Deep learning models structures in image processing problems usually share common building blocks like CNNs. The default CNN consists of a convolutional layer followed by an activation layer to ensure nonlinearity, hence learning, which is followed by a pooling layer. The backpropagation is used to adjust weights at the end of every epoch of training. The fourth chapter of this thesis elaborates the super-resolution algorithms which were proposed to deal with super-resolution problems that present the state-of-the-art performance compared to the other methods. SRCNN was the first suggested model to deal with super-resolution. It is considered as the benchmark of super-resolution using deep learning. This model was followed by the FSRCNN which tried to overcome the backward of the previous model by using the low-resolution image as an input without upscaling and performed the upscaling later by using deconvolution layer. Very Deep Super Resolution model which mainly consists of deep VGG layers to get better results. Then there was the enhanced deep super-resolution model EDSR that exploited the concept of the residual blocks to be able to increase the depth of the network without getting slower training. SRResNet and SRGAN were proposed in the same paper to give a better performance in image super-resolution. SRResNet deployed the residual blocks in its structure in addition to conv layers and uses the mean squared error dased loss or VGG content loss to optimize. The generative model of generative adversarial neural networks consists of two network models that learn together, the generator aims to learn to generate the required data with the help of a discriminator that tries to differentiate between fake data generated by the generator and ground truth. This approach of training in an adversarial manner presents a state of the art performance in several tasks, It was also used in the super-resolution task by what is called as SRGANs super-resolution networks. In addition to the adversarial structure of this model, another factor that improved its performance is the perceptual loss that was used in optimizing the model. Mentioning all of these deep learning super-resolution algorithms, the next chapter gives a general overview of the use of deep learning in remote sensing. This use is expanding with the increased amount of remote sensing data and its quality and with the development of deep learning algorithms and computational abilitıes. From the preprocessing of the remote sensing data, like image fusion, segmentation, and denoising, to other many applications such as anomaly detection, land use classification, and other classification tasks, deep learning is being deployed in remote sensing. The experiment that is done in this thesis is to examine the performance of super-resolution generative adversarial neural networks on the satellite images and ıts abıltıy of generalization when it is trained with the irrelevant dataset. By training an SRGAN model using the UC-MERCED Land Use dataset which consists of 21 classes each class contains 100 images of size 256x256 these images are used as high-resolution images and downsized versions of them with factor x4 are used as low-resolution images. After training, the model was tested with random images from the NWPU-RESISC45 dataset. In order to examine the ability of generalization of the model, the same architecture was trained using a natural images dataset which is Linnaeus 5 256X256 which consists of 5 classes of 256x256 sized images in the same way as the previous training. testing was done with random images from the NWPU-RESISC45 dataset. In addition, the SRResNet model that uses the mean square error-based optimization was trained to compare it with the performance of the previous generative SRGAN models. Peak signal to noise ratio and structural similarity index was used to evaluate the performance and make a comparison between the previously mentioned methods. The experiment was done using Google Colab Pro environment utilizing its provided GPU.