Self-supervised pansharpening: Guided colorization of panchromatic images using generative adversarial networks

Abstract

Satellite images provide images with different properties. Multispectral images have low spatial resolution and high spectral resolution. Panchromatic images have high spatial resolution and low spectral resolution. The fusion process of these two images is called pansharpening. For decades, traditional image processing methods are designed for this process. After the inspirational success of Convolutional Neural Networks(CNN) in computer vision, CNN models are also designed for pansharpening. Convolutional Neural Networks (CNN)-based approaches have shown promising results in pansharpening of satellite images in recent years. However, they still exhibit limitations in producing high-quality pansharpening outputs. We identified a spatial detail disagreement problem between reduced resolution panchromatic images and original multispectral images, which are assumed to have the same resolution. This problem causes an insufficient training process in current CNN-based pansharpening models. We propose a new self-supervised learning framework, where we treat pansharpening as a colorization problem, which brings an entirely novel perspective and solution to the problem compared to existing methods that base their solution solely on producing a super-resolution version of the multispectral image. CNN-based methods provide a reduced resolution panchromatic image as input to their model along with reduced resolution multispectral images, hence learn to increase their resolution together. In the training phase of our model, reduced resolution panchromatic image is substituted with grayscale transformed multispectral image, thus our model learns colorization of the grayscale input. We further address the fixed downscale ratio assumption during training, which does not generalize well to the full-resolution scenario. We introduce a noise injection into the training by randomly varying the downsampling ratios. Those two critical changes, along with the addition of adversarial training in the proposed PanColorization Generative Adversarial Networks (PanColorGAN) framework, help overcome the spatial detail loss and blur problems that are observed in CNN-based pansharpening. The proposed approach outperforms the previous CNN-based and traditional methods as demonstrated in our experiments.