Estimation and restoration for heat haze effects in image and video processing

Abstract

There is various type of distortions that decrease visual quality. Heat haze effect is one of them. It makes the distant objects difficult to see clearly. It causes by such reasons that temperature difference, heterogeneous refractive index, or flow of hot air fluctuation. It is known as heat haze, heat scintillation, heat mirage or atmospheric turbulence. The existing of heat haze can be easily seen with human visual system (HVS) especially from the videos. There is also an estimation method by using digital image correlation (DIC) measurements which shows the displacement and strain errors with colored vector plots. In this method, it is obtained distortion information by subtracting the compared images from each other. A comparison was made for both heat hazy and unhazy situations. To obtain the distortion information, it is used hazy or unhazy set of frames alongside one unhazy reference frame for hazy and unhazy situations, respectively. The reference unhazy frame is obtained from our restoration algorithm which is located at second part of this thesis for not-simulated natural data, due to the nonexistence in reality. Thus, this estimation just to visualize the effect of heat haze. As a restoration of heat haze, there is a model-based enhancement method is used, which is wavelet based, instead of deep learning-based methods. Some steps were applied in enhancement process. RoI alignment, frame selection, image registration and post-processing steps are applied. In an image fusion step that is the most important one in this steps, dual tree complex wavelet transform (DT-CWT) fusion is used having its directional selectivity and shift-invariance properties which is an advantage in accordance with other wavelet transforms. The restoration algorithm needs a few input frames to obtain one restored output frame. Then as an addition to the algorithm and a contribution to the thesis, successive hazy input frames are shifted to create output frames to obtain a restored video. At the end of this thesis, there is a comparison section, with the usage of HVS, full reference (FR) and no-reference (NR) methods. In full reference comparison which includes GSSM, GMSD, single-scale SSIM and multi-scale SSIM methods, a reference and a compared raw video sequences are needed to obtain quality scores. As a reference, hazy video was used due to its existence in natural, and as a compared video, the restored video output is used. In this thesis, the NR method, which is a method that does not require a reference, is also applied. In the NR method, JPEG quality score is obtained by using one restored image frame. Quality methods were applied to the result of our restoration algorithm, which includes two different fusion methods (pixel-based and region-based), with the camera's ready algorithm result, to create a comparison between algorithms. Also, for at different distance image scenes, NR metric was applied. Looking at the overall summary of NR and FR metrics, our DT-CWT fusion-based restoration algorithm is better than the camera algorithm in terms of not losing details and removing heat waves. In the pixel-based or region-based comparison of DT-CWT, it is seen that the region-based fusion method gives better results in terms of obtaining unnoisy data and removing heat waves especially at high distances. As a future work, in order to get rid of the ghost effects, which is seen in moving objects, motion detection algorithm can be added. Another improvable work is to accelerate the implementation to implement the algorithm in real-time video.