Joint calibration and reconstruction for focal plane array imaging

Abstract

Single Pixel Imaging (SPI) is a technique in which a single photo-detector captures a scene. Modern camera systems usually utilize millions of silicon-based detectors to capture a scene. These silicon-based sensors have high sensitivity only in the visible band. Outside the visible band, sensors must be produced with more exotic materials. Specifically, for larger wavelengths, germanium (Ge) and indium gallium arsenide (InGaAs) can be shown as examples of these rare materials. These materials are highly expensive and hard to produce as a focal plane array (FPA), which is a 2-dimensional sensor array and is similar to modern camera system's sensor array. Therefore, decreasing the number of sensors is important for non-visible band systems, and the SPI technique provides a system that only utilizes a single detector. Moreover, this system is adopted by many research fields such as terahertz imaging, remote sensing, and medical imaging. The reconstruction process of the SPI system is an inverse problem and requires the solution of a highly ill-posed inverse problem. To get a unique solution for such an underdetermined system, a regularization term is needed. Compressive sensing (CS) is one of the techniques that use regularization as a prior information function. Moreover, for SPI systems, the CS technique is highly adopted since it can find a solution for a system that has less number of measurements than the number of unknown pixels. The reason is that the CS promotes sparsity in a domain that is incoherent with the measurement domain. Therefore, it provides a nonadaptive sampling procedure under some conditions. These conditions are related to the design process of measurement matrix design and the linear measurement process. If the restricted isometry property (RIP) is met, the entire system has an exact recovery guarantee. In general, it is proven that random measurement matrices are highly probable to satisfy the RIP. For an SPI system, a spatial light modulator (SLM) encodes a scene with different patterns. As a result, SLM filters can provide random sampling for the SPI system. The spatial resolution of an SPI system is high whereas the temporal resolution is low. This is because a single detector must obtain samples for each pixel, but control units of SLM limit the performance of the system. To get faster measurement rates, an FPA system can be used. In contrast to modern camera sensor arrays, the number of sensors in FPA for the SPI field is very few, but these sensors can provide parallel measurements. As a result, the temporal resolution of these FPA systems is higher than the temporal resolution of an SPI system including only a single photo-detector, and this creates a trade-off between measurement speed and the cost of the system. However, improvements in temporal resolution tolerate the cost disadvantage, and the FPA system is used by various applications. For an FPA system, a lens placed between the SLM encoder and sensor array becomes more effective in low-resolution FPA measurements. These effects are called airy disc effects and blurry FPA measurements are obtained. The airy disc effects happen because the airy disc does not fit into a single detector pixel, and other detectors around the detector make undesirable measurements and the FPA measurement becomes blurred. This effect arises due to the optical structure of the lens and requires calibration to reduce its effects. In this thesis, an online calibration technique is proposed to reduce the blur in FPA measurements. In the literature, there are proposed offline calibration techniques. One of them relies on getting a measurement for each pixel of a scene. This makes it a long and difficult process to form a calibration matrix. The reason is that even small temperature changes affect noise in detectors, and some pixels may be needed to be repeated to get healthy measurements. The other technique relies on CS, but it also needs to form a large calibration matrix after some repetitive measurements. There is also a deconvolution technique with an exact point spread function (PSF) information used to deblur FPA images. The PSF information cannot be easily obtained and the technique requires iterative calculations to find calibrated results. To overcome these limitations, an online deep learning-based (DL) design is proposed and it does not require long operations to define a calibration system and to implement it in an FPA system. The recovery process usually involves the solution of an inverse problem through optimization-based approaches. These approaches improve image quality by iteratively utilizing prior information on the image while enforcing data consistency. While various prior information terms such as sparsity in some transformation domains can be used for improved image quality, recent approaches adopt learned prior information terms by changing the related step in the optimization approach with a pre-trained DL-based denoiser which is called the plug-and-play (PnP)-based approach. Within the context of this thesis, we opted to use an alternating direction method of multipliers (ADMM)-based approach since it is a popular choice for CS reconstruction. In contrast to previous techniques that decouple the prior information from the problem model, this thesis involves the integration of the conventional iterative ADMM-based method into a DL framework, which is known as algorithm unrolling. Consequently, a convolutional neural network (CNN) is trained to acquire prior information. The utilization of the unrolling technique enables the acquisition of images in fewer amount of iterations when compared with the conventional iterative approaches. The thesis entails the development of models for varying numbers of measurement scenarios, followed by a comparative analysis of image quality metrics and visuals for both PnP ADMM and unrolling-based techniques. Finally, the thesis presents a joint design where the online calibration model and the unrolling ADMM module are combined and used together. In this joint model, the parameters of the prior information and calibration model are learned using a unified error function obtained by designing a joint objective function between both corrected FPA measurements and high-resolution model outputs. Different numbers of measurement scenarios are examined for each defined airy disc range. As a result, it is shown that the joint design both reduces the airy disc effects and obtains well-reconstructed images. Three distinct methodologies have been employed to conduct numerical investigations. The first part solely consisted of comparisons with alternative calibration techniques for the calibration model. To do this, the average peak signal-to-noise ratio (pSNR) is compared for each defined airy disc radius range and the performance improvement brought by the model taking the SLM filter information as input is shown. In the second part, airy disc effects are disregarded and only detector noise is taken into account. The study demonstrates that the unrolling-based algorithm obtains comparable performance to conventional PnP ADMM methods across various image quality metrics, including pSNR, structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). In addition, execution time analyses showed that the unrolling-based ADMM method achieved the same quantitative performance in much smaller time frames. In the final section, comparisons are made for the joint model, a PnP ADMM technique, and an unrolling-based reconstruction algorithm. Both the PnP ADMM and the unrolling-based reconstruction algorithm take calibrated FPA measurements from the calibration model. For each airy disc range, the joint model achieves the best metric results. Moreover, the differences between the joint model and others are increased as the effects of the airy disc are increased.