Directional regularization based variational models for image recovery

Abstract

This thesis explores how local directional cues can be utilized in image recovery. Our intent is to provide image regularization paradigms that encourage the underlying directionalities. To this end, in the first phase of the thesis work, we design direction-aware analysis-based regularization terms. We boost the structure tensor total variation (STV) functionals used in inverse imaging problems so that they encode directional priors. More specifically, we suggest redefining structure tensors to describe the distribution of the ``directional" first-order derivatives within a local neighborhood. With this decision, we bring direction-awareness to the STV penalty terms, which were originally imposing local structural regularity. We enrich the nonlocal counterpart of the STV in the same way, which were additionally imposing nonlocal image self-similarity beforehand. These two types of regularizers are used to model denoising and deblurring problems within a variational framework. Since they result in convex energy functionals, we also develop convex optimization algorithms by devising the proximal maps of our direction-aware penalty terms. With these contributions in place, the major barrier in making these regularizers applicable lies in the difficulty of estimating directional parameters (i.e., the directions/orientations, the dose of anisotropy). Although, it is possible to come across uni-directional images, the real-world images usually exhibit no directional dominance. It is easy to precisely estimate the underlying directions of uni-directional (or partially directional) images. However, arbitrary and unstable directions call for spatially varying directional parameters. In this regard, we propose two different parameter estimation procedures, each of which employs the eigendecompositions of the semi-local/nonlocal structure tensors. We also make use of total variation (TV) regularization in one of the proposed procedures and a filterbank of anisotropic Gaussian kernels (AGKs) in the other. As our image regularization frameworks require the guidance of the directional parameter maps, we use the term ``direction-guided" in naming our regularizers. Through the quantitative and the visual experiments, we demonstrate how beneficial the involvement of the directional information is by validating the superiority of our regularizers over the state-of-the-art analysis-based regularization schemes, including STV and nonlocal STV. In the second phase of the thesis, we shift our focus from model-driven to data-driven image restoration, more specifically we deal with transfer learning. As the target field, we choose fluorescence microscopy imaging, where noise is a very usual phenomenon but data-driven denoising is less applicable due to lack of the ground-truth images. In order to tackle this challenge, we suggest tailoring a dataset by handpicking images from unrelated source datasets. This selective procedure explores some low-level view-based features (i.e., color, isotropy/anisotropy, and directionality) of the candidate images, and their similarities to those of the fluorescence microscopy images. Based upon our experience on the model-driven restoration techniques, we speculate that these low-level characteristics (especially directions) play an important role on image restoration. In order to encourage a deep learning model to exploit these characteristics, one could embed them into the training data. In fact, we establish the possibility of offering a good balance between content-awareness and universality of the model by transferring only low-level knowledge and letting the unrelated images bring additional knowledge. In addition to training a feed-forward denoising convolutional neural network (DnCNN) on our tailored dataset, we also suggest integrating a small amount of fluorescence data through the use of fine-tuning for better-recovered micrographs. We conduct extensive experiments considering both Gaussian and mixed Poisson-Gaussian denoising problems. On the one hand, the experiments show that our approach is able to curate a dataset, which is significantly superior to the arbitrarily chosen unrelated source datasets, and competitive against the real fluorescence images. On the other hand, the involvement of fine-tuning further boosts the performance by stimulating the content-awareness, at the expense of a limited amount of target-specific data that we assume is available.