Connecting mathematical models for image processing and neural networks

This thesis deals with the connections between mathematical models for image processing and deep learning. While data-driven deep learning models such as neural networks are flexible and well performing, they are often used as a black box. This makes it hard to provide theoretical model guarantees and scientific insights. On the other hand, more traditional, model-driven approaches such as diffusion, wavelet shrinkage, and variational models offer a rich set of mathematical foundations. Our goal is to transfer these foundations to neural networks. To this end, we pursue three strategies. First, we design trainable variants of traditional models and reduce their parameter set after training to obtain transparent and adaptive models. Moreover, we investigate the architectural design of numerical solvers for partial differential equations and translate them into building blocks of popular neural network architectures. This yields criteria for stable networks and inspires novel design concepts. Lastly, we present novel hybrid models for inpainting that rely on our theoretical findings. These strategies provide three ways for combining the best of the two worlds of model- and data-driven approaches. Our work contributes to the overarching goal of closing the gap between these worlds that still exists in performance and understanding.Gegenstand dieser Arbeit sind die Zusammenhänge zwischen mathematischen Modellen zur Bildverarbeitung und Deep Learning. Während datengetriebene Modelle des Deep Learning wie z.B. neuronale Netze flexibel sind und gute Ergebnisse liefern, werden sie oft als Black Box eingesetzt. Das macht es schwierig, theoretische Modellgarantien zu liefern und wissenschaftliche Erkenntnisse zu gewinnen. Im Gegensatz dazu bieten traditionellere, modellgetriebene Ansätze wie Diffusion, Wavelet Shrinkage und Variationsansätze eine Fülle von mathematischen Grundlagen. Unser Ziel ist es, diese auf neuronale Netze zu übertragen. Zu diesem Zweck verfolgen wir drei Strategien. Zunächst entwerfen wir trainierbare Varianten von traditionellen Modellen und reduzieren ihren Parametersatz, um transparente und adaptive Modelle zu erhalten. Außerdem untersuchen wir die Architekturen von numerischen Lösern für partielle Differentialgleichungen und übersetzen sie in Bausteine von populären neuronalen Netzwerken. Daraus ergeben sich Kriterien für stabile Netzwerke und neue Designkonzepte. Schließlich präsentieren wir neuartige hybride Modelle für Inpainting, die auf unseren theoretischen Erkenntnissen beruhen. Diese Strategien bieten drei Möglichkeiten, das Beste aus den beiden Welten der modell- und datengetriebenen Ansätzen zu vereinen. Diese Arbeit liefert einen Beitrag zum übergeordneten Ziel, die Lücke zwischen den zwei Welten zu schließen, die noch in Bezug auf Leistung und Modellverständnis besteht.ERC Advanced Grant INCOVI

Denoising and enhancement of digital images : variational methods, integrodifferential equations, and wavelets

The topics of this thesis are methods for denoising, enhancement, and simplification of digital image data. Special emphasis lies on the relations and structural similarities between several classes of methods which are motivated from different contexts. In particular, one can distinguish the methods treated in this thesis in three classes: For variational approaches and partial differential equations, the notion of the derivative is the tool of choice to model regularity of the data and the desired result. A general framework for such approaches is proposed that involve all partial derivatives of a prescribed order and experimentally are capable of leading to piecewise polynomial approximations of the given data. The second class of methods uses wavelets to represent the data which makes it possible to understand the filtering as very simple pointwise application of a nonlinear function. To view these wavelets as derivatives of smoothing kernels is the basis for relating these methods to integrodifferential equations which are investigated here. In the third case, values of the image in a neighbourhood are averaged where the weights of this averaging can be adapted respecting different criteria. By refinement of the pixel grid and transfer to scaling limits, connections to partial differential equations become visible here, too. They are described in the framework explained before. Numerical aspects of the simplification of images are presented with respect to the NDS energy function, a unifying approach that allows to model many of the aforementioned methods. The behaviour of the filtering methods is documented with numerical examples.Gegenstand der vorliegenden Arbeit sind Verfahren zum Entrauschen, qualitativen Verbessern und Vereinfachen digitaler Bilddaten. Besonderes Augenmerk liegt dabei auf den Beziehungen und der strukturellen Ähnlichkeit zwischen unterschiedlich motivierten Verfahrensklassen. Insbesondere lassen sich die hier behandelten Methoden in drei Klassen einordnen: Bei den Variationsansätzen und partiellen Differentialgleichungen steht der Begriff der Ableitung im Mittelpunkt, um Regularität der Daten und des gewünschten Resultats zu modellieren. Hier wird ein einheitlicher Rahmen für solche Ansätze angegeben, die alle partiellen Ableitungen einer vorgegebenen Ordnung involvieren und experimentell auf stückweise polynomielle Approximationen der gegebenen Daten führen können. Die zweite Klasse von Methoden nutzt Wavelets zur Repräsentation von Daten, mit deren Hilfe sich Filterung als sehr einfache punktweise Anwendung einer nichtlinearen Funktion verstehen lässt. Diese Wavelets als Ableitungen von Glättungskernen aufzufassen bildet die Grundlage für die hier untersuchte Verbindung dieser Verfahren zu Integrodifferentialgleichungen. Im dritten Fall werden Werte des Bildes in einer Nachbarschaft gemittelt, wobei die Gewichtung bei dieser Mittelung adaptiv nach verschiedenen Kriterien angepasst werden kann. Durch Verfeinern des Pixelgitters und Übergang zu Skalierungslimites werden auch hier Verbindungen zu partiellen Differentialgleichungen sichtbar, die in den vorher dargestellten Rahmen eingeordnet werden. Numerische Aspekte beim Vereinfachen von Bildern werden anhand der NDS-Energiefunktion dargestellt, eines einheitlichen Ansatzes, mit dessen Hilfe sich viele der vorgenannten Methoden realisieren lassen. Das Verhalten der einzelnen Filtermethoden wird dabei jeweils durch numerische Beispiele dokumentiert

Machine Learning And Image Processing For Noise Removal And Robust Edge Detection In The Presence Of Mixed Noise

The central goal of this dissertation is to design and model a smoothing filter based on the random single and mixed noise distribution that would attenuate the effect of noise while preserving edge details. Only then could robust, integrated and resilient edge detection methods be deployed to overcome the ubiquitous presence of random noise in images. Random noise effects are modeled as those that could emanate from impulse noise, Gaussian noise and speckle noise. In the first step, evaluation of methods is performed based on an exhaustive review on the different types of denoising methods which focus on impulse noise, Gaussian noise and their related denoising filters. These include spatial filters (linear, non-linear and a combination of them), transform domain filters, neural network-based filters, numerical-based filters, fuzzy based filters, morphological filters, statistical filters, and supervised learning-based filters. In the second step, switching adaptive median and fixed weighted mean filter (SAMFWMF) which is a combination of linear and non-linear filters, is introduced in order to detect and remove impulse noise. Then, a robust edge detection method is applied which relies on an integrated process including non-maximum suppression, maximum sequence, thresholding and morphological operations. The results are obtained on MRI and natural images. In the third step, a combination of transform domain-based filter which is a combination of dual tree – complex wavelet transform (DT-CWT) and total variation, is introduced in order to detect and remove Gaussian noise as well as mixed Gaussian and Speckle noise. Then, a robust edge detection is applied in order to track the true edges. The results are obtained on medical ultrasound and natural images. In the fourth step, a smoothing filter, which is a feed-forward convolutional network (CNN) is introduced to assume a deep architecture, and supported through a specific learning algorithm, l2 loss function minimization, a regularization method, and batch normalization all integrated in order to detect and remove impulse noise as well as mixed impulse and Gaussian noise. Then, a robust edge detection is applied in order to track the true edges. The results are obtained on natural images for both specific and non-specific noise-level

Mathematical Approaches for Image Enhancement Problems

This thesis develops novel techniques that can solve some image enhancement problems using theoretically and technically proven and very useful mathematical tools to image processing such as wavelet transforms, partial differential equations, and variational models. Three subtopics are mainly covered. First, color image denoising framework is introduced to achieve high quality denoising results by considering correlations between color components while existing denoising approaches can be plugged in flexibly. Second, a new and efficient framework for image contrast and color enhancement in the compressed wavelet domain is proposed. The proposed approach is capable of enhancing both global and local contrast and brightness as well as preserving color consistency. The framework does not require inverse transform for image enhancement since linear scale factors are directly applied to both scaling and wavelet coefficients in the compressed domain, which results in high computational efficiency. Also contaminated noise in the image can be efficiently reduced by introducing wavelet shrinkage terms adaptively in different scales. The proposed method is able to enhance a wavelet-coded image computationally efficiently with high image quality and less noise or other artifact. The experimental results show that the proposed method produces encouraging results both visually and numerically compared to some existing approaches. Finally, image inpainting problem is discussed. Literature review, psychological analysis, and challenges on image inpainting problem and related topics are described. An inpainting algorithm using energy minimization and texture mapping is proposed. Mumford-Shah energy minimization model detects and preserves edges in the inpainting domain by detecting both the main structure and the detailed edges. This approach utilizes faster hierarchical level set method and guarantees convergence independent of initial conditions. The estimated segmentation results in the inpainting domain are stored in segmentation map, which is referred by a texture mapping algorithm for filling textured regions. We also propose an inpainting algorithm using wavelet transform that can expect better global structure estimation of the unknown region in addition to shape and texture properties since wavelet transforms have been used for various image analysis problems due to its nice multi-resolution properties and decoupling characteristics

Structure-aware image denoising, super-resolution, and enhancement methods

Denoising, super-resolution and structure enhancement are classical image processing applications. The motive behind their existence is to aid our visual analysis of raw digital images. Despite tremendous progress in these fields, certain difficult problems are still open to research. For example, denoising and super-resolution techniques which possess all the following properties, are very scarce: They must preserve critical structures like corners, should be robust to the type of noise distribution, avoid undesirable artefacts, and also be fast. The area of structure enhancement also has an unresolved issue: Very little efforts have been put into designing models that can tackle anisotropic deformations in the image acquisition process. In this thesis, we design novel methods in the form of partial differential equations, patch-based approaches and variational models to overcome the aforementioned obstacles. In most cases, our methods outperform the existing approaches in both quality and speed, despite being applicable to a broader range of practical situations.Entrauschen, Superresolution und Strukturverbesserung sind klassische Anwendungen der Bildverarbeitung. Ihre Existenz bedingt sich in dem Bestreben, die visuelle Begutachtung digitaler Bildrohdaten zu unterstützen. Trotz erheblicher Fortschritte in diesen Feldern bedürfen bestimmte schwierige Probleme noch weiterer Forschung. So sind beispielsweise Entrauschungsund Superresolutionsverfahren, welche alle der folgenden Eingenschaften besitzen, sehr selten: die Erhaltung wichtiger Strukturen wie Ecken, Robustheit bezüglich der Rauschverteilung, Vermeidung unerwünschter Artefakte und niedrige Laufzeit. Auch im Gebiet der Strukturverbesserung liegt ein ungelöstes Problem vor: Bisher wurde nur sehr wenig Forschungsaufwand in die Entwicklung von Modellen investieret, welche anisotrope Deformationen in bildgebenden Verfahren bewältigen können. In dieser Arbeit entwerfen wir neue Methoden in Form von partiellen Differentialgleichungen, patch-basierten Ansätzen und Variationsmodellen um die oben erwähnten Hindernisse zu überwinden. In den meisten Fällen übertreffen unsere Methoden nicht nur qualitativ die bisher verwendeten Ansätze, sondern lösen die gestellten Aufgaben auch schneller. Zudem decken wir mit unseren Modellen einen breiteren Bereich praktischer Fragestellungen ab

Q-switched fiber laser based on CdS quantum dots as a saturable absorber

In this work, a Q-switched fiber laser is demonstrated using quantum dots (QDs) cadmium sulfide (CdS) as a saturable absorber (SA) in an erbium-doped fiber laser (EDFL) system. The QD CdS is synthesized via the microwave hydrothermal assisted method and embedded into polyvinyl alcohol (PVA). The QD CdS/PVA matrix film is sandwiched in between two fiber ferrules by a fiber adapter. The generation of Q-switched fiber laser having a repetition rate, a pulse width, and a peak-topeak pulse duration of 75.19 kHz, 1.27 μs, and 13.32 μs, respectively. The maximum output power of 3.82 mW and maximum pulse energy of 50.8 nJ are obtained at the maximum pump power of 145.9 mW. The proposed design may add to the alternative material of Q-switched fiber laser generation, which gives a high stability output performance by using quantum dots material as a saturable absorber

Feature-preserving image restoration and its application in biological fluorescence microscopy

This thesis presents a new investigation of image restoration and its application to fluorescence cell microscopy. The first part of the work is to develop advanced image denoising algorithms to restore images from noisy observations by using a novel featurepreserving diffusion approach. I have applied these algorithms to different types of images, including biometric, biological and natural images, and demonstrated their superior performance for noise removal and feature preservation, compared to several state of the art methods. In the second part of my work, I explore a novel, simple and inexpensive super-resolution restoration method for quantitative microscopy in cell biology. In this method, a super-resolution image is restored, through an inverse process, by using multiple diffraction-limited (low) resolution observations, which are acquired from conventional microscopes whilst translating the sample parallel to the image plane, so referred to as translation microscopy (TRAM). A key to this new development is the integration of a robust feature detector, developed in the first part, to the inverse process to restore high resolution images well above the diffraction limit in the presence of strong noise. TRAM is a post-image acquisition computational method and can be implemented with any microscope. Experiments show a nearly 7-fold increase in lateral spatial resolution in noisy biological environments, delivering multi-colour image resolution of ~30 nm

Edge Preservation in Nonlinear Diffusion Filtering

Edge Preservation in Nonlinear Diffusion Filtering Mohammad Reza Hajiaboli, Ph.D. Concordia University, 2012 Image denoising techniques, in which the process of noise diffusion is modeled as a nonlinear partial differential equation, are well known for providing low-complexity solutions to the denoising problem with a minimal amount of image artifacts. In discrete settings of these nonlinear models, the objective of providing a good noise removal while preserving the image edges is heavily dependent on the kernels, diffusion functions and the associated contrast parameters employed by these nonlinear diffusion techniques. This thesis makes an in-depth study of the roles of the kernels and contrast parameters with a view to providing an effective solution to the problem of denoising of the images contaminated with stationary and signal-dependent noise. Within the above unified theme, this thesis has two major parts. In the first part of this study, the impact of anisotropic behavior of the Laplacian operator on the capabilities of nonlinear diffusion filters in preserving the image edges in different orientations is investigated. Based on this study, an analytical scheme is devised to obtain a spatially-varying kernel that adapt itself to the diffusivity function. The proposed edge-adaptive Laplacian kernel is then incorporated into various nonlinear diffusion filters for denoising of images contaminated by additive white Gaussian noise. The performance optimality of the the existing nonlinear diffusion techniques is generally based on the assumption that the noise and signal are uncorrelated. However, in many applications, such as in medical imaging systems and in remote sensing where the images are degraded by Poisson noise, this assumption is not valid. As such, in the second part of the thesis, a study is undertaken for denoising of images contaminated by Poisson noise within the framework of the Perona-Malik nonlinear diffusion filter. Specifically, starting from a Skellam distribution model of the gradient of the Poisson-noise corrupted images and following the diffusion mechanism of the nonlinear filter, a spatially and temporally varying contrast parameter is designed. It is shown that the nonlinear diffusion filters employing the new Laplacian kernel supports the extremum principle and that the proposed contrast parameter satisfies the sufficient conditions for observance of the scale-space properties. Extensive experiments are performed throughout the thesis to demonstrate the effectiveness and validity of the various schemes and techniques developed in this investigation. The simulation results of applying the new Laplacian kernel to a number of nonlinear diffusion filters show its distinctive advantages over the conventional Rosenfeld and Kak kernel, in terms of the filters' noise reduction and edge preservation capabilities for images corrupted by additive white Gaussian noise. The simulation results of incorporating the proposed spatially- and temporally-varying contrast parameter into the Perona-Malik nonlinear diffusion filter demonstrate a performance much superior to that provided by some of the other state-of-the-art techniques in denoising images corrupted by Poisson noise

Directional edge and texture representations for image processing

An efficient representation for natural images is of fundamental importance in image processing and analysis. The commonly used separable transforms such as wavelets axe not best suited for images due to their inability to exploit directional regularities such as edges and oriented textural patterns; while most of the recently proposed directional schemes cannot represent these two types of features in a unified transform. This thesis focuses on the development of directional representations for images which can capture both edges and textures in a multiresolution manner. The thesis first considers the problem of extracting linear features with the multiresolution Fourier transform (MFT). Based on a previous MFT-based linear feature model, the work extends the extraction method into the situation when the image is corrupted by noise. The problem is tackled by the combination of a "Signal+Noise" frequency model, a refinement stage and a robust classification scheme. As a result, the MFT is able to perform linear feature analysis on noisy images on which previous methods failed. A new set of transforms called the multiscale polar cosine transforms (MPCT) are also proposed in order to represent textures. The MPCT can be regarded as real-valued MFT with similar basis functions of oriented sinusoids. It is shown that the transform can represent textural patches more efficiently than the conventional Fourier basis. With a directional best cosine basis, the MPCT packet (MPCPT) is shown to be an efficient representation for edges and textures, despite its high computational burden. The problem of representing edges and textures in a fixed transform with less complexity is then considered. This is achieved by applying a Gaussian frequency filter, which matches the disperson of the magnitude spectrum, on the local MFT coefficients. This is particularly effective in denoising natural images, due to its ability to preserve both types of feature. Further improvements can be made by employing the information given by the linear feature extraction process in the filter's configuration. The denoising results compare favourably against other state-of-the-art directional representations

Splitting Methods in Image Processing

It is often necessary to restore digital images which are affected by noise (denoising), blur (deblurring), or missing data (inpainting). We focus here on variational methods, i.e., the restored image is the minimizer of an energy functional. The first part of this thesis deals with the algorithmic framework of how to compute such a minimizer. It turns out that operator splitting methods are very useful in image processing to derive fast algorithms. The idea is that, in general, the functional we want to minimize has an additive structure and we treat its summands separately in each iteration of the algorithm which yields subproblems that are easier to solve. In our applications, these are typically projections onto simple sets, fast shrinkage operations, and linear systems of equations with a nice structure. The two operator splitting methods we focus on here are the forward-backward splitting algorithm and the Douglas-Rachford splitting algorithm. We show based on older results that the recently proposed alternating split Bregman algorithm is equivalent to the Douglas-Rachford splitting method applied to the dual problem, or, equivalently, to the alternating direction method of multipliers. Moreover, it is illustrated how this algorithm allows us to decouple functionals which are sums of more than two terms. In the second part, we apply the above techniques to existing and new image restoration models. For the Rudin-Osher-Fatemi model, which is well suited to remove Gaussian noise, the following topics are considered: we avoid the staircasing effect by using an additional gradient fitting term or by combining first- and second-order derivatives via an infimal-convolution functional. For a special setting based on Parseval frames, a strong connection between the forward-backward splitting algorithm, the alternating split Bregman method and iterated frame shrinkage is shown. Furthermore, the good performance of the alternating split Bregman algorithm compared to the popular multistep methods is illustrated. A special emphasis lies here on the choice of the step-length parameter. Turning to a corresponding model for removing Poisson noise, we show the advantages of the alternating split Bregman algorithm in the decoupling of more complicated functionals. For the inpainting problem, we improve an existing wavelet-based method by incorporating anisotropic regularization techniques to better restore boundaries in an image. The resulting algorithm is characterized as a forward-backward splitting method. Finally, we consider the denoising of a more general form of images, namely, tensor-valued images where a matrix is assigned to each pixel. This type of data arises in many important applications such as diffusion-tensor MRI