Connecting mathematical models for image processing and neural networks

This thesis deals with the connections between mathematical models for image processing and deep learning. While data-driven deep learning models such as neural networks are flexible and well performing, they are often used as a black box. This makes it hard to provide theoretical model guarantees and scientific insights. On the other hand, more traditional, model-driven approaches such as diffusion, wavelet shrinkage, and variational models offer a rich set of mathematical foundations. Our goal is to transfer these foundations to neural networks. To this end, we pursue three strategies. First, we design trainable variants of traditional models and reduce their parameter set after training to obtain transparent and adaptive models. Moreover, we investigate the architectural design of numerical solvers for partial differential equations and translate them into building blocks of popular neural network architectures. This yields criteria for stable networks and inspires novel design concepts. Lastly, we present novel hybrid models for inpainting that rely on our theoretical findings. These strategies provide three ways for combining the best of the two worlds of model- and data-driven approaches. Our work contributes to the overarching goal of closing the gap between these worlds that still exists in performance and understanding.Gegenstand dieser Arbeit sind die Zusammenhänge zwischen mathematischen Modellen zur Bildverarbeitung und Deep Learning. Während datengetriebene Modelle des Deep Learning wie z.B. neuronale Netze flexibel sind und gute Ergebnisse liefern, werden sie oft als Black Box eingesetzt. Das macht es schwierig, theoretische Modellgarantien zu liefern und wissenschaftliche Erkenntnisse zu gewinnen. Im Gegensatz dazu bieten traditionellere, modellgetriebene Ansätze wie Diffusion, Wavelet Shrinkage und Variationsansätze eine Fülle von mathematischen Grundlagen. Unser Ziel ist es, diese auf neuronale Netze zu übertragen. Zu diesem Zweck verfolgen wir drei Strategien. Zunächst entwerfen wir trainierbare Varianten von traditionellen Modellen und reduzieren ihren Parametersatz, um transparente und adaptive Modelle zu erhalten. Außerdem untersuchen wir die Architekturen von numerischen Lösern für partielle Differentialgleichungen und übersetzen sie in Bausteine von populären neuronalen Netzwerken. Daraus ergeben sich Kriterien für stabile Netzwerke und neue Designkonzepte. Schließlich präsentieren wir neuartige hybride Modelle für Inpainting, die auf unseren theoretischen Erkenntnissen beruhen. Diese Strategien bieten drei Möglichkeiten, das Beste aus den beiden Welten der modell- und datengetriebenen Ansätzen zu vereinen. Diese Arbeit liefert einen Beitrag zum übergeordneten Ziel, die Lücke zwischen den zwei Welten zu schließen, die noch in Bezug auf Leistung und Modellverständnis besteht.ERC Advanced Grant INCOVI

Image Inpainting Methods

Tato práce se zabývá zpracováním přehledu moderních metod pro automatické doplnění chybějících částí obrazu. V teoretické části této práce je vybráno a popsáno několik nejznámějších metod. Každá z vybraných metod je nejdříve uvedena, poté je popsán její algoritmus a nakonec je zhodnocena za pomoci informací, nabytých z dostupné literatury. Mezi metody, které byly vybrány a následně popsány v této práci patří Image Inpainting, Fragment-Based Image Completion, Exemplar-Based Image Inpainting, Gradient-Based Image Completion by Solving Poisson Equation a nakonec Inpainting by Flexible Haar- Wavelet Shrinkage. V praktické části bakalářské práce byl vybrán algoritmus A Framelet-Based Image Inpa- inting, který byl naprogramován a implementován v programovém prostředí MATLAB. Pro tento algoritmus bylo také naprogramováno vlastní funkční řešení Framelet trans- formace. Dále bylo vytvořeno GUI, které poskytuje možnost uživatelské interakce. Toto v prostředí MATLAB realizované GUI umožňuje jednoduše spravovat vstupy a parame- try algoritmu a pracovat s jeho výstupy. Uživatel je vždy informován o aktuálním stavu výpočtu a je mu zobrazen aktuální výsledek doplnění obrazu. Navíc byl pro GUI vytvo- řen nástroj, který poskytuje uživateli možnost definovat pomocí myši oblasti, jež mají být doplněny. Nakonec byly zhodnoceny výsledky implementovaného algoritmu jak při použití Framelet transformace, tak při použití Contoulet transformace.This thesis deals with an overview of modern Image Inpainting Methods. There are several best-known methods selected and described in the theoretical part of this work. Each of the selected methods is described and evaluated according to the informations available in literature. Among the methods that were selected and subsequently described in this work are Image Inpainting, Fragment-Based Image Completion, Exemplar-Based Image Inpainting, Gradient-Based Image Completion by Solving Poisson Equation and Inpainting by Flexible Haar-Wavelet Shrinkage. The MATLAB implementation of the Framelet-Based Image Inpainting algorithm forms practical part of the thesis. The Framelet transform was created for the purposes of the algorithm. The user interaction provides GUI, which was also implemented in MATLAB. The GUI allows setting input images, algorithm parameters and interaction with the output. The user is always informed about the current state of the computation, and the current result of image completion is shown to him. Moreover, it was created a tool that allows the user to define the areas to be supplemented, using the mouse. Finally, the algorithm performance is evaluated and compared using both Framelet and Contourlet transform.

Splitting Methods in Image Processing

It is often necessary to restore digital images which are affected by noise (denoising), blur (deblurring), or missing data (inpainting). We focus here on variational methods, i.e., the restored image is the minimizer of an energy functional. The first part of this thesis deals with the algorithmic framework of how to compute such a minimizer. It turns out that operator splitting methods are very useful in image processing to derive fast algorithms. The idea is that, in general, the functional we want to minimize has an additive structure and we treat its summands separately in each iteration of the algorithm which yields subproblems that are easier to solve. In our applications, these are typically projections onto simple sets, fast shrinkage operations, and linear systems of equations with a nice structure. The two operator splitting methods we focus on here are the forward-backward splitting algorithm and the Douglas-Rachford splitting algorithm. We show based on older results that the recently proposed alternating split Bregman algorithm is equivalent to the Douglas-Rachford splitting method applied to the dual problem, or, equivalently, to the alternating direction method of multipliers. Moreover, it is illustrated how this algorithm allows us to decouple functionals which are sums of more than two terms. In the second part, we apply the above techniques to existing and new image restoration models. For the Rudin-Osher-Fatemi model, which is well suited to remove Gaussian noise, the following topics are considered: we avoid the staircasing effect by using an additional gradient fitting term or by combining first- and second-order derivatives via an infimal-convolution functional. For a special setting based on Parseval frames, a strong connection between the forward-backward splitting algorithm, the alternating split Bregman method and iterated frame shrinkage is shown. Furthermore, the good performance of the alternating split Bregman algorithm compared to the popular multistep methods is illustrated. A special emphasis lies here on the choice of the step-length parameter. Turning to a corresponding model for removing Poisson noise, we show the advantages of the alternating split Bregman algorithm in the decoupling of more complicated functionals. For the inpainting problem, we improve an existing wavelet-based method by incorporating anisotropic regularization techniques to better restore boundaries in an image. The resulting algorithm is characterized as a forward-backward splitting method. Finally, we consider the denoising of a more general form of images, namely, tensor-valued images where a matrix is assigned to each pixel. This type of data arises in many important applications such as diffusion-tensor MRI

MCALab: Reproducible Research in Signal and Image Decomposition and Inpainting

International audienceMorphological Component Analysis (MCA) of signals and images is an ambitious and important goal in signal processing; successful methods for MCA have many far-reaching applications in science and technology. Because MCA is related to solving underdetermined systems of linear equations it might also be considered, by some, to be problematic or even intractable. Reproducible research is essential to to give such a concept a firm scientific foundation and broad base of trusted results. MCALab has been developed to demonstrate key concepts of MCA and make them available to interested researchers and technologists. MCALab is a library of Matlab routines that implement the decomposition and inpainting algorithms that we previously proposed in [1], [2], [3], [4]. The MCALab package provides the research community with open source tools for sparse decomposition and inpainting and is made available for anonymous download over the Internet. It contains a variety of scripts to reproduce the figures in our own articles, as well as other exploratory examples not included in the papers. One can also run the same experiments on one's own data or tune the parameters by simply modifying the scripts. The MCALab is under continuing development by the authors; who welcome feedback and suggestions for further enhancements, and any contributions by interested researchers