465 research outputs found
Connecting mathematical models for image processing and neural networks
This thesis deals with the connections between mathematical models for image processing and deep learning. While data-driven deep learning models such as neural networks are flexible and well performing, they are often used as a black box. This makes it hard to provide theoretical model guarantees and scientific insights. On the other hand, more traditional, model-driven approaches such as diffusion, wavelet shrinkage, and variational models offer a rich set of mathematical foundations. Our goal is to transfer these foundations to neural networks. To this end, we pursue three strategies. First, we design trainable variants of traditional models and reduce their parameter set after training to obtain transparent and adaptive models. Moreover, we investigate the architectural design of numerical solvers for partial differential equations and translate them into building blocks of popular neural network architectures. This yields criteria for stable networks and inspires novel design concepts. Lastly, we present novel hybrid models for inpainting that rely on our theoretical findings. These strategies provide three ways for combining the best of the two worlds of model- and data-driven approaches. Our work contributes to the overarching goal of closing the gap between these worlds that still exists in performance and understanding.Gegenstand dieser Arbeit sind die Zusammenhänge zwischen mathematischen Modellen zur Bildverarbeitung und Deep Learning. Während datengetriebene Modelle des Deep Learning wie z.B. neuronale Netze flexibel sind und gute Ergebnisse liefern, werden sie oft als Black Box eingesetzt. Das macht es schwierig, theoretische Modellgarantien zu liefern und wissenschaftliche Erkenntnisse zu gewinnen. Im Gegensatz dazu bieten traditionellere, modellgetriebene Ansätze wie Diffusion, Wavelet Shrinkage und Variationsansätze eine Fülle von mathematischen Grundlagen. Unser Ziel ist es, diese auf neuronale Netze zu übertragen. Zu diesem Zweck verfolgen wir drei Strategien. Zunächst entwerfen wir trainierbare Varianten von traditionellen Modellen und reduzieren ihren Parametersatz, um transparente und adaptive Modelle zu erhalten. Außerdem untersuchen wir die Architekturen von numerischen Lösern für partielle Differentialgleichungen und übersetzen sie in Bausteine von populären neuronalen Netzwerken. Daraus ergeben sich Kriterien für stabile Netzwerke und neue Designkonzepte. Schließlich präsentieren wir neuartige hybride Modelle für Inpainting, die auf unseren theoretischen Erkenntnissen beruhen. Diese Strategien bieten drei Möglichkeiten, das Beste aus den beiden Welten der modell- und datengetriebenen Ansätzen zu vereinen. Diese Arbeit liefert einen Beitrag zum übergeordneten Ziel, die Lücke zwischen den zwei Welten zu schließen, die noch in Bezug auf Leistung und Modellverständnis besteht.ERC Advanced Grant INCOVI
Image restoration: Wavelet frame shrinkage, nonlinear evolution PDEs, and beyond
In the past few decades, mathematics based approaches have been widely adopted in various image restoration problems; the partial differential equation (PDE) based approach (e.g., the total variation model [L. Rudin, S. Osher, and E. Fatemi, Phys. D, 60 (1992), pp. 259-268] and its generalizations, nonlinear diffusions [P. Perona and J. Malik, IEEE Trans. Pattern Anal. Mach. Intel., 12 (1990), pp. 629-639; F. Catte et al., SIAM J. Numer. Anal., 29 (1992), pp. 182-193], etc.) and wavelet frame based approach are some successful examples. These approaches were developed through different paths and generally provided understanding from different angles of the same problem. As shown in numerical simulations, implementations of the wavelet frame based approach and the PDE based approach quite often end up solving a similar numerical problem with similar numerical behaviors, even though different approaches have advantages in different applications. Since wavelet frame based and PDE based approaches have all been modeling the same types of problems with success, it is natural to ask whether the wavelet frame based approach is fundamentally connected with the PDE based approach when we trace them all the way back to their roots. A fundamental connection of a wavelet frame based approach with a total variation model and its generalizations was established in [J. Cai, B. Dong, S. Osher, and Z. Shen, J. Amer. Math. Soc., 25 (2012), pp. 1033-1089]. This connection gives the wavelet frame based approach a geometric explanation and, at the same time, it equips a PDE based approach with a time frequency analysis. Cai et al. showed that a special type of wavelet frame model using generic wavelet frame systems can be regarded as an approximation of a generic variational model (with the total variation model as a special case) in the discrete setting. A systematic convergence analysis, as the resolution of the image goes to infinity, which is the key step in linking the two approaches, is also given in Cai et al. Motivated by Cai et al. and [Q. Jiang, Appl. Numer. Math., 62 (2012), pp. 51-66], this paper establishes a fundamental connection between the wavelet frame based approach and nonlinear evolution PDEs, provides interpretations and analytical studies of such connections, and proposes new algorithms for image restoration based on the new understandings. Together with the results in [J. Cai et al., J. Amer. Math. Soc., 25 (2012), pp. 1033-1089], we now have a better picture of how the wavelet frame based approach can be used to interpret the general PDE based approach (e.g., the variational models or nonlinear evolution PDEs) and can be used as a new and useful tool in numerical analysis to discretize and solve various variational and PDE models. To be more precise, we shall establish the following: (1) The connections between wavelet frame shrinkage and nonlinear evolution PDEs provide new and inspiring interpretations of both approaches that enable us to derive new PDE models and (better) wavelet frame shrinkage algorithms for image restoration. (2) A generic nonlinear evolution PDE (of parabolic or hyperbolic type) can be approximated by wavelet frame shrinkage with properly chosen wavelet frame systems and carefully designed shrinkage functions. (3) The main idea of this work is beyond the scope of image restoration. Our analysis and discussions indicate that wavelet frame shrinkage is a new way of solving PDEs in general, which will provide a new insight that will enrich the existing theory and applications of numerical PDEs, as well as those of wavelet frames
BLADE: Filter Learning for General Purpose Computational Photography
The Rapid and Accurate Image Super Resolution (RAISR) method of Romano,
Isidoro, and Milanfar is a computationally efficient image upscaling method
using a trained set of filters. We describe a generalization of RAISR, which we
name Best Linear Adaptive Enhancement (BLADE). This approach is a trainable
edge-adaptive filtering framework that is general, simple, computationally
efficient, and useful for a wide range of problems in computational
photography. We show applications to operations which may appear in a camera
pipeline including denoising, demosaicing, and stylization
Hyperbolic Wavelet-Fisz denoising for a model arising in Ultrasound Imaging
International audienceWe present an algorithm and its fully data-driven extension for noise reduction in ultrasound imaging. Our proposed method computes the hyperbolic wavelet transform of the image, before applying a multiscale variance stabilization technique, via a Fisz transformation. This adapts the wavelet coefficients statistics to the wavelet thresholding paradigm. The aim of the hyperbolic setting is to recover the image while respecting the anisotropic nature of structural details. The data-driven extension removes the need for any prior knowledge of the noise model parameters by estimating the noise variance using an isotonic Nadaraya-Watson estimator. Experiments on synthetic and real data, and comparisons with other noise reduction methods demonstrate the potential of our method at recovering ultrasound images while preserving tissue details. Finally, we emphasize the noise model we consider by applying our variance estimation procedure on real images
Feature-preserving image restoration and its application in biological fluorescence microscopy
This thesis presents a new investigation of image restoration and its application to
fluorescence cell microscopy. The first part of the work is to develop advanced image
denoising algorithms to restore images from noisy observations by using a novel featurepreserving
diffusion approach. I have applied these algorithms to different types of
images, including biometric, biological and natural images, and demonstrated their
superior performance for noise removal and feature preservation, compared to several
state of the art methods. In the second part of my work, I explore a novel, simple and
inexpensive super-resolution restoration method for quantitative microscopy in cell
biology. In this method, a super-resolution image is restored, through an inverse process,
by using multiple diffraction-limited (low) resolution observations, which are acquired
from conventional microscopes whilst translating the sample parallel to the image plane,
so referred to as translation microscopy (TRAM). A key to this new development is the
integration of a robust feature detector, developed in the first part, to the inverse process
to restore high resolution images well above the diffraction limit in the presence of strong
noise. TRAM is a post-image acquisition computational method and can be implemented
with any microscope. Experiments show a nearly 7-fold increase in lateral spatial
resolution in noisy biological environments, delivering multi-colour image resolution of
~30 nm
CT Image Reconstruction by Spatial-Radon Domain Data-Driven Tight Frame Regularization
This paper proposes a spatial-Radon domain CT image reconstruction model
based on data-driven tight frames (SRD-DDTF). The proposed SRD-DDTF model
combines the idea of joint image and Radon domain inpainting model of
\cite{Dong2013X} and that of the data-driven tight frames for image denoising
\cite{cai2014data}. It is different from existing models in that both CT image
and its corresponding high quality projection image are reconstructed
simultaneously using sparsity priors by tight frames that are adaptively
learned from the data to provide optimal sparse approximations. An alternative
minimization algorithm is designed to solve the proposed model which is
nonsmooth and nonconvex. Convergence analysis of the algorithm is provided.
Numerical experiments showed that the SRD-DDTF model is superior to the model
by \cite{Dong2013X} especially in recovering some subtle structures in the
images
Structure-aware image denoising, super-resolution, and enhancement methods
Denoising, super-resolution and structure enhancement are classical image processing applications. The motive behind their existence is to aid our visual analysis of raw digital images. Despite tremendous progress in these fields, certain difficult problems are still open to research. For example, denoising and super-resolution techniques which possess all the following properties, are very scarce: They must preserve critical structures like corners, should be robust to the type of noise distribution, avoid undesirable artefacts, and also be fast. The area of structure enhancement also has an unresolved issue: Very little efforts have been put into designing models that can tackle anisotropic deformations in the image acquisition process. In this thesis, we design novel methods in the form of partial differential equations, patch-based approaches and variational models to overcome the aforementioned obstacles. In most cases, our methods outperform the existing approaches in both quality and speed, despite being applicable to a broader range of practical situations.Entrauschen, Superresolution und Strukturverbesserung sind klassische Anwendungen der Bildverarbeitung. Ihre Existenz bedingt sich in dem Bestreben, die visuelle Begutachtung digitaler Bildrohdaten zu unterstützen. Trotz erheblicher Fortschritte in diesen Feldern bedürfen bestimmte schwierige Probleme noch weiterer Forschung. So sind beispielsweise Entrauschungsund Superresolutionsverfahren, welche alle der folgenden Eingenschaften besitzen, sehr selten: die Erhaltung wichtiger Strukturen wie Ecken, Robustheit bezüglich der Rauschverteilung, Vermeidung unerwünschter Artefakte und niedrige Laufzeit. Auch im Gebiet der Strukturverbesserung liegt ein ungelöstes Problem vor: Bisher wurde nur sehr wenig Forschungsaufwand in die Entwicklung von Modellen investieret, welche anisotrope Deformationen in bildgebenden Verfahren bewältigen können. In dieser Arbeit entwerfen wir neue Methoden in Form von partiellen Differentialgleichungen, patch-basierten Ansätzen und Variationsmodellen um die oben erwähnten Hindernisse zu überwinden. In den meisten Fällen übertreffen unsere Methoden nicht nur qualitativ die bisher verwendeten Ansätze, sondern lösen die gestellten Aufgaben auch schneller. Zudem decken wir mit unseren Modellen einen breiteren Bereich praktischer Fragestellungen ab
Statistical Diffusion Tensor Imaging
Magnetic resonance diffusion tensor imaging (DTI) allows to infere the ultrastructure of living tissue. In brain mapping, neural fiber trajectories can be identified by exploiting the anisotropy of diffusion processes. Manifold statistical methods may be linked into the comprehensive processing chain that is spanned between DTI raw images and the reliable visualization of fibers. In this work, a space varying coefficients model (SVCM) using penalized B-splines was developed to integrate diffusion tensor estimation, regularization and interpolation into a unified framework. The implementation challenges originating in multiple 3d space varying coefficient surfaces and the large dimensions of realistic datasets were met by incorporating matrix sparsity and efficient model approximation. Superiority of B-spline based SVCM to the standard approach was demonstrable from simulation studies in terms of the precision and accuracy of the individual tensor elements. The integration with a probabilistic fiber tractography algorithm and application on real brain data revealed that the unified approach is at least equivalent to the serial application of voxelwise estimation, smoothing and interpolation. From the error analysis using boxplots and visual inspection the conclusion was drawn that both the standard approach and the B-spline based SVCM may suffer from low local adaptivity. Therefore, wavelet basis functions were employed for filtering diffusion tensor fields. While excellent local smoothing was indeed achieved by combining voxelwise tensor estimation with wavelet filtering, no immediate improvement was gained for fiber tracking. However, the thresholding strategy needs to be refined and the proposed model of an incorporation of wavelets into an SVCM needs to be implemented to finally assess their utility for DTI data processing.
In summary, an SVCM with specific consideration of the demands of human brain DTI data was developed and implemented, eventually representing a unified postprocessing framework. This represents an experimental and statistical platform to further improve the reliability of tractography
- …