MDL Denoising Revisited

We refine and extend an earlier MDL denoising criterion for wavelet-based denoising. We start by showing that the denoising problem can be reformulated as a clustering problem, where the goal is to obtain separate clusters for informative and non-informative wavelet coefficients, respectively. This suggests two refinements, adding a code-length for the model index, and extending the model in order to account for subband-dependent coefficient distributions. A third refinement is derivation of soft thresholding inspired by predictive universal coding with weighted mixtures. We propose a practical method incorporating all three refinements, which is shown to achieve good performance and robustness in denoising both artificial and natural signals.Comment: Submitted to IEEE Transactions on Information Theory, June 200

Learning to compress and search visual data in large-scale systems

The problem of high-dimensional and large-scale representation of visual data is addressed from an unsupervised learning perspective. The emphasis is put on discrete representations, where the description length can be measured in bits and hence the model capacity can be controlled. The algorithmic infrastructure is developed based on the synthesis and analysis prior models whose rate-distortion properties, as well as capacity vs. sample complexity trade-offs are carefully optimized. These models are then extended to multi-layers, namely the RRQ and the ML-STC frameworks, where the latter is further evolved as a powerful deep neural network architecture with fast and sample-efficient training and discrete representations. For the developed algorithms, three important applications are developed. First, the problem of large-scale similarity search in retrieval systems is addressed, where a double-stage solution is proposed leading to faster query times and shorter database storage. Second, the problem of learned image compression is targeted, where the proposed models can capture more redundancies from the training images than the conventional compression codecs. Finally, the proposed algorithms are used to solve ill-posed inverse problems. In particular, the problems of image denoising and compressive sensing are addressed with promising results.Comment: PhD thesis dissertatio

Statistical Multiresolution Estimatiors in Linear Inverse Problems - Foundations and Algorithmic Aspects

In jüngerer Vergangenheit haben statistische Multiresolutionstechniken viel Aufmerksamkeit erregt. Dies liegt vor allem daran, dass die hieraus resultierenden statistischen Multiresolutionsschätzer (SMR) lokal- und multiskalenadaptiv sind, das heißt, dass sie sich automatisch der Glattheit des wahren Objects lokal und auf verschiedenen Skalen anpassen. In dieser Dissertation wird eine neuartige Methodik eingeführt, um SMR-Schätzer in der Praxis zu berechnen.Hierzu werden SMR-Schätzer rigoros als Lösung eines Optimierungsproblems mit Nebenbedingungen definiert. Neben einer Herleitung dieses Ansatzes wird auch ein Konsistenzresultat erbracht. Die eigentliche Berechnung wird dann über eine Augmented-Lagrangian-Methode durchgeführt, mittels derer das Problem in ein unrestringiertes Minimierungsproblem und ein hochskaliges Projektionsproblem zerlegt wird. Letzteres wird durch den Dykstra-Algorithmius attackiert; eine Methode, welche die Projektion auf den Schnitt von abgeschlossenen und konvexen Mengen berechnet, indem sie sukzessive auf die einzelnen Mengen projiziert. Diese individuellen Projektionen können im hier vorliegenden Kontext explizit angegeben werden, wodurch der Dykstra-Algorithmus zu einer besonders schnellen und somit attraktiven Lösungsmethode wird.Hierdurch können mit unserer Methodik auch vergleichsweise große Datensätze behandelt werden. Insbesondere können zweidimensionale Datensätze bearbeitet werden, während die meisten Publikationen in diesem Themenbereich bislang auf ein eindimensionales Rahmenwerk beschränkt waren. Auf Regressionsprobleme angewendet liefert unsere Methode bessere Ergebnisse als andere aktuelle Methoden im Bereich der SMR-Schätzung. Darüber hinaus ist unser Algorithmus der erste, welcher die Berechnung von SMR-Schätzern in (möglicherweise schlecht-gestellten) inversen Problemen ermöglicht und kann mit einer Vielzahl von Straffunktionalen kombiniert werden

Model Based Principal Component Analysis with Application to Functional Magnetic Resonance Imaging.

Functional Magnetic Resonance Imaging (fMRI) has allowed better understanding of human brain organization and function by making it possible to record either autonomous or stimulus induced brain activity. After appropriate preprocessing fMRI produces a large spatio-temporal data set, which requires sophisticated signal processing. The aim of the signal processing is usually to produce spatial maps of statistics that capture the effects of interest, e.g., brain activation, time delay between stimulation and activation, or connectivity between brain regions. Two broad signal processing approaches have been pursued; univoxel methods and multivoxel methods. This proposal will focus on multivoxel methods and review Principal Component Analysis (PCA), and other closely related methods, and describe their advantages and disadvantages in fMRI research. These existing multivoxel methods have in common that they are exploratory, i.e., they are not based on a statistical model. A crucial observation which is central to this thesis, is that there is in fact an underlying model behind PCA, which we call noisy PCA (nPCA). In the main part of this thesis, we use nPCA to develop methods that solve three important problems in fMRI. 1) We introduce a novel nPCA based spatio-temporal model that combines the standard univoxel regression model with nPCA and automatically recognizes the temporal smoothness of the fMRI data. Furthermore, unlike standard univoxel methods, it can handle non-stationary noise. 2) We introduce a novel sparse variable PCA (svPCA) method that automatically excludes whole voxel timeseries, and yields sparse eigenimages. This is achieved by a novel nonlinear penalized likelihood function which is optimized. An iterative estimation algorithm is proposed that makes use of geodesic descent methods. 3) We introduce a novel method based on Stein’s Unbiased Risk Estimator (SURE) and Random Matrix Theory (RMT) to select the number of principal components for the increasingly important case where the number of observations is of similar order as the number of variables.Ph.D.Electrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/57638/2/mulfarss_1.pd

Complexity-Regularized Image Denoising

We describe a new approach to image denoising based on complexity regularization techniques. This approach represents a flexible alternative to the more conventional l 2 , l 1 , and Besov regularization techniques. Di#erent forms of complexity regularization are studied. We derive a connection between complexity--regularized denoising and operational rate--distortion optimization for a Gaussian denoising problem, and show how to apply state--of--the--art image coders to this problem. Our complexity--regularized estimators inherit the flexibility and robustness of those coders. We establish bounds on denoising performance in terms of an index of resolvability that characterizes the compressibility of the true image. Compared to Donoho and Johnstone's wavelet thresholding method, which can also be regarded as a simple complexity--regularized estimator, we find that the use of sophisticated complexity measures yields significant improvements in denoised image quality. Keywords --- ima..