10,874 research outputs found
Covariance Estimation: The GLM and Regularization Perspectives
Finding an unconstrained and statistically interpretable reparameterization
of a covariance matrix is still an open problem in statistics. Its solution is
of central importance in covariance estimation, particularly in the recent
high-dimensional data environment where enforcing the positive-definiteness
constraint could be computationally expensive. We provide a survey of the
progress made in modeling covariance matrices from two relatively complementary
perspectives: (1) generalized linear models (GLM) or parsimony and use of
covariates in low dimensions, and (2) regularization or sparsity for
high-dimensional data. An emerging, unifying and powerful trend in both
perspectives is that of reducing a covariance estimation problem to that of
estimating a sequence of regression problems. We point out several instances of
the regression-based formulation. A notable case is in sparse estimation of a
precision matrix or a Gaussian graphical model leading to the fast graphical
LASSO algorithm. Some advantages and limitations of the regression-based
Cholesky decomposition relative to the classical spectral (eigenvalue) and
variance-correlation decompositions are highlighted. The former provides an
unconstrained and statistically interpretable reparameterization, and
guarantees the positive-definiteness of the estimated covariance matrix. It
reduces the unintuitive task of covariance estimation to that of modeling a
sequence of regressions at the cost of imposing an a priori order among the
variables. Elementwise regularization of the sample covariance matrix such as
banding, tapering and thresholding has desirable asymptotic properties and the
sparse estimated covariance matrix is positive definite with probability
tending to one for large samples and dimensions.Comment: Published in at http://dx.doi.org/10.1214/11-STS358 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Sparse Modeling for Image and Vision Processing
In recent years, a large amount of multi-disciplinary research has been
conducted on sparse models and their applications. In statistics and machine
learning, the sparsity principle is used to perform model selection---that is,
automatically selecting a simple model among a large collection of them. In
signal processing, sparse coding consists of representing data with linear
combinations of a few dictionary elements. Subsequently, the corresponding
tools have been widely adopted by several scientific communities such as
neuroscience, bioinformatics, or computer vision. The goal of this monograph is
to offer a self-contained view of sparse modeling for visual recognition and
image processing. More specifically, we focus on applications where the
dictionary is learned and adapted to data, yielding a compact representation
that has been successful in various contexts.Comment: 205 pages, to appear in Foundations and Trends in Computer Graphics
and Visio
Simultaneously Structured Models with Application to Sparse and Low-rank Matrices
The topic of recovery of a structured model given a small number of linear
observations has been well-studied in recent years. Examples include recovering
sparse or group-sparse vectors, low-rank matrices, and the sum of sparse and
low-rank matrices, among others. In various applications in signal processing
and machine learning, the model of interest is known to be structured in
several ways at the same time, for example, a matrix that is simultaneously
sparse and low-rank.
Often norms that promote each individual structure are known, and allow for
recovery using an order-wise optimal number of measurements (e.g.,
norm for sparsity, nuclear norm for matrix rank). Hence, it is reasonable to
minimize a combination of such norms. We show that, surprisingly, if we use
multi-objective optimization with these norms, then we can do no better,
order-wise, than an algorithm that exploits only one of the present structures.
This result suggests that to fully exploit the multiple structures, we need an
entirely new convex relaxation, i.e. not one that is a function of the convex
relaxations used for each structure. We then specialize our results to the case
of sparse and low-rank matrices. We show that a nonconvex formulation of the
problem can recover the model from very few measurements, which is on the order
of the degrees of freedom of the matrix, whereas the convex problem obtained
from a combination of the and nuclear norms requires many more
measurements. This proves an order-wise gap between the performance of the
convex and nonconvex recovery problems in this case. Our framework applies to
arbitrary structure-inducing norms as well as to a wide range of measurement
ensembles. This allows us to give performance bounds for problems such as
sparse phase retrieval and low-rank tensor completion.Comment: 38 pages, 9 figure
Image Fusion via Sparse Regularization with Non-Convex Penalties
The L1 norm regularized least squares method is often used for finding sparse
approximate solutions and is widely used in 1-D signal restoration. Basis
pursuit denoising (BPD) performs noise reduction in this way. However, the
shortcoming of using L1 norm regularization is the underestimation of the true
solution. Recently, a class of non-convex penalties have been proposed to
improve this situation. This kind of penalty function is non-convex itself, but
preserves the convexity property of the whole cost function. This approach has
been confirmed to offer good performance in 1-D signal denoising. This paper
demonstrates the aforementioned method to 2-D signals (images) and applies it
to multisensor image fusion. The problem is posed as an inverse one and a
corresponding cost function is judiciously designed to include two data
attachment terms. The whole cost function is proved to be convex upon suitably
choosing the non-convex penalty, so that the cost function minimization can be
tackled by convex optimization approaches, which comprise simple computations.
The performance of the proposed method is benchmarked against a number of
state-of-the-art image fusion techniques and superior performance is
demonstrated both visually and in terms of various assessment measures
Sufficient Dimension Reduction and Modeling Responses Conditioned on Covariates: An Integrated Approach via Convex Optimization
Given observations of a collection of covariates and responses , sufficient dimension reduction (SDR)
techniques aim to identify a mapping
with such that is independent of . The image
summarizes the relevant information in a potentially large number of covariates
that influence the responses . In many contemporary settings, the number
of responses is also quite large, in addition to a large number of
covariates. This leads to the challenge of fitting a succinctly parameterized
statistical model to , which is a problem that is usually not addressed
in a traditional SDR framework. In this paper, we present a computationally
tractable convex relaxation based estimator for simultaneously (a) identifying
a linear dimension reduction of the covariates that is sufficient with
respect to the responses, and (b) fitting several types of structured
low-dimensional models -- factor models, graphical models, latent-variable
graphical models -- to the conditional distribution of . We analyze the
consistency properties of our estimator in a high-dimensional scaling regime.
We also illustrate the performance of our approach on a newsgroup dataset and
on a dataset consisting of financial asset prices.Comment: 34 pages, 1 figur
- …