24,818 research outputs found
A Generic Path Algorithm for Regularized Statistical Estimation
Regularization is widely used in statistics and machine learning to prevent
overfitting and gear solution towards prior information. In general, a
regularized estimation problem minimizes the sum of a loss function and a
penalty term. The penalty term is usually weighted by a tuning parameter and
encourages certain constraints on the parameters to be estimated. Particular
choices of constraints lead to the popular lasso, fused-lasso, and other
generalized penalized regression methods. Although there has been a lot
of research in this area, developing efficient optimization methods for many
nonseparable penalties remains a challenge. In this article we propose an exact
path solver based on ordinary differential equations (EPSODE) that works for
any convex loss function and can deal with generalized penalties as well
as more complicated regularization such as inequality constraints encountered
in shape-restricted regressions and nonparametric density estimation. In the
path following process, the solution path hits, exits, and slides along the
various constraints and vividly illustrates the tradeoffs between goodness of
fit and model parsimony. In practice, the EPSODE can be coupled with AIC, BIC,
or cross-validation to select an optimal tuning parameter. Our
applications to generalized regularized generalized linear models,
shape-restricted regressions, Gaussian graphical models, and nonparametric
density estimation showcase the potential of the EPSODE algorithm.Comment: 28 pages, 5 figure
Robust computation of linear models by convex relaxation
Consider a dataset of vector-valued observations that consists of noisy
inliers, which are explained well by a low-dimensional subspace, along with
some number of outliers. This work describes a convex optimization problem,
called REAPER, that can reliably fit a low-dimensional model to this type of
data. This approach parameterizes linear subspaces using orthogonal projectors,
and it uses a relaxation of the set of orthogonal projectors to reach the
convex formulation. The paper provides an efficient algorithm for solving the
REAPER problem, and it documents numerical experiments which confirm that
REAPER can dependably find linear structure in synthetic and natural data. In
addition, when the inliers lie near a low-dimensional subspace, there is a
rigorous theory that describes when REAPER can approximate this subspace.Comment: Formerly titled "Robust computation of linear models, or How to find
a needle in a haystack
Democratic Representations
Minimization of the (or maximum) norm subject to a constraint
that imposes consistency to an underdetermined system of linear equations finds
use in a large number of practical applications, including vector quantization,
approximate nearest neighbor search, peak-to-average power ratio (or "crest
factor") reduction in communication systems, and peak force minimization in
robotics and control. This paper analyzes the fundamental properties of signal
representations obtained by solving such a convex optimization problem. We
develop bounds on the maximum magnitude of such representations using the
uncertainty principle (UP) introduced by Lyubarskii and Vershynin, and study
the efficacy of -norm-based dynamic range reduction. Our
analysis shows that matrices satisfying the UP, such as randomly subsampled
Fourier or i.i.d. Gaussian matrices, enable the computation of what we call
democratic representations, whose entries all have small and similar magnitude,
as well as low dynamic range. To compute democratic representations at low
computational complexity, we present two new, efficient convex optimization
algorithms. We finally demonstrate the efficacy of democratic representations
for dynamic range reduction in a DVB-T2-based broadcast system.Comment: Submitted to a Journa
Radio Astronomical Image Formation using Constrained Least Squares and Krylov Subspaces
Image formation for radio astronomy can be defined as estimating the spatial
power distribution of celestial sources over the sky, given an array of
antennas. One of the challenges with image formation is that the problem
becomes ill-posed as the number of pixels becomes large. The introduction of
constraints that incorporate a-priori knowledge is crucial. In this paper we
show that in addition to non-negativity, the magnitude of each pixel in an
image is also bounded from above. Indeed, the classical "dirty image" is an
upper bound, but a much tighter upper bound can be formed from the data using
array processing techniques. This formulates image formation as a least squares
optimization problem with inequality constraints. We propose to solve this
constrained least squares problem using active set techniques, and the steps
needed to implement it are described. It is shown that the least squares part
of the problem can be efficiently implemented with Krylov subspace based
techniques, where the structure of the problem allows massive parallelism and
reduced storage needs. The performance of the algorithm is evaluated using
simulations
- β¦