3,484 research outputs found
On the Minimization of Convex Functionals of Probability Distributions Under Band Constraints
The problem of minimizing convex functionals of probability distributions is
solved under the assumption that the density of every distribution is bounded
from above and below. A system of sufficient and necessary first-order
optimality conditions as well as a bound on the optimality gap of feasible
candidate solutions are derived. Based on these results, two numerical
algorithms are proposed that iteratively solve the system of optimality
conditions on a grid of discrete points. Both algorithms use a block coordinate
descent strategy and terminate once the optimality gap falls below the desired
tolerance. While the first algorithm is conceptually simpler and more
efficient, it is not guaranteed to converge for objective functions that are
not strictly convex. This shortcoming is overcome in the second algorithm,
which uses an additional outer proximal iteration, and, which is proven to
converge under mild assumptions. Two examples are given to demonstrate the
theoretical usefulness of the optimality conditions as well as the high
efficiency and accuracy of the proposed numerical algorithms.Comment: 13 pages, 5 figures, 2 tables, published in the IEEE Transactions on
Signal Processing. In previous versions, the example in Section VI.B
contained some mistakes and inaccuracies, which have been fixed in this
versio
Fixed-point and coordinate descent algorithms for regularized kernel methods
In this paper, we study two general classes of optimization algorithms for
kernel methods with convex loss function and quadratic norm regularization, and
analyze their convergence. The first approach, based on fixed-point iterations,
is simple to implement and analyze, and can be easily parallelized. The second,
based on coordinate descent, exploits the structure of additively separable
loss functions to compute solutions of line searches in closed form. Instances
of these general classes of algorithms are already incorporated into state of
the art machine learning software for large scale problems. We start from a
solution characterization of the regularized problem, obtained using
sub-differential calculus and resolvents of monotone operators, that holds for
general convex loss functions regardless of differentiability. The two
methodologies described in the paper can be regarded as instances of non-linear
Jacobi and Gauss-Seidel algorithms, and are both well-suited to solve large
scale problems
A Smoothed Dual Approach for Variational Wasserstein Problems
Variational problems that involve Wasserstein distances have been recently
proposed to summarize and learn from probability measures. Despite being
conceptually simple, such problems are computationally challenging because they
involve minimizing over quantities (Wasserstein distances) that are themselves
hard to compute. We show that the dual formulation of Wasserstein variational
problems introduced recently by Carlier et al. (2014) can be regularized using
an entropic smoothing, which leads to smooth, differentiable, convex
optimization problems that are simpler to implement and numerically more
stable. We illustrate the versatility of this approach by applying it to the
computation of Wasserstein barycenters and gradient flows of spacial
regularization functionals
A Singular Value Thresholding Algorithm for Matrix Completion
This paper introduces a novel algorithm to approximate the matrix with minimum
nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood
as the convex relaxation of a rank minimization problem and arises in many important
applications as in the task of recovering a large matrix from a small subset of its entries (the famous
Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable
to large problems of this kind with over a million unknown entries. This paper develops a simple
first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in
which the optimal solution has low rank. The algorithm is iterative, produces a sequence of matrices
{X^k,Y^k}, and at each step mainly performs a soft-thresholding operation on the singular values
of the matrix Y^k. There are two remarkable features making this attractive for low-rank matrix
completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix;
the second is that the rank of the iterates {X^k} is empirically nondecreasing. Both these facts allow
the algorithm to make use of very minimal storage space and keep the computational cost of each
iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence
of iterates converges. On the practical side, we provide numerical examples in which 1,000 × 1,000
matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate
that our approach is amenable to very large scale problems by recovering matrices of rank about
10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are
connected with the recent literature on linearized Bregman iterations for ℓ_1 minimization, and we
develop a framework in which one can understand these algorithms in terms of well-known Lagrange
multiplier algorithms
Minimization of multi-penalty functionals by alternating iterative thresholding and optimal parameter choices
Inspired by several recent developments in regularization theory,
optimization, and signal processing, we present and analyze a numerical
approach to multi-penalty regularization in spaces of sparsely represented
functions. The sparsity prior is motivated by the largely expected
geometrical/structured features of high-dimensional data, which may not be
well-represented in the framework of typically more isotropic Hilbert spaces.
In this paper, we are particularly interested in regularizers which are able to
correctly model and separate the multiple components of additively mixed
signals. This situation is rather common as pure signals may be corrupted by
additive noise. To this end, we consider a regularization functional composed
by a data-fidelity term, where signal and noise are additively mixed, a
non-smooth and non-convex sparsity promoting term, and a penalty term to model
the noise. We propose and analyze the convergence of an iterative alternating
algorithm based on simple iterative thresholding steps to perform the
minimization of the functional. By means of this algorithm, we explore the
effect of choosing different regularization parameters and penalization norms
in terms of the quality of recovering the pure signal and separating it from
additive noise. For a given fixed noise level numerical experiments confirm a
significant improvement in performance compared to standard one-parameter
regularization methods. By using high-dimensional data analysis methods such as
Principal Component Analysis, we are able to show the correct geometrical
clustering of regularized solutions around the expected solution. Eventually,
for the compressive sensing problems considered in our experiments we provide a
guideline for a choice of regularization norms and parameters.Comment: 32 page
- …