6,466 research outputs found
A Unified Primal Dual Active Set Algorithm for Nonconvex Sparse Recovery
In this paper, we consider the problem of recovering a sparse signal based on
penalized least squares formulations. We develop a novel algorithm of
primal-dual active set type for a class of nonconvex sparsity-promoting
penalties, including , bridge, smoothly clipped absolute deviation,
capped and minimax concavity penalty. First we establish the existence
of a global minimizer for the related optimization problems. Then we derive a
novel necessary optimality condition for the global minimizer using the
associated thresholding operator. The solutions to the optimality system are
coordinate-wise minimizers, and under minor conditions, they are also local
minimizers. Upon introducing the dual variable, the active set can be
determined using the primal and dual variables together. Further, this relation
lends itself to an iterative algorithm of active set type which at each step
involves first updating the primal variable only on the active set and then
updating the dual variable explicitly. When combined with a continuation
strategy on the regularization parameter, the primal dual active set method is
shown to converge globally to the underlying regression target under certain
regularity conditions. Extensive numerical experiments with both simulated and
real data demonstrate its superior performance in efficiency and accuracy
compared with the existing sparse recovery methods
The Sparse Principal Component Analysis Problem: Optimality Conditions and Algorithms
Sparse principal component analysis addresses the problem of finding a linear
combination of the variables in a given data set with a sparse coefficients
vector that maximizes the variability of the data. This model enhances the
ability to interpret the principal components, and is applicable in a wide
variety of fields including genetics and finance, just to name a few.
We suggest a necessary coordinate-wise-based optimality condition, and show
its superiority over the stationarity-based condition that is commonly used in
the literature, and which is the basis for many of the algorithms designed to
solve the problem. We devise algorithms that are based on the new optimality
condition, and provide numerical experiments that support our assertion that
algorithms, which are guaranteed to converge to stronger optimality conditions,
perform better than algorithms that converge to points satisfying weaker
optimality conditions
On the Linear Convergence of the Approximate Proximal Splitting Method for Non-Smooth Convex Optimization
Consider the problem of minimizing the sum of two convex functions, one being
smooth and the other non-smooth. In this paper, we introduce a general class of
approximate proximal splitting (APS) methods for solving such minimization
problems. Methods in the APS class include many well-known algorithms such as
the proximal splitting method (PSM), the block coordinate descent method (BCD)
and the approximate gradient projection methods for smooth convex optimization.
We establish the linear convergence of APS methods under a local error bound
assumption. Since the latter is known to hold for compressive sensing and
sparse group LASSO problems, our analysis implies the linear convergence of the
BCD method for these problems without strong convexity assumption.Comment: 21 pages, no figure
Convex Global 3D Registration with Lagrangian Duality
The registration of 3D models by a Euclidean transformation is a fundamental task at the core of many application in computer vision. This problem is non-convex due to the presence of rotational constraints, making traditional local optimization methods prone to getting stuck in local minima. This paper addresses finding the globally optimal transformation in various 3D registration problems by a unified formulation that integrates common geometric registration modalities (namely point-to-point, point-to-line and point-to-plane). This formulation renders the optimization problem independent of both the number and nature of the correspondences.
The main novelty of our proposal is the introduction of a strengthened Lagrangian dual relaxation for this problem, which surpasses previous similar approaches [32] in effectiveness.
In fact, even though with no theoretical guarantees, exhaustive empirical evaluation in both synthetic and real experiments always resulted on a tight relaxation that allowed to recover a guaranteed globally optimal solution by exploiting duality theory.
Thus, our approach allows for effectively solving the 3D registration with global optimality guarantees while running at a fraction of the time for the state-of-the-art alternative [34], based on a more computationally intensive Branch and Bound method.Universidad de Málaga. Campus de Excelencia Internacional AndalucÃa Tech
Screening Rules for Lasso with Non-Convex Sparse Regularizers
Leveraging on the convexity of the Lasso problem , screening rules help in
accelerating solvers by discarding irrelevant variables, during the
optimization process. However, because they provide better theoretical
guarantees in identifying relevant variables, several non-convex regularizers
for the Lasso have been proposed in the literature. This work is the first that
introduces a screening rule strategy into a non-convex Lasso solver. The
approach we propose is based on a iterative majorization-minimization (MM)
strategy that includes a screening rule in the inner solver and a condition for
propagating screened variables between iterations of MM. In addition to improve
efficiency of solvers, we also provide guarantees that the inner solver is able
to identify the zeros components of its critical point in finite time. Our
experimental analysis illustrates the significant computational gain brought by
the new screening rule compared to classical coordinate-descent or proximal
gradient descent methods
A consistent operator splitting algorithm and a two-metric variant: Application to topology optimization
In this work, we explore the use of operator splitting algorithms for solving
regularized structural topology optimization problems. The context is the
classical structural design problems (e.g., compliance minimization and
compliant mechanism design), parameterized by means of density functions, whose
ill-posendess is addressed by introducing a Tikhonov regularization term. The
proposed forward-backward splitting algorithm treats the constituent terms of
the cost functional separately which allows suitable approximations of the
structural objective. We will show that one such approximation, inspired by the
optimality criteria algorithm and reciprocal expansions, improves the
convergence characteristics and leads to an update scheme that resembles the
well-known heuristic sensitivity filtering method. We also discuss a two-metric
variant of the splitting algorithm that removes the computational overhead
associated with bound constraints on the density field without compromising
convergence and quality of optimal solutions. We present several numerical
results and investigate the influence of various algorithmic parameters
Accelerated Parallel and Distributed Algorithm using Limited Internal Memory for Nonnegative Matrix Factorization
Nonnegative matrix factorization (NMF) is a powerful technique for dimension
reduction, extracting latent factors and learning part-based representation.
For large datasets, NMF performance depends on some major issues: fast
algorithms, fully parallel distributed feasibility and limited internal memory.
This research aims to design a fast fully parallel and distributed algorithm
using limited internal memory to reach high NMF performance for large datasets.
In particular, we propose a flexible accelerated algorithm for NMF with all its
regularized variants based on full decomposition, which is a
combination of an anti-lopsided algorithm and a fast block coordinate descent
algorithm. The proposed algorithm takes advantages of both these algorithms to
achieve a linear convergence rate of in
optimizing each factor matrix when fixing the other factor one in the sub-space
of passive variables, where is the number of latent components; where
. In addition, the algorithm can exploit the data
sparseness to run on large datasets with limited internal memory of machines.
Furthermore, our experimental results are highly competitive with 7
state-of-the-art methods about three significant aspects of convergence,
optimality and average of the iteration number. Therefore, the proposed
algorithm is superior to fast block coordinate descent methods and accelerated
methods
An equivalence between stationary points for rank constraints versus low-rank factorizations
Two common approaches in low-rank optimization problems are either working
directly with a rank constraint on the matrix variable, or optimizing over a
low-rank factorization so that the rank constraint is implicitly ensured. In
this paper, we study the natural connection between the rank-constrained and
factorized approaches. We show that all second-order stationary points of the
factorized objective function correspond to stationary points of projected
gradient descent run on the original problem (where the projection step
enforces the rank constraint). This result allows us to unify many existing
optimization guarantees that have been proved specifically in either the
rank-constrained or the factorized setting, and leads to new results for
certain settings of the problem
Certifiably Optimal Low Rank Factor Analysis
Factor Analysis (FA) is a technique of fundamental importance that is widely
used in classical and modern multivariate statistics, psychometrics and
econometrics. In this paper, we revisit the classical rank-constrained FA
problem, which seeks to approximate an observed covariance matrix
(), by the sum of a Positive Semidefinite (PSD) low-rank
component () and a diagonal matrix () (with
nonnegative entries) subject to being
PSD. We propose a flexible family of rank-constrained, nonlinear Semidefinite
Optimization based formulations for this task. We introduce a reformulation of
the problem as a smooth optimization problem with convex compact constraints
and propose a unified algorithmic framework, utilizing state of the art
techniques in nonlinear optimization to obtain high-quality feasible solutions
for our proposed formulation. At the same time, by using a variety of
techniques from discrete and global optimization, we show that these solutions
are certifiably optimal in many cases, even for problems with thousands of
variables. Our techniques are general and make no assumption on the underlying
problem data. The estimator proposed herein, aids statistical interpretability,
provides computational scalability and significantly improved accuracy when
compared to current, publicly available popular methods for rank-constrained
FA. We demonstrate the effectiveness of our proposal on an array of synthetic
and real-life datasets. To our knowledge, this is the first paper that
demonstrates how a previously intractable rank-constrained optimization problem
can be solved to provable optimality by coupling developments in convex
analysis and in discrete optimization
Fixed Points of Generalized Approximate Message Passing with Arbitrary Matrices
The estimation of a random vector with independent components passed through
a linear transform followed by a componentwise (possibly nonlinear) output map
arises in a range of applications. Approximate message passing (AMP) methods,
based on Gaussian approximations of loopy belief propagation, have recently
attracted considerable attention for such problems. For large random
transforms, these methods exhibit fast convergence and admit precise analytic
characterizations with testable conditions for optimality, even for certain
non-convex problem instances. However, the behavior of AMP under general
transforms is not fully understood. In this paper, we consider the generalized
AMP (GAMP) algorithm and relate the method to more common optimization
techniques. This analysis enables a precise characterization of the GAMP
algorithm fixed-points that applies to arbitrary transforms. In particular, we
show that the fixed points of the so-called max-sum GAMP algorithm for MAP
estimation are critical points of a constrained maximization of the posterior
density. The fixed-points of the sum-product GAMP algorithm for estimation of
the posterior marginals can be interpreted as critical points of a certain free
energy
- …