32,195 research outputs found
Successive Convex Approximation Algorithms for Sparse Signal Estimation with Nonconvex Regularizations
In this paper, we propose a successive convex approximation framework for
sparse optimization where the nonsmooth regularization function in the
objective function is nonconvex and it can be written as the difference of two
convex functions. The proposed framework is based on a nontrivial combination
of the majorization-minimization framework and the successive convex
approximation framework proposed in literature for a convex regularization
function. The proposed framework has several attractive features, namely, i)
flexibility, as different choices of the approximate function lead to different
type of algorithms; ii) fast convergence, as the problem structure can be
better exploited by a proper choice of the approximate function and the
stepsize is calculated by the line search; iii) low complexity, as the
approximate function is convex and the line search scheme is carried out over a
differentiable function; iv) guaranteed convergence to a stationary point. We
demonstrate these features by two example applications in subspace learning,
namely, the network anomaly detection problem and the sparse subspace
clustering problem. Customizing the proposed framework by adopting the
best-response type approximation, we obtain soft-thresholding with exact line
search algorithms for which all elements of the unknown parameter are updated
in parallel according to closed-form expressions. The attractive features of
the proposed algorithms are illustrated numerically.Comment: submitted to IEEE Journal of Selected Topics in Signal Processing,
special issue in Robust Subspace Learnin
Algorithms for the continuous nonlinear resource allocation problem---new implementations and numerical studies
Patriksson (2008) provided a then up-to-date survey on the
continuous,separable, differentiable and convex resource allocation problem
with a single resource constraint. Since the publication of that paper the
interest in the problem has grown: several new applications have arisen where
the problem at hand constitutes a subproblem, and several new algorithms have
been developed for its efficient solution. This paper therefore serves three
purposes. First, it provides an up-to-date extension of the survey of the
literature of the field, complementing the survey in Patriksson (2008) with
more then 20 books and articles. Second, it contributes improvements of some of
these algorithms, in particular with an improvement of the pegging (that is,
variable fixing) process in the relaxation algorithm, and an improved means to
evaluate subsolutions. Third, it numerically evaluates several relaxation
(primal) and breakpoint (dual) algorithms, incorporating a variety of pegging
strategies, as well as a quasi-Newton method. Our conclusion is that our
modification of the relaxation algorithm performs the best. At least for
problem sizes up to 30 million variables the practical time complexity for the
breakpoint and relaxation algorithms is linear
Conic Optimization Theory: Convexification Techniques and Numerical Algorithms
Optimization is at the core of control theory and appears in several areas of
this field, such as optimal control, distributed control, system
identification, robust control, state estimation, model predictive control and
dynamic programming. The recent advances in various topics of modern
optimization have also been revamping the area of machine learning. Motivated
by the crucial role of optimization theory in the design, analysis, control and
operation of real-world systems, this tutorial paper offers a detailed overview
of some major advances in this area, namely conic optimization and its emerging
applications. First, we discuss the importance of conic optimization in
different areas. Then, we explain seminal results on the design of hierarchies
of convex relaxations for a wide range of nonconvex problems. Finally, we study
different numerical algorithms for large-scale conic optimization problems.Comment: 18 page
Decomposition by Partial Linearization: Parallel Optimization of Multi-Agent Systems
We propose a novel decomposition framework for the distributed optimization
of general nonconvex sum-utility functions arising naturally in the system
design of wireless multiuser interfering systems. Our main contributions are:
i) the development of the first class of (inexact) Jacobi best-response
algorithms with provable convergence, where all the users simultaneously and
iteratively solve a suitably convexified version of the original sum-utility
optimization problem; ii) the derivation of a general dynamic pricing mechanism
that provides a unified view of existing pricing schemes that are based,
instead, on heuristics; and iii) a framework that can be easily particularized
to well-known applications, giving rise to very efficient practical (Jacobi or
Gauss-Seidel) algorithms that outperform existing adhoc methods proposed for
very specific problems. Interestingly, our framework contains as special cases
well-known gradient algorithms for nonconvex sum-utility problems, and many
blockcoordinate descent schemes for convex functions.Comment: submitted to IEEE Transactions on Signal Processin
Strongly polynomial algorithm for a class of minimum-cost flow problems with separable convex objectives
A well-studied nonlinear extension of the minimum-cost flow problem is to
minimize the objective over feasible flows ,
where on every arc of the network, is a convex function. We give
a strongly polynomial algorithm for the case when all 's are convex
quadratic functions, settling an open problem raised e.g. by Hochbaum [1994].
We also give strongly polynomial algorithms for computing market equilibria in
Fisher markets with linear utilities and with spending constraint utilities,
that can be formulated in this framework (see Shmyrev [2009], Devanur et al.
[2011]). For the latter class this resolves an open question raised by Vazirani
[2010]. The running time is for quadratic costs,
for Fisher's markets with linear utilities and
for spending constraint utilities.
All these algorithms are presented in a common framework that addresses the
general problem setting. Whereas it is impossible to give a strongly polynomial
algorithm for the general problem even in an approximate sense (see Hochbaum
[1994]), we show that assuming the existence of certain black-box oracles, one
can give an algorithm using a strongly polynomial number of arithmetic
operations and oracle calls only. The particular algorithms can be derived by
implementing these oracles in the respective settings
- …