836 research outputs found
Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization
Majorization-minimization algorithms consist of iteratively minimizing a
majorizing surrogate of an objective function. Because of its simplicity and
its wide applicability, this principle has been very popular in statistics and
in signal processing. In this paper, we intend to make this principle scalable.
We introduce a stochastic majorization-minimization scheme which is able to
deal with large-scale or possibly infinite data sets. When applied to convex
optimization problems under suitable assumptions, we show that it achieves an
expected convergence rate of after iterations, and of
for strongly convex functions. Equally important, our scheme almost
surely converges to stationary points for a large class of non-convex problems.
We develop several efficient algorithms based on our framework. First, we
propose a new stochastic proximal gradient method, which experimentally matches
state-of-the-art solvers for large-scale -logistic regression. Second,
we develop an online DC programming algorithm for non-convex sparse estimation.
Finally, we demonstrate the effectiveness of our approach for solving
large-scale structured matrix factorization problems.Comment: accepted for publication for Neural Information Processing Systems
(NIPS) 2013. This is the 9-pages version followed by 16 pages of appendices.
The title has changed compared to the first technical repor
Stochastic Variance Reduction Methods for Saddle-Point Problems
We consider convex-concave saddle-point problems where the objective
functions may be split in many components, and extend recent stochastic
variance reduction methods (such as SVRG or SAGA) to provide the first
large-scale linearly convergent algorithms for this class of problems which is
common in machine learning. While the algorithmic extension is straightforward,
it comes with challenges and opportunities: (a) the convex minimization
analysis does not apply and we use the notion of monotone operators to prove
convergence, showing in particular that the same algorithm applies to a larger
class of problems, such as variational inequalities, (b) there are two notions
of splits, in terms of functions, or in terms of partial derivatives, (c) the
split does need to be done with convex-concave terms, (d) non-uniform sampling
is key to an efficient algorithm, both in theory and practice, and (e) these
incremental algorithms can be easily accelerated using a simple extension of
the "catalyst" framework, leading to an algorithm which is always superior to
accelerated batch algorithms.Comment: Neural Information Processing Systems (NIPS), 2016, Barcelona, Spai
System Level Synthesis
This article surveys the System Level Synthesis framework, which presents a
novel perspective on constrained robust and optimal controller synthesis for
linear systems. We show how SLS shifts the controller synthesis task from the
design of a controller to the design of the entire closed loop system, and
highlight the benefits of this approach in terms of scalability and
transparency. We emphasize two particular applications of SLS, namely
large-scale distributed optimal control and robust control. In the case of
distributed control, we show how SLS allows for localized controllers to be
computed, extending robust and optimal control methods to large-scale systems
under practical and realistic assumptions. In the case of robust control, we
show how SLS allows for novel design methodologies that, for the first time,
quantify the degradation in performance of a robust controller due to model
uncertainty -- such transparency is key in allowing robust control methods to
interact, in a principled way, with modern techniques from machine learning and
statistical inference. Throughout, we emphasize practical and efficient
computational solutions, and demonstrate our methods on easy to understand case
studies.Comment: To appear in Annual Reviews in Contro
- …