1,699 research outputs found
UNIFYING MIRROR DESCENT AND DUAL AVERAGING
International audienceWe introduce and analyze a new family of first-order optimization algorithms which generalizes and unifies both mirror descent and dual averaging. Within the framework of this family, we define new algorithms for constrained optimization that combines the advantages of mirror descent and dual averaging. Our preliminary simulation study shows that these new algorithms significantly outperform available methods in some situations. 21 References 21 Appendix A. Convex analysis tools 24 Appendix B. Postponed proofs 2
A Family of Subgradient-Based Methods for Convex Optimization Problems in a Unifying Framework
We propose a new family of subgradient- and gradient-based methods which
converges with optimal complexity for convex optimization problems whose
feasible region is simple enough. This includes cases where the objective
function is non-smooth, smooth, have composite/saddle structure, or are given
by an inexact oracle model. We unified the way of constructing the subproblems
which are necessary to be solved at each iteration of these methods. This
permitted us to analyze the convergence of these methods in a unified way
compared to previous results which required different approaches for each
method/algorithm. Our contribution rely on two well-known methods in non-smooth
convex optimization: the mirror-descent method by Nemirovski-Yudin and the
dual-averaging method by Nesterov. Therefore, our family of methods includes
them and many other methods as particular cases. For instance, the proposed
family of classical gradient methods and its accelerations generalize Devolder
et al.'s, Nesterov's primal/dual gradient methods, and Tseng's accelerated
proximal gradient methods. Also our family of methods can partially become
special cases of other universal methods, too. As an additional contribution,
the novel extended mirror-descent method removes the compactness assumption of
the feasible region and the fixation of the total number of iterations which is
required by the original mirror-descent method in order to attain the optimal
complexity.Comment: 31 pages. v3: Major revision. Research Report B-477, Department of
Mathematical and Computing Sciences, Tokyo Institute of Technology, February
201
A Generalized Online Mirror Descent with Applications to Classification and Regression
Online learning algorithms are fast, memory-efficient, easy to implement, and
applicable to many prediction problems, including classification, regression,
and ranking. Several online algorithms were proposed in the past few decades,
some based on additive updates, like the Perceptron, and some on multiplicative
updates, like Winnow. A unifying perspective on the design and the analysis of
online algorithms is provided by online mirror descent, a general prediction
strategy from which most first-order algorithms can be obtained as special
cases. We generalize online mirror descent to time-varying regularizers with
generic updates. Unlike standard mirror descent, our more general formulation
also captures second order algorithms, algorithms for composite losses and
algorithms for adaptive filtering. Moreover, we recover, and sometimes improve,
known regret bounds as special cases of our analysis using specific
regularizers. Finally, we show the power of our approach by deriving a new
second order algorithm with a regret bound invariant with respect to arbitrary
rescalings of individual features
- …