44 research outputs found
SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
We propose a new algorithm for minimizing regularized empirical loss:
Stochastic Dual Newton Ascent (SDNA). Our method is dual in nature: in each
iteration we update a random subset of the dual variables. However, unlike
existing methods such as stochastic dual coordinate ascent, SDNA is capable of
utilizing all curvature information contained in the examples, which leads to
striking improvements in both theory and practice - sometimes by orders of
magnitude. In the special case when an L2-regularizer is used in the primal,
the dual problem is a concave quadratic maximization problem plus a separable
term. In this regime, SDNA in each step solves a proximal subproblem involving
a random principal submatrix of the Hessian of the quadratic function; whence
the name of the method. If, in addition, the loss functions are quadratic, our
method can be interpreted as a novel variant of the recently introduced
Iterative Hessian Sketch
Semistochastic Quadratic Bound Methods
Partition functions arise in a variety of settings, including conditional
random fields, logistic regression, and latent gaussian models. In this paper,
we consider semistochastic quadratic bound (SQB) methods for maximum likelihood
inference based on partition function optimization. Batch methods based on the
quadratic bound were recently proposed for this class of problems, and
performed favorably in comparison to state-of-the-art techniques.
Semistochastic methods fall in between batch algorithms, which use all the
data, and stochastic gradient type methods, which use small random selections
at each iteration. We build semistochastic quadratic bound-based methods, and
prove both global convergence (to a stationary point) under very weak
assumptions, and linear convergence rate under stronger assumptions on the
objective. To make the proposed methods faster and more stable, we consider
inexact subproblem minimization and batch-size selection schemes. The efficacy
of SQB methods is demonstrated via comparison with several state-of-the-art
techniques on commonly used datasets.Comment: 11 pages, 1 figur
Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers
We consider the setting where a master wants to run a distributed stochastic
gradient descent (SGD) algorithm on workers each having a subset of the
data. Distributed SGD may suffer from the effect of stragglers, i.e., slow or
unresponsive workers who cause delays. One solution studied in the literature
is to wait at each iteration for the responses of the fastest workers
before updating the model, where is a fixed parameter. The choice of the
value of presents a trade-off between the runtime (i.e., convergence rate)
of SGD and the error of the model. Towards optimizing the error-runtime
trade-off, we investigate distributed SGD with adaptive . We first design an
adaptive policy for varying that optimizes this trade-off based on an upper
bound on the error as a function of the wall-clock time which we derive. Then,
we propose an algorithm for adaptive distributed SGD that is based on a
statistical heuristic. We implement our algorithm and provide numerical
simulations which confirm our intuition and theoretical analysis.Comment: Accepted to IEEE ICASSP 202