56,111 research outputs found
GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
For distributed computing environment, we consider the empirical risk
minimization problem and propose a distributed and communication-efficient
Newton-type optimization method. At every iteration, each worker locally finds
an Approximate NewTon (ANT) direction, which is sent to the main driver. The
main driver, then, averages all the ANT directions received from workers to
form a {\it Globally Improved ANT} (GIANT) direction. GIANT is highly
communication efficient and naturally exploits the trade-offs between local
computations and global communications in that more local computations result
in fewer overall rounds of communications. Theoretically, we show that GIANT
enjoys an improved convergence rate as compared with first-order methods and
existing distributed Newton-type methods. Further, and in sharp contrast with
many existing distributed Newton-type methods, as well as popular first-order
methods, a highly advantageous practical feature of GIANT is that it only
involves one tuning parameter. We conduct large-scale experiments on a computer
cluster and, empirically, demonstrate the superior performance of GIANT.Comment: Fixed some typos. Improved writin
Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure
The numerical solution of partial differential equations on high-dimensional
domains gives rise to computationally challenging linear systems. When using
standard discretization techniques, the size of the linear system grows
exponentially with the number of dimensions, making the use of classic
iterative solvers infeasible. During the last few years, low-rank tensor
approaches have been developed that allow to mitigate this curse of
dimensionality by exploiting the underlying structure of the linear operator.
In this work, we focus on tensors represented in the Tucker and tensor train
formats. We propose two preconditioned gradient methods on the corresponding
low-rank tensor manifolds: A Riemannian version of the preconditioned
Richardson method as well as an approximate Newton scheme based on the
Riemannian Hessian. For the latter, considerable attention is given to the
efficient solution of the resulting Newton equation. In numerical experiments,
we compare the efficiency of our Riemannian algorithms with other established
tensor-based approaches such as a truncated preconditioned Richardson method
and the alternating linear scheme. The results show that our approximate
Riemannian Newton scheme is significantly faster in cases when the application
of the linear operator is expensive.Comment: 24 pages, 8 figure
Fast, Accurate Second Order Methods for Network Optimization
Dual descent methods are commonly used to solve network flow optimization
problems, since their implementation can be distributed over the network. These
algorithms, however, often exhibit slow convergence rates. Approximate Newton
methods which compute descent directions locally have been proposed as
alternatives to accelerate the convergence rates of conventional dual descent.
The effectiveness of these methods, is limited by the accuracy of such
approximations. In this paper, we propose an efficient and accurate distributed
second order method for network flow problems. The proposed approach utilizes
the sparsity pattern of the dual Hessian to approximate the the Newton
direction using a novel distributed solver for symmetric diagonally dominant
linear equations. Our solver is based on a distributed implementation of a
recent parallel solver of Spielman and Peng (2014). We analyze the properties
of the proposed algorithm and show that, similar to conventional Newton
methods, superlinear convergence within a neighbor- hood of the optimal value
is attained. We finally demonstrate the effectiveness of the approach in a set
of experiments on randomly generated networks.Comment: arXiv admin note: text overlap with arXiv:1502.0315
- …