1,981 research outputs found
Proximal Stochastic Newton-type Gradient Descent Methods for Minimizing Regularized Finite Sums
In this work, we generalized and unified recent two completely different
works of Jascha \cite{sohl2014fast} and Lee \cite{lee2012proximal} respectively
into one by proposing the \textbf{prox}imal s\textbf{to}chastic
\textbf{N}ewton-type gradient (PROXTONE) method for optimizing the sums of two
convex functions: one is the average of a huge number of smooth convex
functions, and the other is a non-smooth convex function. While a set of
recently proposed proximal stochastic gradient methods, include MISO,
Prox-SDCA, Prox-SVRG, and SAG, converge at linear rates, the PROXTONE
incorporates second order information to obtain stronger convergence results,
that it achieves a linear convergence rate not only in the value of the
objective function, but also in the \emph{solution}. The proof is simple and
intuitive, and the results and technique can be served as a initiate for the
research on the proximal stochastic methods that employ second order
information.Comment: arXiv admin note: text overlap with arXiv:1309.2388, arXiv:1403.4699
by other author
Optimization Methods for Inverse Problems
Optimization plays an important role in solving many inverse problems.
Indeed, the task of inversion often either involves or is fully cast as a
solution of an optimization problem. In this light, the mere non-linear,
non-convex, and large-scale nature of many of these inversions gives rise to
some very challenging optimization problems. The inverse problem community has
long been developing various techniques for solving such optimization tasks.
However, other, seemingly disjoint communities, such as that of machine
learning, have developed, almost in parallel, interesting alternative methods
which might have stayed under the radar of the inverse problem community. In
this survey, we aim to change that. In doing so, we first discuss current
state-of-the-art optimization methods widely used in inverse problems. We then
survey recent related advances in addressing similar challenges in problems
faced by the machine learning community, and discuss their potential advantages
for solving inverse problems. By highlighting the similarities among the
optimization challenges faced by the inverse problem and the machine learning
communities, we hope that this survey can serve as a bridge in bringing
together these two communities and encourage cross fertilization of ideas.Comment: 13 page
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
- …