1,120 research outputs found
Ghost Penalties in Nonconvex Constrained Optimization: Diminishing Stepsizes and Iteration Complexity
We consider nonconvex constrained optimization problems and propose a new
approach to the convergence analysis based on penalty functions. We make use of
classical penalty functions in an unconventional way, in that penalty functions
only enter in the theoretical analysis of convergence while the algorithm
itself is penalty-free. Based on this idea, we are able to establish several
new results, including the first general analysis for diminishing stepsize
methods in nonconvex, constrained optimization, showing convergence to
generalized stationary points, and a complexity study for SQP-type algorithms.Comment: To appear on Mathematics of Operations Researc
Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver
We consider a variant of inexact Newton Method, called Newton-MR, in which
the least-squares sub-problems are solved approximately using Minimum Residual
method. By construction, Newton-MR can be readily applied for unconstrained
optimization of a class of non-convex problems known as invex, which subsumes
convexity as a sub-class. For invex optimization, instead of the classical
Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global
convergence can be guaranteed under a weaker notion of joint regularity of
Hessian and gradient. We also obtain Newton-MR's problem-independent local
convergence to the set of minima. We show that fast local/global convergence
can be guaranteed under a novel inexactness condition, which, to our knowledge,
is much weaker than the prior related works. Numerical results demonstrate the
performance of Newton-MR as compared with several other Newton-type
alternatives on a few machine learning problems.Comment: 35 page
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
Bethe Projections for Non-Local Inference
Many inference problems in structured prediction are naturally solved by
augmenting a tractable dependency structure with complex, non-local auxiliary
objectives. This includes the mean field family of variational inference
algorithms, soft- or hard-constrained inference using Lagrangian relaxation or
linear programming, collective graphical models, and forms of semi-supervised
learning such as posterior regularization. We present a method to
discriminatively learn broad families of inference objectives, capturing
powerful non-local statistics of the latent variables, while maintaining
tractable and provably fast inference using non-Euclidean projected gradient
descent with a distance-generating function given by the Bethe entropy. We
demonstrate the performance and flexibility of our method by (1) extracting
structured citations from research papers by learning soft global constraints,
(2) achieving state-of-the-art results on a widely-used handwriting recognition
task using a novel learned non-convex inference procedure, and (3) providing a
fast and highly scalable algorithm for the challenging problem of inference in
a collective graphical model applied to bird migration.Comment: minor bug fix to appendix. appeared in UAI 201
- …