9,291 research outputs found
Forward-backward truncated Newton methods for convex composite optimization
This paper proposes two proximal Newton-CG methods for convex nonsmooth
optimization problems in composite form. The algorithms are based on a a
reformulation of the original nonsmooth problem as the unconstrained
minimization of a continuously differentiable function, namely the
forward-backward envelope (FBE). The first algorithm is based on a standard
line search strategy, whereas the second one combines the global efficiency
estimates of the corresponding first-order methods, while achieving fast
asymptotic convergence rates. Furthermore, they are computationally attractive
since each Newton iteration requires the approximate solution of a linear
system of usually small dimension
Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver
We consider a variant of inexact Newton Method, called Newton-MR, in which
the least-squares sub-problems are solved approximately using Minimum Residual
method. By construction, Newton-MR can be readily applied for unconstrained
optimization of a class of non-convex problems known as invex, which subsumes
convexity as a sub-class. For invex optimization, instead of the classical
Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global
convergence can be guaranteed under a weaker notion of joint regularity of
Hessian and gradient. We also obtain Newton-MR's problem-independent local
convergence to the set of minima. We show that fast local/global convergence
can be guaranteed under a novel inexactness condition, which, to our knowledge,
is much weaker than the prior related works. Numerical results demonstrate the
performance of Newton-MR as compared with several other Newton-type
alternatives on a few machine learning problems.Comment: 35 page
The power of sum-of-squares for detecting hidden structures
We study planted problems---finding hidden structures in random noisy
inputs---through the lens of the sum-of-squares semidefinite programming
hierarchy (SoS). This family of powerful semidefinite programs has recently
yielded many new algorithms for planted problems, often achieving the best
known polynomial-time guarantees in terms of accuracy of recovered solutions
and robustness to noise. One theme in recent work is the design of spectral
algorithms which match the guarantees of SoS algorithms for planted problems.
Classical spectral algorithms are often unable to accomplish this: the twist in
these new spectral algorithms is the use of spectral structure of matrices
whose entries are low-degree polynomials of the input variables. We prove that
for a wide class of planted problems, including refuting random constraint
satisfaction problems, tensor and sparse PCA, densest-k-subgraph, community
detection in stochastic block models, planted clique, and others, eigenvalues
of degree-d matrix polynomials are as powerful as SoS semidefinite programs of
roughly degree d. For such problems it is therefore always possible to match
the guarantees of SoS without solving a large semidefinite program. Using
related ideas on SoS algorithms and low-degree matrix polynomials (and inspired
by recent work on SoS and the planted clique problem by Barak et al.), we prove
new nearly-tight SoS lower bounds for the tensor and sparse principal component
analysis problems. Our lower bounds for sparse principal component analysis are
the first to suggest that going beyond existing algorithms for this problem may
require sub-exponential time
Convex recovery from interferometric measurements
This note formulates a deterministic recovery result for vectors from
quadratic measurements of the form for some
left-invertible . Recovery is exact, or stable in the noisy case, when the
couples are chosen as edges of a well-connected graph. One possible way
of obtaining the solution is as a feasible point of a simple semidefinite
program. Furthermore, we show how the proportionality constant in the error
estimate depends on the spectral gap of a data-weighted graph Laplacian. Such
quadratic measurements have found applications in phase retrieval, angular
synchronization, and more recently interferometric waveform inversion
- …