181 research outputs found
A quasi-Newton proximal splitting method
A new result in convex analysis on the calculation of proximity operators in
certain scaled norms is derived. We describe efficient implementations of the
proximity calculation for a useful class of functions; the implementations
exploit the piece-wise linear nature of the dual problem. The second part of
the paper applies the previous result to acceleration of convex minimization
problems, and leads to an elegant quasi-Newton method. The optimization method
compares favorably against state-of-the-art alternatives. The algorithm has
extensive applications including signal processing, sparse recovery and machine
learning and classification
On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergence
We introduce a framework for quasi-Newton forward--backward splitting
algorithms (proximal quasi-Newton methods) with a metric induced by diagonal
rank- symmetric positive definite matrices. This special type of
metric allows for a highly efficient evaluation of the proximal mapping. The
key to this efficiency is a general proximal calculus in the new metric. By
using duality, formulas are derived that relate the proximal mapping in a
rank- modified metric to the original metric. We also describe efficient
implementations of the proximity calculation for a large class of functions;
the implementations exploit the piece-wise linear nature of the dual problem.
Then, we apply these results to acceleration of composite convex minimization
problems, which leads to elegant quasi-Newton methods for which we prove
convergence. The algorithm is tested on several numerical examples and compared
to a comprehensive list of alternatives in the literature. Our quasi-Newton
splitting algorithm with the prescribed metric compares favorably against
state-of-the-art. The algorithm has extensive applications including signal
processing, sparse recovery, machine learning and classification to name a few.Comment: arXiv admin note: text overlap with arXiv:1206.115
A dynamical view of nonlinear conjugate gradient methods with applications to FFT-based computational micromechanics
For fast Fourier transform (FFT)-based computational micromechanics, solvers need to be fast, memory-efficient, and independent of tedious parameter calibration. In this work, we investigate the benefits of nonlinear conjugate gradient (CG) methods in the context of FFT-based computational micromechanics. Traditionally, nonlinear CG methods require dedicated line-search procedures to be efficient, rendering them not competitive in the FFT-based context. We contribute to nonlinear CG methods devoid of line searches by exploiting similarities between nonlinear CG methods and accelerated gradient methods. More precisely, by letting the step-size go to zero, we exhibit the Fletcher–Reeves nonlinear CG as a dynamical system with state-dependent nonlinear damping. We show how to implement nonlinear CG methods for FFT-based computational micromechanics, and demonstrate by numerical experiments that the Fletcher–Reeves nonlinear CG represents a competitive, memory-efficient and parameter-choice free solution method for linear and nonlinear homogenization problems, which, in addition, decreases the residual monotonically
Nonmonotone Barzilai-Borwein Gradient Algorithm for -Regularized Nonsmooth Minimization in Compressive Sensing
This paper is devoted to minimizing the sum of a smooth function and a
nonsmooth -regularized term. This problem as a special cases includes
the -regularized convex minimization problem in signal processing,
compressive sensing, machine learning, data mining, etc. However, the
non-differentiability of the -norm causes more challenging especially
in large problems encountered in many practical applications. This paper
proposes, analyzes, and tests a Barzilai-Borwein gradient algorithm. At each
iteration, the generated search direction enjoys descent property and can be
easily derived by minimizing a local approximal quadratic model and
simultaneously taking the favorable structure of the -norm. Moreover, a
nonmonotone line search technique is incorporated to find a suitable stepsize
along this direction. The algorithm is easily performed, where the values of
the objective function and the gradient of the smooth term are required at
per-iteration. Under some conditions, the proposed algorithm is shown to be
globally convergent. The limited experiments by using some nonconvex
unconstrained problems from CUTEr library with additive -regularization
illustrate that the proposed algorithm performs quite well. Extensive
experiments for -regularized least squares problems in compressive
sensing verify that our algorithm compares favorably with several
state-of-the-art algorithms which are specifically designed in recent years.Comment: 20 page
Implementation of an Optimal First-Order Method for Strongly Convex Total Variation Regularization
We present a practical implementation of an optimal first-order method, due
to Nesterov, for large-scale total variation regularization in tomographic
reconstruction, image deblurring, etc. The algorithm applies to -strongly
convex objective functions with -Lipschitz continuous gradient. In the
framework of Nesterov both and are assumed known -- an assumption
that is seldom satisfied in practice. We propose to incorporate mechanisms to
estimate locally sufficient and during the iterations. The mechanisms
also allow for the application to non-strongly convex functions. We discuss the
iteration complexity of several first-order methods, including the proposed
algorithm, and we use a 3D tomography problem to compare the performance of
these methods. The results show that for ill-conditioned problems solved to
high accuracy, the proposed method significantly outperforms state-of-the-art
first-order methods, as also suggested by theoretical results.Comment: 23 pages, 4 figure
- …