73 research outputs found
Derivative-free separable quadratic modeling and cubic regularization for unconstrained optimization
Funding Information: Open access funding provided by FCT|FCCN (b-on). The first and second authors are funded by national funds through FCT - Fundação para a Ciência e a Tecnologia I.P., under the scope of projects PTDC/MAT-APL/28400/2017, UIDP/MAT/00297/2020, and UIDB/MAT/00297/2020 (Center for Mathematics and Applications). The third author is funded by national funds through the FCT - Fundação para a Ciência e a Tecnologia, I.P., under the scope of the projects CEECIND/02211/2017, UIDP/MAT/00297/2020, and UIDB/MAT/00297/2020 (Center for Mathematics and Applications). Publisher Copyright: © 2023, The Author(s).We present a derivative-free separable quadratic modeling and cubic regularization technique for solving smooth unconstrained minimization problems. The derivative-free approach is mainly concerned with building a quadratic model that could be generated by numerical interpolation or using a minimum Frobenius norm approach, when the number of points available does not allow to build a complete quadratic model. This model plays a key role to generate an approximated gradient vector and Hessian matrix of the objective function at every iteration. We add a specialized cubic regularization strategy to minimize the quadratic model at each iteration, that makes use of separability. We discuss convergence results, including worst case complexity, of the proposed schemes to first-order stationary points. Some preliminary numerical results are presented to illustrate the robustness of the specialized separable cubic algorithm.publishersversionepub_ahead_of_prin
Nonmonotone spectral projected gradient methods on convex sets
Nonmonotone projected gradient techniques are considered for the minimization of differentiable functions on closed convex sets. The classical projected gradient schemes are extended to include a nonmonotone steplength strategy that is based on the Grippo-Lampariello-Lucidi nonmonotone line search. In particular, the nonmonotone strategy is combined with the spectral gradient choice of steplength to accelerate the convergence process. In addition to the classical projected gradient nonlinear path, the feasible spectral projected gradient is used as a search direction to avoid additional trial projections during the one-dimensional search process. Convergence properties and extensive numerical results are presented.1041196121
Special Issue Dedicated To Selected Surveys In Nonlinear Programming
[No abstract available]34337137
Implementation of an Optimal First-Order Method for Strongly Convex Total Variation Regularization
We present a practical implementation of an optimal first-order method, due
to Nesterov, for large-scale total variation regularization in tomographic
reconstruction, image deblurring, etc. The algorithm applies to -strongly
convex objective functions with -Lipschitz continuous gradient. In the
framework of Nesterov both and are assumed known -- an assumption
that is seldom satisfied in practice. We propose to incorporate mechanisms to
estimate locally sufficient and during the iterations. The mechanisms
also allow for the application to non-strongly convex functions. We discuss the
iteration complexity of several first-order methods, including the proposed
algorithm, and we use a 3D tomography problem to compare the performance of
these methods. The results show that for ill-conditioned problems solved to
high accuracy, the proposed method significantly outperforms state-of-the-art
first-order methods, as also suggested by theoretical results.Comment: 23 pages, 4 figure
Nonmonotone Barzilai-Borwein Gradient Algorithm for -Regularized Nonsmooth Minimization in Compressive Sensing
This paper is devoted to minimizing the sum of a smooth function and a
nonsmooth -regularized term. This problem as a special cases includes
the -regularized convex minimization problem in signal processing,
compressive sensing, machine learning, data mining, etc. However, the
non-differentiability of the -norm causes more challenging especially
in large problems encountered in many practical applications. This paper
proposes, analyzes, and tests a Barzilai-Borwein gradient algorithm. At each
iteration, the generated search direction enjoys descent property and can be
easily derived by minimizing a local approximal quadratic model and
simultaneously taking the favorable structure of the -norm. Moreover, a
nonmonotone line search technique is incorporated to find a suitable stepsize
along this direction. The algorithm is easily performed, where the values of
the objective function and the gradient of the smooth term are required at
per-iteration. Under some conditions, the proposed algorithm is shown to be
globally convergent. The limited experiments by using some nonconvex
unconstrained problems from CUTEr library with additive -regularization
illustrate that the proposed algorithm performs quite well. Extensive
experiments for -regularized least squares problems in compressive
sensing verify that our algorithm compares favorably with several
state-of-the-art algorithms which are specifically designed in recent years.Comment: 20 page
Quasi-Newton-Based Preconditioning and Damped Quasi-Newton Schemes for Nonlinear Conjugate Gradient Methods
In this paper, we deal with matrix-free preconditioners for Nonlinear Conjugate Gradient (NCG) methods. In particular, we review proposals based on quasi-Newton updates, and either satisfying the secant equation or a secant-like equation at some of the previous iterates. Conditions are given proving that, in some sense, the proposed preconditioners also approximate the inverse of the Hessian matrix. In particular, the structure of the preconditioners depends both on low-rank updates along with some specific parameters. The low-rank updates are obtained as by-product of NCG iterations. Moreover, we consider the possibility to embed damped techniques within a class of preconditioners based on quasi-Newton updates. Damped methods have proved to be effective to enhance the performance of quasi-Newton updates, in those cases where the Wolfe linesearch conditions are hardly fulfilled. The purpose is to extend the idea behind damped methods also to improve NCG schemes, following a novel line of research in the literature. The results, which summarize an extended numerical experience using large-scale CUTEst problems, is reported, showing that these approaches can considerably improve the performance of NCG methods
Convergence properties of the Barzilai and Borwein gradient method
In a recent paper, Barzilai and Borwein presented a new choice of steplength for the gradient method. Their choice does not guarantee descent in the objective function and greatly speeds up the convergence of the method. We derive an interesting relationship between any gradient method and the shifted power method. This relationship allows us to establish the convergence of the Barzilai and Borwein method when applied to the problem of minimizing any strictly convex quadratic function (Barzilai and Borwein considered only 2-dimensional problems). Our point of view also allows us to explain the remarkable improvement obtained by using this new choice of steplength.
For the two eigenvalues case we present some very interesting convergence rate results. We show that our Q and R-rate of convergence analysis is sharp and we compare it with the Barzilai and Borwein analysis.
We derive the preconditioned Barzilai and Borwein method and present preliminary numerical results indicating that it is an effective method, as compared to the preconditioned Conjugate Gradient method, for the numerical solution of some special symmetric positive definite linear systems that arise in the numerical solution of Partial Differential Equations
Separable Cubic Modeling And A Trust-region Strategy For Unconstrained Minimization With Impact In Global Optimization
A separable cubic model, for smooth unconstrained minimization, is proposed and evaluated. The cubic model uses some novel secant-type choices for the parameters in the cubic terms. A suitable hard-case-free trust-region strategy that takes advantage of the separable cubic modeling is also presented. For the convergence analysis of our specialized trust region strategy we present as a general framework a model (Formula presented.)-order trust region algorithm with variable metric and we prove its convergence to (Formula presented.)-stationary points. Some preliminary numerical examples are also presented to illustrate the tendency of the specialized trust region algorithm, when combined with our cubic modeling, to escape from local minimizers
- …