4 research outputs found
Historical development of the BFGS secant method and its characterization properties
The BFGS secant method is the preferred secant method for finite-dimensional unconstrained optimization. The first part of this research consists of recounting the historical development of secant methods in general and the BFGS secant method in particular. Many people believe that the secant method arose from Newton's method using finite difference approximations to the derivative. We compile historical evidence revealing that a special case of the secant method predated Newton's method by more than 3000 years. We trace the evolution of secant methods from 18th-century B.C. Babylonian clay tablets and the Egyptian Rhind Papyrus. Modifications to Newton's method yielding secant methods are discussed and methods we believe influenced and led to the construction of the BFGS secant method are explored.
In the second part of our research, we examine the construction of several rank-two secant update classes that had not received much recognition in the literature. Our study of the underlying mathematical principles and characterizations inherent in the updates classes led to theorems and their proofs concerning secant updates. One class of symmetric rank-two updates that we investigate is the Dennis class. We demonstrate how it can be derived from the general rank-one update formula in a purely algebraic manner not utilizing Powell's method of iterated projections as Dennis did it. The literature abounds with update classes; we show how some are related and show containment when possible. We derive the general formula that could be used to represent all symmetric rank-two secant updates. From this, particular parameter choices yielding well-known updates and update classes are presented. We include two derivations of the Davidon class and prove that it is a maximal class. We detail known characterization properties of the BFGS secant method and describe new characterizations of several secant update classes known to contain the BFGS update. Included is a formal proof of the conjecture made by Schnabel in his 1977 Ph.D. thesis that the BFGS update is in some asymptotic sense the average of the DFP update and the Greenstadt update
Secant Penalized BFGS: A Noise Robust Quasi-Newton Method Via Penalizing The Secant Condition
In this paper, we introduce a new variant of the BFGS method designed to
perform well when gradient measurements are corrupted by noise. We show that by
treating the secant condition with a penalty method approach motivated by
regularized least squares estimation, one can smoothly interpolate between
updating the inverse Hessian approximation with the original BFGS update
formula and not updating the inverse Hessian approximation. Furthermore, we
find the curvature condition is smoothly relaxed as the interpolation moves
towards not updating the inverse Hessian approximation, disappearing entirely
when the inverse Hessian approximation is not updated. These developments allow
us to develop a method we refer to as secant penalized BFGS (SP-BFGS) that
allows one to relax the secant condition based on the amount of noise in the
gradient measurements. SP-BFGS provides a means of incrementally updating the
new inverse Hessian approximation with a controlled amount of bias towards the
previous inverse Hessian approximation, which allows one to replace the
overwriting nature of the original BFGS update with an averaging nature that
resists the destructive effects of noise and can cope with negative curvature
measurements. We discuss the theoretical properties of SP-BFGS, including
convergence when minimizing strongly convex functions in the presence of
uniformly bounded noise. Finally, we present extensive numerical experiments
using over 30 problems from the CUTEst test problem set that demonstrate the
superior performance of SP-BFGS compared to BFGS in the presence of both noisy
function and gradient evaluations.Comment: 38 pages, 3 figures; corrected errors, added numerical experiment