In this paper, we introduce a new variant of the BFGS method designed to
perform well when gradient measurements are corrupted by noise. We show that by
treating the secant condition with a penalty method approach motivated by
regularized least squares estimation, one can smoothly interpolate between
updating the inverse Hessian approximation with the original BFGS update
formula and not updating the inverse Hessian approximation. Furthermore, we
find the curvature condition is smoothly relaxed as the interpolation moves
towards not updating the inverse Hessian approximation, disappearing entirely
when the inverse Hessian approximation is not updated. These developments allow
us to develop a method we refer to as secant penalized BFGS (SP-BFGS) that
allows one to relax the secant condition based on the amount of noise in the
gradient measurements. SP-BFGS provides a means of incrementally updating the
new inverse Hessian approximation with a controlled amount of bias towards the
previous inverse Hessian approximation, which allows one to replace the
overwriting nature of the original BFGS update with an averaging nature that
resists the destructive effects of noise and can cope with negative curvature
measurements. We discuss the theoretical properties of SP-BFGS, including
convergence when minimizing strongly convex functions in the presence of
uniformly bounded noise. Finally, we present extensive numerical experiments
using over 30 problems from the CUTEst test problem set that demonstrate the
superior performance of SP-BFGS compared to BFGS in the presence of both noisy
function and gradient evaluations.Comment: 38 pages, 3 figures; corrected errors, added numerical experiment