Relationship between the inexact Newton method and the continuous analogy of Newton's method

In this paper we propose two new strategies to determine the forcing terms that allow one to improve the efficiency and robustness of the inexact Newton method. The choices are based on the relationship between the inexact Newton method and the continuous analogy of Newton's method. With the new forcing terms, the inexact Newton method is locally $Q$-superlinearly and quadratically convergent. Numerical results are presented to support the effectiveness of the new forcing terms

Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver

We consider a variant of inexact Newton Method, called Newton-MR, in which the least-squares sub-problems are solved approximately using Minimum Residual method. By construction, Newton-MR can be readily applied for unconstrained optimization of a class of non-convex problems known as invex, which subsumes convexity as a sub-class. For invex optimization, instead of the classical Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global convergence can be guaranteed under a weaker notion of joint regularity of Hessian and gradient. We also obtain Newton-MR's problem-independent local convergence to the set of minima. We show that fast local/global convergence can be guaranteed under a novel inexactness condition, which, to our knowledge, is much weaker than the prior related works. Numerical results demonstrate the performance of Newton-MR as compared with several other Newton-type alternatives on a few machine learning problems.Comment: 35 page

An inexact Newton method for systems arising from the finite element method

In this paper, we introduce an efficient and robust technique for approximating the Jacobian matrix for a nonlinear system of algebraic equations which arises from the finite element discretization of a system of nonlinear partial differential equations. It is demonstrated that when an iterative solver, such as preconditioned GMRES, is used to solve the linear systems of equations that result from the application of Newton's method, this approach is generally more efficient than using matrix-free techniques: the price paid being the extra memory requirement for storing the sparse Jacobian. The advantages of this approach over attempting to calculate the Jacobian exactly or of using other approximations are also discussed. A numerical example is included which is based upon the solution of a 2-d compressible viscous flow problem