A Newton-type method and its application

We prove an existence and uniqueness theorem for solving the operator equation F(x)+G(x)=0, where F is a continuous and Gâteaux differentiable operator and the operator G satisfies Lipschitz condition on an open convex subset of a Banach space. As corollaries, a recent theorem of Argyros (2003) and the classical convergence theorem for modified Newton iterates are deduced. We further obtain an existence theorem for a class of nonlinear functional integral equations involving the Urysohn operator

How good are Global Newton methods? Part 2

Newton's method applied to certain problems with a discontinuous derivative operator is shown to be effective. A global Newton method in this setting is exhibited and its computational complexity is estimated. As an application a method is proposed to solve problems of linear inequalities (linear programming, phase 1). Using an example of the Klee-Minty type due to Blair, it was found that the simplex method (used in super-lindo) required over 2,000 iterations, while the method above required an average of 8 iterations (Newton steps) over 15 random starting values.Naval Surface Weapons Center, Dahlgren, VAhttp://archive.org/details/howgoodareglobal00goldO&MN Direct FundingApproved for public release; distribution is unlimited

Non-asymptotic study of quasi-Newton methods with application in large-scale machine learning

Quasi-Newton methods are among popular optimization algorithms for solving convex programs since they serve as a middle ground between first-order methods and second-order Newton-type algorithms. They improve the linear convergence rate of first-order methods and achieve a local superlinear rate, while their computational cost per iteration is O(d²) instead of O(d³) of Newton’s method, where d is the problem dimension. The main idea of quasi-Newton methods is to approximate the step of Newton’s method without computing the objective function Hessian or its inverse at every iteration. To be more specific, quasi-Newton methods aim at approximating the curvature of the objective function by using only first-order information about the function. There are several different approaches for approximating the objective function Hessian and its inverse using first-order information, which leads to different quasi-Newton updates, but perhaps the most popular quasi-Newton algorithms are the Davidon-Fletcher-Powell (DFP) method and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method. As mentioned earlier, a major advantage of quasi-Newton methods is their asymptotic superlinear convergence rate. Although this convergence result is promising and lies between the linear rate of first-order methods and the quadratic rate of Newton’s method, it only holds asymptotically and does not characterize an explicit upper bound on the error of quasi-Newton methods after a finite number of iterations. As a result, the overall complexity of quasi-Newton methods for achieving a target accuracy cannot be explicitly characterized. This shortcoming of the analysis of quasi-Newton methods limits their application in many applications. In this thesis, we aim to investigate both the local and global non-asymptotic superlinear convergence rates of the quasi-Newton methods and leverage these results to study the application of this class of optimization algorithms in large-scale machine learning problems.Electrical and Computer Engineerin

Regularized Nonsmooth Newton Algorithms for Best Approximation

We consider the problem of finding the best approximation point from a polyhedral set, and its applications, in particular to solving large-scale linear programs. The classical projection problem has many various and many applications. We study a regularized nonsmooth Newton type solution method where the Jacobian is singular; and we compare the computational performance to that of the classical projection method of Halperin-Lions-Wittmann-Bauschke (HLWB). We observe empirically that the regularized nonsmooth method significantly outperforms the HLWB method. However, the HLWB has a convergence guarantee while the nonsmooth method is not monotonic and does not guarantee convergence due in part to singularity of the generalized Jacobian. Our application to solving large-scale linear programs uses a parametrized projection problem. This leads to a \emph{stepping stone external path following} algorithm. Other applications are finding triangles from branch and bound methods, and generalized constrained linear least squares. We include scaling methods that improve the efficiency and robustness.Comment: 38 pages, 7 tables, 8 figure

Directional k-Step Newton Methods in n Variables and its Semilocal Convergence Analysis

[EN] The directional k-step Newton methods (k a positive integer) is developed for solving a single nonlinear equation in n variables. Its semilocal convergence analysis is established by using two different approaches (recurrent relations and recurrent functions) under the assumption that the first derivative satisfies a combination of the Lipschitz and the center-Lipschitz continuity conditions instead of only Lipschitz condition. The convergence theorems for the existence and uniqueness of the solution for each of them are established. Numerical examples including nonlinear Hammerstein-type integral equations are worked out and significantly improved results are obtained. It is shown that the second approach based on recurrent functions solves problems failed to be solved by first one using recurrent relations. This demonstrates the efficacy and applicability of these approaches. This work extends the directional one and two-step Newton methods for solving a single nonlinear equation in n variables. Their semilocal convergence analysis using majorizing sequences are studied in Levin (Math Comput 71(237): 251-262, 2002) and Ioannis (Num Algorithms 55(4): 503-528, 2010) under the assumption that the first derivative satisfies the Lipschitz and the combination of the Lipschitz and the center-Lipschitz continuity conditions, respectively. Finally, the computational order of convergence and the computational efficiency of developed method are studied.The authors thank the referees for their fruitful suggestions which have uncovered several weaknesses leading to the improvement in the paper. A. Kumar wishes to thank UGC-CSIR(Grant no. 2061441001), New Delhi and IIT Kharagpur, India, for their financial assistance during this work.Kumar, A.; Gupta, D.; Martínez Molada, E.; Singh, S. (2018). Directional k-Step Newton Methods in n Variables and its Semilocal Convergence Analysis. Mediterranean Journal of Mathematics. 15(2):15-34. https://doi.org/10.1007/s00009-018-1077-0S1534152Levin, Y., Ben-Israel, A.: Directional Newton methods in n variables. Math. Comput. 71(237), 251–262 (2002)Argyros, I.K., Hilout, S.: A convergence analysis for directional two-step Newton methods. Num. Algorithms 55(4), 503–528 (2010)Lukács, G.: The generalized inverse matrix and the surface-surface intersection problem. In: Theory and Practice of Geometric Modeling, pp. 167–185. Springer (1989)Argyros, I.K., Magreñán, Á.A.: Extending the applicability of Gauss–Newton method for convex composite optimization on Riemannian manifolds. Appl. Math. Comput. 249, 453–467 (2014)Argyros, I.K.: A semilocal convergence analysis for directional Newton methods. Math. Comput. 80(273), 327–343 (2011)Ortega, J.M., Rheinboldt, W.C.: Iterative solution of nonlinear equations in several variables. SIAM (2000)Argyros, I.K., Hilout, S.: Weaker conditions for the convergence of Newton’s method. J. Complex. 28(3), 364–387 (2012)Argyros, I.K., Hilout, S.: On an improved convergence analysis of Newton’s method. Appl. Math. Comput. 225, 372–386 (2013)Tapia, R.A.: The Kantorovich theorem for Newton’s method. Am. Math. Mon. 78(4), 389–392 (1971)Argyros, I.K., George, S.: Local convergence for some high convergence order Newton-like methods with frozen derivatives. SeMA J. 70(1), 47–59 (2015)Martínez, E., Singh, S., Hueso, J.L., Gupta, D.K.: Enlarging the convergence domain in local convergence studies for iterative methods in Banach spaces. Appl. Math. Comput. 281, 252–265 (2016)Argyros, I.K., Behl, R. Motsa,S.S.: Ball convergence for a family of quadrature-based methods for solving equations in banach Space. Int. J. Comput. Methods, pp. 1750017 (2016)Parhi, S.K., Gupta, D.K.: Convergence of Stirling’s method under weak differentiability condition. Math. Methods Appl. Sci. 34(2), 168–175 (2011)Prashanth, M., Gupta, D.K.: A continuation method and its convergence for solving nonlinear equations in Banach spaces. Int. J. Comput. Methods 10(04), 1350021 (2013)Parida, P.K., Gupta, D.K.: Recurrence relations for semilocal convergence of a Newton-like method in banach spaces. J. Math. Anal. Appl. 345(1), 350–361 (2008)Argyros, I.K., Hilout, S.: Convergence of Directional Methods under mild differentiability and applications. Appl. Math. Comput. 217(21), 8731–8746 (2011)Amat, S, Bermúdez, C., Hernández-Verón, M.A., Martínez, E.: On an efficient k-step iterative method for nonlinear equations. J. Comput. Appl. Math. 302, 258–271 (2016)Hernández-Verón, M.A., Martínez, E., Teruel, C.: Semilocal convergence of a k-step iterative process and its application for solving a special kind of conservative problems. Num. Algorithms, pp. 1–23Argyros, M., Hernández, I.K., Hilout, S., Romero, N.: Directional Chebyshev-type methods for solving equations. Math. Comput. 84(292), 815–830 (2015)Davis, P.J., Rabinowitz, P.: Methods of numerical integration. Courier Corporation (2007)Cordero, A, Torregrosa, J.R.: Variants of Newton’s method using fifth-order quadrature formulas. Appl. Math. Computation . 190(1), 686–698 (2007)Weerakoon, S., Fernando, T.G.I.: A variant of Newton’s method with accelerated third-order convergence. Appl. Math. Lett. 13(8), 87–93 (2000

Semilocal convergence of a k-step iterative process and its application for solving a special kind of conservative problems

[EN] In this paper, we analyze the semilocal convergence of k-steps Newton's method with frozen first derivative in Banach spaces. The method reaches order of convergence k + 1. By imposing only the assumption that the Fr,chet derivative satisfies the Lipschitz continuity, we define appropriate recurrence relations for obtaining the domains of convergence and uniqueness. We also define the accessibility regions for this iterative process in order to guarantee the semilocal convergence and perform a complete study of their efficiency. Our final aim is to apply these theoretical results to solve a special kind of conservative systems.Hernández-Verón, MA.; Martínez Molada, E.; Teruel-Ferragud, C. (2017). Semilocal convergence of a k-step iterative process and its application for solving a special kind of conservative problems. Numerical Algorithms. 76(2):309-331. https://doi.org/10.1007/s11075-016-0255-zS309331762Amat, S., Busquier, S., Bermúdez, C., Plaza, S.: On two families of high order Newton type methods. Appl. Math. Comput. 25, 2209–2217 (2012)Argyros, I.K., Hilout, S., Tabatabai, M.A.: Mathematical Modelling with Applications in Biosciences and Engineering. Nova Publishers, New York (2011)Argyros, I.K., George, S.: A unified local convergence for Jarratt-type methods in Banach space under weak conditions. Thai. J. Math. 13, 165–176 (2015)Argyros, I.K., Hilout, S.: On the local convergence of fast two-step Newton-like methods for solving nonlinear equations. J. Comput. Appl. Math. 245, 1–9 (2013)Argyros, I.K., Ezquerro, J.A., Gutiérrez, J.M., Hernández, M.A., Hilout, S.: On the semilocal convergence of efficient Chebyshev–Secant-type methods. J. Comput. Appl. Math. 235, 3195–2206 (2011)Cordero, A., Hueso, J.L., Martínez, E., Torregrosa, J.R.: Generating optimal derivative free iterative methods for nonlinear equations by using polynomial interpolation. Math. Comput. Mod. 57, 1950–1956 (2013)Ezquerro, J.A., Grau-Sánchez, M., Hernández, M. A., Noguera, M.: Semilocal convergence of secant-like methods for differentiable and nondifferentiable operators equations. J. Math. Anal. Appl. 398(1), 100–112 (2013)Honorato, G., Plaza, S., Romero, N.: Dynamics of a higher-order family of iterative methods. J. Complexity 27(2), 221–229 (2011)Jerome, J.W., Varga, R.S.: Generalizations of Spline Functions and Applications to Nonlinear Boundary Value and Eigenvalue Problems, Theory and Applications of Spline Functions. Academic Press, New York (1969)Kantorovich, L.V., Akilov, G.P.: Functional analysis Pergamon Press. Oxford (1982)Keller, H.B.: Numerical Methods for Two-Point Boundary-Value Problems. Dover Publications, New York (1992)Na, T.Y.: Computational Methods in Engineering Boundary Value Problems. Academic Press, New York (1979)Ortega, J.M.: The Newton-Kantorovich theorem. Amer. Math. Monthly 75, 658–660 (1968)Ostrowski, A.M.: Solutions of Equations in Euclidean and Banach Spaces. Academic Press, New York (1973)Plaza, S., Romero, N.: Attracting cycles for the relaxed Newton’s method. J. Comput. Appl. Math. 235(10), 3238–3244 (2011)Porter, D., Stirling, D.: Integral Equations: A Practical Treatment, From Spectral Theory to Applications. Cambridge University Press, Cambridge (1990)Traub, J.F.: Iterative Methods for the Solution of Equations. Prentice-Hall. Englewood Cliffs, New Jersey (1964)Argyros, I.K., George, S.: Extending the applicability of Gauss-Newton method for convex composite optimization on Riemannian manifolds using restricted convergence domains. Journal of Nonlinear Functional Analysis 2016 (2016). Article ID 27Xiao, J.Z., Sun, J., Huang, X.: Approximating common fixed points of asymptotically quasi-nonexpansive mappings by a k+1-step iterative scheme with error terms. J. Comput. Appl. Math 233, 2062–2070 (2010)Qin, X., Dehaish, B.A.B., Cho, S.Y.: Viscosity splitting methods for variational inclusion and fixed point problems in Hilbert spaces. J. Nonlinear Sci. Appl. 9, 2789–2797 (2016