NLTGCR: A class of Nonlinear Acceleration Procedures based on Conjugate Residuals

This paper develops a new class of nonlinear acceleration algorithms based on extending conjugate residual-type procedures from linear to nonlinear equations. The main algorithm has strong similarities with Anderson acceleration as well as with inexact Newton methods - depending on which variant is implemented. We prove theoretically and verify experimentally, on a variety of problems from simulation experiments to deep learning applications, that our method is a powerful accelerated iterative algorithm.Comment: Under Revie

Pseudo-factorials, elliptic functions, and continued fractions

This study presents miscellaneous properties of pseudo-factorials, which are numbers whose recurrence relation is a twisted form of that of usual factorials. These numbers are associated with special elliptic functions, most notably, a Dixonian and a Weierstrass function, which parametrize the Fermat cubic curve and are relative to a hexagonal lattice. A continued fraction expansion of the ordinary generating function of pseudo-factorials, first discovered empirically, is established here. This article also provides a characterization of the associated orthogonal polynomials, which appear to form a new family of "elliptic polynomials", as well as various other properties of pseudo-factorials, including a hexagonal lattice sum expression and elementary congruences.Comment: 24 pages; with correction of typos and minor revision. To appear in The Ramanujan Journa

Regularization of Limited Memory Quasi-Newton Methods for Large-Scale Nonconvex Minimization

This paper deals with regularized Newton methods, a flexible class of unconstrained optimization algorithms that is competitive with line search and trust region methods and potentially combines attractive elements of both. The particular focus is on combining regularization with limited memory quasi-Newton methods by exploiting the special structure of limited memory algorithms. Global convergence of regularization methods is shown under mild assumptions and the details of regularized limited memory quasi-Newton updates are discussed including their compact representations. Numerical results using all large-scale test problems from the CUTEst collection indicate that our regularized version of L-BFGS is competitive with state-of-the-art line search and trust-region L-BFGS algorithms and previous attempts at combining L-BFGS with regularization, while potentially outperforming some of them, especially when nonmonotonicity is involved.Comment: 23 pages, 4 figure

Shanks sequence transformations and Anderson acceleration

This paper presents a general framework for Shanks transformations of sequences of elements in a vector space. It is shown that the Minimal Polynomial Extrapolation (MPE), the Modified Minimal Polynomial Extrapolation (MMPE), the Reduced Rank Extrapolation (RRE), the Vector Epsilon Algorithm (VEA), the Topological Epsilon Algorithm (TEA), and Anderson Acceleration (AA), which are standard general techniques designed for accelerating arbitrary sequences and/or solving nonlinear equations, all fall into this framework. Their properties and their connections with quasi-Newton and Broyden methods are studied. The paper then exploits this framework to compare these methods. In the linear case, it is known that AA and GMRES are \u2018essentially\u2019 equivalent in a certain sense while GMRES and RRE are mathematically equivalent. This paper discusses the connection between AA, the RRE, the MPE, and other methods in the nonlinear case

Triangular Recurrences, Generalized Eulerian Numbers, and Related Number Triangles

Many combinatorial and other number triangles are solutions of recurrences of the Graham-Knuth-Patashnik (GKP) type. Such triangles and their defining recurrences are investigated analytically. They are acted on by a transformation group generated by two involutions: a left-right reflection and an upper binomial transformation, acting row-wise. The group also acts on the bivariate exponential generating function (EGF) of the triangle. By the method of characteristics, the EGF of any GKP triangle has an implicit representation in terms of the Gauss hypergeometric function. There are several parametric cases when this EGF can be obtained in closed form. One is when the triangle elements are the generalized Stirling numbers of Hsu and Shiue. Another is when they are generalized Eulerian numbers of a newly defined kind. These numbers are related to the Hsu-Shiue ones by an upper binomial transformation, and can be viewed as coefficients of connection between polynomial bases, in a manner that generalizes the classical Worpitzky identity. Many identities involving these generalized Eulerian numbers and related generalized Narayana numbers are derived, including closed-form evaluations in combinatorially significant cases.Comment: 62 pages, final version, accepted by Advances in Applied Mathematic