Diffusion Approximations for Online Principal Component Estimation and Global Convergence

In this paper, we propose to adopt the diffusion approximation tools to study the dynamics of Oja's iteration which is an online stochastic gradient descent method for the principal component analysis. Oja's iteration maintains a running estimate of the true principal component from streaming data and enjoys less temporal and spatial complexities. We show that the Oja's iteration for the top eigenvector generates a continuous-state discrete-time Markov chain over the unit sphere. We characterize the Oja's iteration in three phases using diffusion approximation and weak convergence tools. Our three-phase analysis further provides a finite-sample error bound for the running estimate, which matches the minimax information lower bound for principal component analysis under the additional assumption of bounded samples.Comment: Appeared in NIPS 201

Nonmonotone local minimax methods for finding multiple saddle points

In this paper, by designing a normalized nonmonotone search strategy with the Barzilai--Borwein-type step-size, a novel local minimax method (LMM), which is a globally convergent iterative method, is proposed and analyzed to find multiple (unstable) saddle points of nonconvex functionals in Hilbert spaces. Compared to traditional LMMs with monotone search strategies, this approach, which does not require strict decrease of the objective functional value at each iterative step, is observed to converge faster with less computations. Firstly, based on a normalized iterative scheme coupled with a local peak selection that pulls the iterative point back onto the solution submanifold, by generalizing the Zhang--Hager (ZH) search strategy in the optimization theory to the LMM framework, a kind of normalized ZH-type nonmonotone step-size search strategy is introduced, and then a novel nonmonotone LMM is constructed. Its feasibility and global convergence results are rigorously carried out under the relaxation of the monotonicity for the functional at the iterative sequences. Secondly, in order to speed up the convergence of the nonmonotone LMM, a globally convergent Barzilai--Borwein-type LMM (GBBLMM) is presented by explicitly constructing the Barzilai--Borwein-type step-size as a trial step-size of the normalized ZH-type nonmonotone step-size search strategy in each iteration. Finally, the GBBLMM algorithm is implemented to find multiple unstable solutions of two classes of semilinear elliptic boundary value problems with variational structures: one is the semilinear elliptic equations with the homogeneous Dirichlet boundary condition and another is the linear elliptic equations with semilinear Neumann boundary conditions. Extensive numerical results indicate that our approach is very effective and speeds up the LMMs significantly.Comment: 32 pages, 7 figures; Accepted by Journal of Computational Mathematics on January 3, 202

Minimax methods for finding multiple saddle critical points in Banach spaces and their applications

This dissertation was to study computational theory and methods for ?nding multiple saddle critical points in Banach spaces. Two local minimax methods were developed for this purpose. One was for unconstrained cases and the other was for constrained cases. First, two local minmax characterization of saddle critical points in Banach spaces were established. Based on these two local minmax characterizations, two local minimax algorithms were designed. Their ?ow charts were presented. Then convergence analysis of the algorithms were carried out. Under certain assumptions, a subsequence convergence and a point-to-set convergence were obtained. Furthermore, a relation between the convergence rates of the functional value sequence and corresponding gradient sequence was derived. Techniques to implement the algorithms were discussed. In numerical experiments, those techniques have been successfully implemented to solve for multiple solutions of several quasilinear elliptic boundary value problems and multiple eigenpairs of the well known nonlinear p-Laplacian operator. Numerical solutions were presented by their pro?les for visualization. Several interesting phenomena of the solutions of quasilinear elliptic boundary value problems and the eigenpairs of the p-Laplacian operator have been observed and are open for further investigation. As a generalization of the above results, nonsmooth critical points were considered for locally Lipschitz continuous functionals. A local minmax characterization of nonsmooth saddle critical points was also established. To establish its version in Banach spaces, a new notion, pseudo-generalized-gradient has to be introduced. Based on the characterization, a local minimax algorithm for ?nding multiple nonsmooth saddle critical points was proposed for further study

Adaptive local minimax Galerkin methods for variational problems

In many applications of practical interest, solutions of partial differential equation models arise as critical points of an underlying (energy) functional. If such solutions are saddle points, rather than being maxima or minima, then the theoretical framework is non-standard, and the development of suitable numerical approximation procedures turns out to be highly challenging. In this paper, our aim is to present an iterative discretization methodology for the numerical solution of nonlinear variational problems with multiple (saddle point) solutions. In contrast to traditional numerical approximation schemes, which typically fail in such situations, the key idea of the current work is to employ a simultaneous interplay of a previously developed local minimax approach and adaptive Galerkin discretizations. We thereby derive an adaptive local minimax Galerkin (LMMG) method, which combines the search for saddle point solutions and their approximation in finite-dimensional spaces in a highly effective way. Under certain assumptions, we will prove that the generated sequence of approximate solutions converges to the solution set of the variational problem. This general framework will be applied to the specific context of finite element discretizations of (singularly perturbed) semilinear elliptic boundary value problems, and a series of numerical experiments will be presented