On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization

Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine learning tasks. However, it has not been analyzed for non-convex minimization and there still remains a gap between the theory and the practice. In this paper, we analyze gradient descent and stochastic gradient descent with extrapolation for finding an approximate first-order stationary point in smooth non-convex optimization problems. Our convergence upper bounds show that the algorithms with extrapolation can be accelerated than without extrapolation

A Nonlinear Projection Neural Network for Solving Interval Quadratic Programming Problems and Its Stability Analysis

This paper presents a nonlinear projection neural network for solving interval quadratic programs subject to box-set constraints in engineering applications. Based on the Saddle point theorem, the equilibrium point of the proposed neural network is proved to be equivalent to the optimal solution of the interval quadratic optimization problems. By employing Lyapunov function approach, the global exponential stability of the proposed neural network is analyzed. Two illustrative examples are provided to show the feasibility and the efficiency of the proposed method in this paper

Fixed-Time Stable Proximal Dynamical System for Solving MVIPs

In this paper, a novel modified proximal dynamical system is proposed to compute the solution of a mixed variational inequality problem (MVIP) within a fixed time, where the time of convergence is finite, and is uniformly bounded for all initial conditions. Under the assumptions of strong monotonicity and Lipschitz continuity, it is shown that a solution of the modified proximal dynamical system exists, is uniquely determined and converges to the unique solution of the associated MVIP within a fixed time. As a special case for solving variational inequality problems, the modified proximal dynamical system reduces to a fixed-time stable projected dynamical system. Furthermore, the fixed-time stability of the modified projected dynamical system continues to hold, even if the assumption of strong monotonicity is relaxed to that of strong pseudomonotonicity. Connections to convex optimization problems are discussed, and commonly studied dynamical systems in the continuous-time optimization literature follow as special limiting cases of the modified proximal dynamical system proposed in this paper. Finally, it is shown that the solution obtained using the forward-Euler discretization of the proposed modified proximal dynamical system converges to an arbitrarily small neighborhood of the solution of the associated MVIP within a fixed number of time steps, independent of the initial conditions. Two numerical examples are presented to substantiate the theoretical convergence guarantees.Comment: 12 pages, 5 figure