How to project onto extended second order cones

The extended second order cones were introduced by S. Z. N\'emeth and G. Zhang in [S. Z. N\'emeth and G. Zhang. Extended Lorentz cones and variational inequalities on cylinders. J. Optim. Theory Appl., 168(3):756-768, 2016] for solving mixed complementarity problems and variational inequalities on cylinders. R. Sznajder in [R. Sznajder. The Lyapunov rank of extended second order cones. Journal of Global Optimization, 66(3):585-593, 2016] determined the automorphism groups and the Lyapunov or bilinearity ranks of these cones. S. Z. N\'emeth and G. Zhang in [S.Z. N\'emeth and G. Zhang. Positive operators of Extended Lorentz cones. arXiv:1608.07455v2, 2016] found both necessary conditions and sufficient conditions for a linear operator to be a positive operator of an extended second order cone. This note will give formulas for projecting onto the extended second order cones. In the most general case the formula will depend on a piecewise linear equation for one real variable which will be solved by using numerical methods

On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization

Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine learning tasks. However, it has not been analyzed for non-convex minimization and there still remains a gap between the theory and the practice. In this paper, we analyze gradient descent and stochastic gradient descent with extrapolation for finding an approximate first-order stationary point in smooth non-convex optimization problems. Our convergence upper bounds show that the algorithms with extrapolation can be accelerated than without extrapolation