491 research outputs found
Large Scale Constrained Linear Regression Revisited: Faster Algorithms via Preconditioning
In this paper, we revisit the large-scale constrained linear regression
problem and propose faster methods based on some recent developments in
sketching and optimization. Our algorithms combine (accelerated) mini-batch SGD
with a new method called two-step preconditioning to achieve an approximate
solution with a time complexity lower than that of the state-of-the-art
techniques for the low precision case. Our idea can also be extended to the
high precision case, which gives an alternative implementation to the Iterative
Hessian Sketch (IHS) method with significantly improved time complexity.
Experiments on benchmark and synthetic datasets suggest that our methods indeed
outperform existing ones considerably in both the low and high precision cases.Comment: Appear in AAAI-1
Fast Objective & Duality Gap Convergence for Nonconvex-Strongly-Concave Min-Max Problems
This paper focuses on stochastic methods for solving smooth non-convex
strongly-concave min-max problems, which have received increasing attention due
to their potential applications in deep learning (e.g., deep AUC maximization,
distributionally robust optimization). However, most of the existing algorithms
are slow in practice, and their analysis revolves around the convergence to a
nearly stationary point. We consider leveraging the Polyak-\L ojasiewicz (PL)
condition to design faster stochastic algorithms with stronger convergence
guarantee. Although PL condition has been utilized for designing many
stochastic minimization algorithms, their applications for non-convex min-max
optimization remain rare. In this paper, we propose and analyze a generic
framework of proximal epoch-based method with many well-known stochastic
updates embeddable. Fast convergence is established in terms of both {\bf the
primal objective gap and the duality gap}. Compared with existing studies, (i)
our analysis is based on a novel Lyapunov function consisting of the primal
objective gap and the duality gap of a regularized function, and (ii) the
results are more comprehensive with improved rates that have better dependence
on the condition number under different assumptions. We also conduct deep and
non-deep learning experiments to verify the effectiveness of our methods
- …