Fast, Provably convergent IRLS Algorithm for p-norm Linear Regression

Linear regression in $\ell_p$-norm is a canonical optimization problem that arises in several applications, including sparse recovery, semi-supervised learning, and signal processing. Generic convex optimization algorithms for solving $\ell_p$-regression are slow in practice. Iteratively Reweighted Least Squares (IRLS) is an easy to implement family of algorithms for solving these problems that has been studied for over 50 years. However, these algorithms often diverge for p > 3, and since the work of Osborne (1985), it has been an open problem whether there is an IRLS algorithm that is guaranteed to converge rapidly for p > 3. We propose p-IRLS, the first IRLS algorithm that provably converges geometrically for any $p \in [2,\infty).$ Our algorithm is simple to implement and is guaranteed to find a $(1+\varepsilon)$-approximate solution in $O(p^{3.5} m^{\frac{p-2}{2(p-1)}} \log \frac{m}{\varepsilon}) \le O_p(\sqrt{m} \log \frac{m}{\varepsilon} )$ iterations. Our experiments demonstrate that it performs even better than our theoretical bounds, beats the standard Matlab/CVX implementation for solving these problems by 10--50x, and is the fastest among available implementations in the high-accuracy regime.Comment: Code for this work is available at https://github.com/utoronto-theory/pIRL

Faster p-norm minimizing flows, via smoothed q-norm problems

We present faster high-accuracy algorithms for computing $\ell_p$-norm minimizing flows. On a graph with $m$ edges, our algorithm can compute a $(1+1/\text{poly}(m))$-approximate unweighted $\ell_p$-norm minimizing flow with $pm^{1+\frac{1}{p-1}+o(1)}$ operations, for any $p \ge 2,$ giving the best bound for all $p\gtrsim 5.24.$ Combined with the algorithm from the work of Adil et al. (SODA '19), we can now compute such flows for any $2\le p\le m^{o(1)}$ in time at most $O(m^{1.24}).$ In comparison, the previous best running time was $\Omega(m^{1.33})$ for large constant $p.$ For $p\sim\delta^{-1}\log m,$ our algorithm computes a $(1+\delta)$-approximate maximum flow on undirected graphs using $m^{1+o(1)}\delta^{-1}$ operations, matching the current best bound, albeit only for unit-capacity graphs. We also give an algorithm for solving general $\ell_{p}$-norm regression problems for large $p.$ Our algorithm makes $pm^{\frac{1}{3}+o(1)}\log^2(1/\varepsilon)$ calls to a linear solver. This gives the first high-accuracy algorithm for computing weighted $\ell_{p}$-norm minimizing flows that runs in time $o(m^{1.5})$ for some $p=m^{\Omega(1)}.$ Our key technical contribution is to show that smoothed $\ell_p$-norm problems introduced by Adil et al., are interreducible for different values of $p.$ No such reduction is known for standard $\ell_p$-norm problems.Comment: ACM-SIAM Symposium on Discrete Algorithms (SODA 2020

Iteratively Reweighted Least Squares for Basis Pursuit with Global Linear Convergence Rate

The recovery of sparse data is at the core of many applications in machine learning and signal processing. While such problems can be tackled using $\ell_1$-regularization as in the LASSO estimator and in the Basis Pursuit approach, specialized algorithms are typically required to solve the corresponding high-dimensional non-smooth optimization for large instances. Iteratively Reweighted Least Squares (IRLS) is a widely used algorithm for this purpose due its excellent numerical performance. However, while existing theory is able to guarantee convergence of this algorithm to the minimizer, it does not provide a global convergence rate. In this paper, we prove that a variant of IRLS converges with a global linear rate to a sparse solution, i.e., with a linear error decrease occurring immediately from any initialization, if the measurements fulfill the usual null space property assumption. We support our theory by numerical experiments showing that our linear rate captures the correct dimension dependence. We anticipate that our theoretical findings will lead to new insights for many other use cases of the IRLS algorithm, such as in low-rank matrix recovery.Comment: 26 pages, 3 figure

Acceleration with a Ball Optimization Oracle

Consider an oracle which takes a point $x$ and returns the minimizer of a convex function $f$ in an $\ell_2$ ball of radius $r$ around $x$. It is straightforward to show that roughly $r^{-1}\log\frac{1}{\epsilon}$ calls to the oracle suffice to find an $\epsilon$-approximate minimizer of $f$ in an $\ell_2$ unit ball. Perhaps surprisingly, this is not optimal: we design an accelerated algorithm which attains an $\epsilon$-approximate minimizer with roughly $r^{-2/3} \log \frac{1}{\epsilon}$ oracle queries, and give a matching lower bound. Further, we implement ball optimization oracles for functions with locally stable Hessians using a variant of Newton's method. The resulting algorithm applies to a number of problems of practical and theoretical import, improving upon previous results for logistic and $\ell_\infty$ regression and achieving guarantees comparable to the state-of-the-art for $\ell_p$ regression.Comment: 37 page

Almost-linear-time Weighted ℓp\ell_p-norm Solvers in Slightly Dense Graphs via Sparsification

We give almost-linear-time algorithms for constructing sparsifiers with $n\ poly(\log n)$edges that approximately preserve weighted$(\ell^{2}_2 + \ell^{p}_p)$flow or voltage objectives on graphs. For flow objectives, this is the first sparsifier construction for such mixed objectives beyond unit$\ell_p$weights, and is based on expander decompositions. For voltage objectives, we give the first sparsifier construction for these objectives, which we build using graph spanners and leverage score sampling. Together with the iterative refinement framework of [Adil et al, SODA 2019], and a new multiplicative-weights based constant-approximation algorithm for mixed-objective flows or voltages, we show how to find$(1+2^{-\text{poly}(\log n)})$approximations for weighted$\ell_p$-norm minimizing flows or voltages in$p(m^{1+o(1)} + n^{4/3 + o(1)})$time for$p=\omega(1),$which is almost-linear for graphs that are slightly dense ($m \ge n^{4/3 + o(1)}$)