Faster Algorithms for Structured Linear and Kernel Support Vector Machines

Quadratic programming is a ubiquitous prototype in convex programming. Many combinatorial optimizations on graphs and machine learning problems can be formulated as quadratic programming; for example, Support Vector Machines (SVMs). Linear and kernel SVMs have been among the most popular models in machine learning over the past three decades, prior to the deep learning era. Generally, a quadratic program has an input size of $\Theta(n^2)$, where $n$ is the number of variables. Assuming the Strong Exponential Time Hypothesis ($\textsf{SETH}$), it is known that no $O(n^{2-o(1)})$ algorithm exists (Backurs, Indyk, and Schmidt, NIPS'17). However, problems such as SVMs usually feature much smaller input sizes: one is given $n$ data points, each of dimension $d$, with $d \ll n$. Furthermore, SVMs are variants with only $O(1)$ linear constraints. This suggests that faster algorithms are feasible, provided the program exhibits certain underlying structures. In this work, we design the first nearly-linear time algorithm for solving quadratic programs whenever the quadratic objective has small treewidth or admits a low-rank factorization, and the number of linear constraints is small. Consequently, we obtain a variety of results for SVMs: * For linear SVM, where the quadratic constraint matrix has treewidth $\tau$, we can solve the corresponding program in time $\widetilde O(n\tau^{(\omega+1)/2}\log(1/\epsilon))$; * For linear SVM, where the quadratic constraint matrix admits a low-rank factorization of rank-$k$, we can solve the corresponding program in time $\widetilde O(nk^{(\omega+1)/2}\log(1/\epsilon))$; * For Gaussian kernel SVM, where the data dimension $d = \Theta(\log n)$ and the squared dataset radius is small, we can solve it in time $O(n^{1+o(1)}\log(1/\epsilon))$. We also prove that when the squared dataset radius is large, then $\Omega(n^{2-o(1)})$ time is required.Comment: New results: almost-linear time algorithm for Gaussian kernel SVM and complementary lower bounds. Abstract shortened to meet arxiv requiremen

Research on Bending Fatigue Properties of Reinforced Macadam Foundation

Existing macadam base structures have poor resistance to bending deformation of pavement. The structural layer is challenged with serious cracking due to the long-term dynamic loads. Despite these issues, however, studies on reinforcement techniques for macadam base structure have been scarce. No reasonable and feasible method on how to improve the bending resistance of flexible base structures (e.g. macadam base structure) and prolong their bending fatigue life has yet been reported. Thus, this study proposed a method for reinforcing macadam base structures with geogrids to strengthen bending performance, anti-fatigue performance and dynamic stability of flexible base structures under different loading levels. The reinforced macadam base structure was investigated through a laboratory test of flexural-tensile strength and a three-point fatigue bending test. A cyclic loading test of reinforced and non-reinforced girder specimens was carried out under five stress levels. The effect of such a new reinforcement method in improving the bending fatigue properties of macadam base structures was analyzed by comparing the yield curves and fatigue lives of reinforced and non-reinforced specimens under different loading levels. Results demonstrate that the non-reinforced macadam base has poor bending resistance. The yield curve and fatigue life of non-reinforced specimens attenuated quickly as it approached the ultimate loading level. Non-reinforced specimens have low-cycle fatigue failures when the loads reach the 0.8 stress level and the ultimate displacement at failure is relatively small, accompanied by obvious failure surface. After geogrids are added, the fatigue life of specimens improved significantly. The yield curve shows no significant attenuation at a high loading level. The reinforced specimens can still maintain high-cycle fatigue failure under the 0.8 stress level and avoid the occurrence of low-cycle fatigue failures. Moreover, reinforced specimens can inhibit transmission of reflection cracks at midspan of cement-stabilized macadam base, delay the propagation of oblique shearing fractures close to the support, and strengthen the dynamic stability and durability of pavement structure with a macadam base. The conclusions of this study provide theoretical references to practical engineering applications of such new reinforcement technology for macadam bases

Discrete Element Simulation of Bending Deformation of Geogrid-Reinforced Macadam Base

The pavement bending deformation resistance of the existing macadam base structure is poor. The geogrid-reinforced macadam base can effectively strengthen the bending resistance of the pavement, but no international consensus has been reached at present over bending failure laws of reinforced macadam base structure. Discrete element models of semi-rigid base pavement structure, macadam base pavement structure, and geogrid-reinforced macadam base pavement structure were built based on MATDEM discrete element simulation program; loading calculation of the three models was conducted by taking their centers as loading positions; and model displacement nephogram, strain nephogram, and effects of different spans on their bending deformation were analyzed to reveal bending failure laws of reinforced macadam base and improvement effect of the geogrid on the anti-bending performance of the macadam structural layer. Finally, bending deformation laws of the three pavement structures and improvement effect of geogrid reinforcement on bending properties of the macadam base structure were established. The results show that under bending deformation of semi-rigid base, the vertical strain at the contract surface between the baseplate and soil base and horizontal strain at midspan position reach the maximum, which can easily lead to fracture and shear failure, and the macadam base layer can effectively isolate the tensile strain transmitted from bottom up. Through their own deformation, grids can transform surface pressure load into frictional resistance at the geogrid/soil interface and partial kinetic energy in the system into their own elastic potential energy to reduce the kinetic energy at the subbase layer. Geogrid reinforcement can improve the nonlinearity of macadam materials, reduce the fluctuation amplitude of the strain curve and displacement curve, lengthen the service life of the macadam base pavement structure, and improve its structural soundness under bending deformation. This study can provide a theoretical reference for numerical simulation of bending failure of geogrid-reinforced macadam base

Streaming Semidefinite Programs: O(n)O(\sqrt{n}) Passes, Small Space and Fast Runtime

We study the problem of solving semidefinite programs (SDP) in the streaming model. Specifically, $m$ constraint matrices and a target matrix $C$, all of size $n\times n$ together with a vector $b\in \mathbb{R}^m$ are streamed to us one-by-one. The goal is to find a matrix $X\in \mathbb{R}^{n\times n}$ such that $\langle C, X\rangle$ is maximized, subject to $\langle A_i, X\rangle=b_i$ for all $i\in [m]$ and $X\succeq 0$. Previous algorithmic studies of SDP primarily focus on \emph{time-efficiency}, and all of them require a prohibitively large $\Omega(mn^2)$ space in order to store \emph{all the constraints}. Such space consumption is necessary for fast algorithms as it is the size of the input. In this work, we design an interior point method (IPM) that uses $\widetilde O(m^2+n^2)$ space, which is strictly sublinear in the regime $n\gg m$. Our algorithm takes $O(\sqrt n\log(1/\epsilon))$ passes, which is standard for IPM. Moreover, when $m$ is much smaller than $n$, our algorithm also matches the time complexity of the state-of-the-art SDP solvers. To achieve such a sublinear space bound, we design a novel sketching method that enables one to compute a spectral approximation to the Hessian matrix in $O(m^2)$ space. To the best of our knowledge, this is the first method that successfully applies sketching technique to improve SDP algorithm in terms of space (also time)

Dynamic Tensor Product Regression

In this work, we initiate the study of \emph{Dynamic Tensor Product Regression}. One has matrices $A_1\in \mathbb{R}^{n_1\times d_1},\ldots,A_q\in \mathbb{R}^{n_q\times d_q}$ and a label vector $b\in \mathbb{R}^{n_1\ldots n_q}$, and the goal is to solve the regression problem with the design matrix $A$ being the tensor product of the matrices $A_1, A_2, \dots, A_q$ i.e. $\min_{x\in \mathbb{R}^{d_1\ldots d_q}}~\|(A_1\otimes \ldots\otimes A_q)x-b\|_2$. At each time step, one matrix $A_i$ receives a sparse change, and the goal is to maintain a sketch of the tensor product $A_1\otimes\ldots \otimes A_q$ so that the regression solution can be updated quickly. Recomputing the solution from scratch for each round is very slow and so it is important to develop algorithms which can quickly update the solution with the new design matrix. Our main result is a dynamic tree data structure where any update to a single matrix can be propagated quickly throughout the tree. We show that our data structure can be used to solve dynamic versions of not only Tensor Product Regression, but also Tensor Product Spline regression (which is a generalization of ridge regression) and for maintaining Low Rank Approximations for the tensor product.Comment: NeurIPS 202

Accelerating Frank-Wolfe Algorithm using Low-Dimensional and Adaptive Data Structures

In this paper, we study the problem of speeding up a type of optimization algorithms called Frank-Wolfe, a conditional gradient method. We develop and employ two novel inner product search data structures, improving the prior fastest algorithm in [Shrivastava, Song and Xu, NeurIPS 2021]. * The first data structure uses low-dimensional random projection to reduce the problem to a lower dimension, then uses efficient inner product data structure. It has preprocessing time $\tilde O(nd^{\omega-1}+dn^{1+o(1)})$ and per iteration cost $\tilde O(d+n^\rho)$ for small constant $\rho$. * The second data structure leverages the recent development in adaptive inner product search data structure that can output estimations to all inner products. It has preprocessing time $\tilde O(nd)$ and per iteration cost $\tilde O(d+n)$. The first algorithm improves the state-of-the-art (with preprocessing time $\tilde O(d^2n^{1+o(1)})$ and per iteration cost $\tilde O(dn^\rho)$) in all cases, while the second one provides an even faster preprocessing time and is suitable when the number of iterations is small

Efficient Algorithm for Solving Hyperbolic Programs

Hyperbolic polynomials is a class of real-roots polynomials that has wide range of applications in theoretical computer science. Each hyperbolic polynomial also induces a hyperbolic cone that is of particular interest in optimization due to its generality, as by choosing the polynomial properly, one can easily recover the classic optimization problems such as linear programming and semidefinite programming. In this work, we develop efficient algorithms for hyperbolic programming, the problem in each one wants to minimize a linear objective, under a system of linear constraints and the solution must be in the hyperbolic cone induced by the hyperbolic polynomial. Our algorithm is an instance of interior point method (IPM) that, instead of following the central path, it follows the central Swath, which is a generalization of central path. To implement the IPM efficiently, we utilize a relaxation of the hyperbolic program to a quadratic program, coupled with the first four moments of the hyperbolic eigenvalues that are crucial to update the optimization direction. We further show that, given an evaluation oracle of the polynomial, our algorithm only requires $O(n^2d^{2.5})$ oracle calls, where $n$ is the number of variables and $d$ is the degree of the polynomial, with extra $O((n+m)^3 d^{0.5})$ arithmetic operations, where $m$ is the number of constraints

Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time

Given a matrix $M\in \mathbb{R}^{m\times n}$, the low rank matrix completion problem asks us to find a rank-$k$ approximation of $M$ as $UV^\top$ for $U\in \mathbb{R}^{m\times k}$ and $V\in \mathbb{R}^{n\times k}$ by only observing a few entries specified by a set of entries $\Omega\subseteq [m]\times [n]$. In particular, we examine an approach that is widely used in practice -- the alternating minimization framework. Jain, Netrapalli and Sanghavi~\cite{jns13} showed that if $M$ has incoherent rows and columns, then alternating minimization provably recovers the matrix $M$ by observing a nearly linear in $n$ number of entries. While the sample complexity has been subsequently improved~\cite{glz17}, alternating minimization steps are required to be computed exactly. This hinders the development of more efficient algorithms and fails to depict the practical implementation of alternating minimization, where the updates are usually performed approximately in favor of efficiency. In this paper, we take a major step towards a more efficient and error-robust alternating minimization framework. To this end, we develop an analytical framework for alternating minimization that can tolerate moderate amount of errors caused by approximate updates. Moreover, our algorithm runs in time $\widetilde O(|\Omega| k)$, which is nearly linear in the time to verify the solution while preserving the sample complexity. This improves upon all prior known alternating minimization approaches which require $\widetilde O(|\Omega| k^2)$ time.Comment: Improve the runtime from $O(mnk)$ to $O|\Omega| k)

Convex Minimization with Integer Minima in O~(n4)\widetilde O(n^4) Time

Given a convex function $f$ on $\mathbb{R}^n$ with an integer minimizer, we show how to find an exact minimizer of $f$ using $O(n^2 \log n)$ calls to a separation oracle and $O(n^4 \log n)$ time. The previous best polynomial time algorithm for this problem given in [Jiang, SODA 2021, JACM 2022] achieves $O(n^2\log\log n/\log n)$ oracle complexity. However, the overall runtime of Jiang's algorithm is at least $\widetilde{\Omega}(n^8)$, due to expensive sub-routines such as the Lenstra-Lenstra-Lov\'asz (LLL) algorithm [Lenstra, Lenstra, Lov\'asz, Math. Ann. 1982] and random walk based cutting plane method [Bertsimas, Vempala, JACM 2004]. Our significant speedup is obtained by a nontrivial combination of a faster version of the LLL algorithm due to [Neumaier, Stehl\'e, ISSAC 2016] that gives similar guarantees, the volumetric center cutting plane method (CPM) by [Vaidya, FOCS 1989] and its fast implementation given in [Jiang, Lee, Song, Wong, STOC 2020]. For the special case of submodular function minimization (SFM), our result implies a strongly polynomial time algorithm for this problem using $O(n^3 \log n)$ calls to an evaluation oracle and $O(n^4 \log n)$ additional arithmetic operations. Both the oracle complexity and the number of arithmetic operations of our more general algorithm are better than the previous best-known runtime algorithms for this specific problem given in [Lee, Sidford, Wong, FOCS 2015] and [Dadush, V\'egh, Zambelli, SODA 2018, MOR 2021].Comment: SODA 202