We study oblivious sketching for k-sparse linear regression under various
loss functions such as an ℓp norm, or from a broad class of hinge-like
loss functions, which includes the logistic and ReLU losses. We show that for
sparse ℓ2 norm regression, there is a distribution over oblivious
sketches with Θ(klog(d/k)/ε2) rows, which is tight up to a
constant factor. This extends to ℓp loss with an additional additive
O(klog(k/ε)/ε2) term in the upper bound. This
establishes a surprising separation from the related sparse recovery problem,
which is an important special case of sparse regression. For this problem,
under the ℓ2 norm, we observe an upper bound of O(klog(d)/ε+klog(k/ε)/ε2) rows, showing that sparse recovery is
strictly easier to sketch than sparse regression. For sparse regression under
hinge-like loss functions including sparse logistic and sparse ReLU regression,
we give the first known sketching bounds that achieve o(d) rows showing that
O(μ2klog(μnd/ε)/ε2) rows suffice, where μ
is a natural complexity parameter needed to obtain relative error bounds for
these loss functions. We again show that this dimension is tight, up to lower
order terms and the dependence on μ. Finally, we show that similar
sketching bounds can be achieved for LASSO regression, a popular convex
relaxation of sparse regression, where one aims to minimize
∥Ax−b∥22+λ∥x∥1 over x∈Rd. We show that sketching
dimension O(log(d)/(λε)2) suffices and that the dependence
on d and λ is tight.Comment: AISTATS 202