Joint support recovery under high-dimensional scaling: Benefits and perils of `1,∞-regularization

Abstract

Given a collection of r ≥ 2 linear regression problems in p dimensions, suppose that the regression coefficients share partially common supports. This set-up suggests the use of `1/`∞-regularized regression for joint estimation of the p × r matrix of regression coeffi-cients. We analyze the high-dimensional scaling of `1/`∞-regularized quadratic program-ming, considering both consistency rates in `∞-norm, and also how the minimal sample size n required for performing variable selection grows as a function of the model dimension, sparsity, and overlap between the supports. We begin by establishing bounds on the `∞-error as well sufficient conditions for exact variable selection for fixed design matrices, as well as designs drawn randomly from general Gaussian matrices. These results show that the high-dimensional scaling of `1/`∞-regularization is qualitatively similar to that of ordinary `1-regularization. Our second set of results applies to design matrices drawn from standard Gaussian ensembles, for which we provide a sharp set of necessary and sufficient conditions: the `1/`∞-regularized method undergoes a phase transition characterized by the rescaled sam-ple size θ1,∞(n, p, s, α) = n/{(4 − 3α)s log(p − (2−α) s)}. More precisely, for any δ> 0, the probability of successfully recovering both supports converges to 1 for scalings such that θ1, ∞ ≥ 1+ δ, and converges to 0 for scalings for which θ1, ∞ ≤ 1 − δ. An implication of this threshold is that use of `1,∞-regularization yields improved statistical efficiency if the overlap parameter is large enough (α> 2/3), but performs worse than a naive Lasso-based approach for moderate to small overlap (α < 2/3). We illustrate the close agreement between these theoretical predictions, and the actual behavior in simulations.