Sharp thresholds for high-dimensional and noisy recovery of sparsity

The problem of consistently estimating the sparsity pattern of a vector \betastar \in \real^\mdim based on observations contaminated by noise arises in various contexts, including subset selection in regression, structure estimation in graphical models, sparse approximation, and signal denoising. We analyze the behavior of $\ell_1$-constrained quadratic programming (QP), also referred to as the Lasso, for recovering the sparsity pattern. Our main result is to establish a sharp relation between the problem dimension \mdim, the number \spindex of non-zero elements in \betastar, and the number of observations \numobs that are required for reliable recovery. For a broad class of Gaussian ensembles satisfying mutual incoherence conditions, we establish existence and compute explicit values of thresholds \ThreshLow and \ThreshUp with the following properties: for any $\epsilon > 0$, if \numobs > 2 (\ThreshUp + \epsilon) \log (\mdim - \spindex) + \spindex + 1, then the Lasso succeeds in recovering the sparsity pattern with probability converging to one for large problems, whereas for \numobs < 2 (\ThreshLow - \epsilon) \log (\mdim - \spindex) + \spindex + 1, then the probability of successful recovery converges to zero. For the special case of the uniform Gaussian ensemble, we show that \ThreshLow = \ThreshUp = 1, so that the threshold is sharp and exactly determined.Comment: Appeared as Technical Report 708, Department of Statistics, UC Berkele

On the measurement of ecological novelty: scale-eating pupfish are separated by 168 my from other scale-eating fishes.

The colonization of new adaptive zones is widely recognized as one of the hallmarks of adaptive radiation. However, the adoption of novel resources during this process is rarely distinguished from phenotypic change because morphology is a common proxy for ecology. How can we quantify ecological novelty independent of phenotype? Our study is split into two parts: we first document a remarkable example of ecological novelty, scale-eating (lepidophagy), within a rapidly-evolving adaptive radiation of Cyprinodon pupfishes on San Salvador Island, Bahamas. This specialized predatory niche is known in several other fish groups, but is not found elsewhere among the 1,500 species of atherinomorphs. Second, we quantify this ecological novelty by measuring the time-calibrated phylogenetic distance in years to the most closely-related species with convergent ecology. We find that scale-eating pupfish are separated by 168 million years of evolution from the nearest scale-eating fish. We apply this approach to a variety of examples and highlight the frequent decoupling of ecological novelty from phenotypic divergence. We observe that novel ecology is not always tightly correlated with rates of phenotypic or species diversification, particularly within recent adaptive radiations, necessitating the use of additional measures of ecological novelty independent of phenotype

Restricted strong convexity and weighted matrix completion: Optimal bounds with noise

We consider the matrix completion problem under a form of row/column weighted entrywise sampling, including the case of uniform entrywise sampling as a special case. We analyze the associated random observation operator, and prove that with high probability, it satisfies a form of restricted strong convexity with respect to weighted Frobenius norm. Using this property, we obtain as corollaries a number of error bounds on matrix completion in the weighted Frobenius norm under noisy sampling and for both exact and near low-rank matrices. Our results are based on measures of the "spikiness" and "low-rankness" of matrices that are less restrictive than the incoherence conditions imposed in previous work. Our technique involves an $M$-estimator that includes controls on both the rank and spikiness of the solution, and we establish non-asymptotic error bounds in weighted Frobenius norm for recovering matrices lying with $\ell_q$-"balls" of bounded spikiness. Using information-theoretic methods, we show that no algorithm can achieve better estimates (up to a logarithmic factor) over these same sets, showing that our conditions on matrices and associated rates are essentially optimal