35,114 research outputs found
Communication Lower Bounds for Statistical Estimation Problems via a Distributed Data Processing Inequality
We study the tradeoff between the statistical error and communication cost of
distributed statistical estimation problems in high dimensions. In the
distributed sparse Gaussian mean estimation problem, each of the machines
receives data points from a -dimensional Gaussian distribution with
unknown mean which is promised to be -sparse. The machines
communicate by message passing and aim to estimate the mean . We
provide a tight (up to logarithmic factors) tradeoff between the estimation
error and the number of bits communicated between the machines. This directly
leads to a lower bound for the distributed \textit{sparse linear regression}
problem: to achieve the statistical minimax error, the total communication is
at least , where is the number of observations that
each machine receives and is the ambient dimension. These lower results
improve upon [Sha14,SD'14] by allowing multi-round iterative communication
model. We also give the first optimal simultaneous protocol in the dense case
for mean estimation.
As our main technique, we prove a \textit{distributed data processing
inequality}, as a generalization of usual data processing inequalities, which
might be of independent interest and useful for other problems.Comment: To appear at STOC 2016. Fixed typos in theorem 4.5 and incorporated
reviewers' suggestion
Regularity Properties for Sparse Regression
Statistical and machine learning theory has developed several conditions
ensuring that popular estimators such as the Lasso or the Dantzig selector
perform well in high-dimensional sparse regression, including the restricted
eigenvalue, compatibility, and sensitivity properties. However, some
of the central aspects of these conditions are not well understood. For
instance, it is unknown if these conditions can be checked efficiently on any
given data set. This is problematic, because they are at the core of the theory
of sparse regression.
Here we provide a rigorous proof that these conditions are NP-hard to check.
This shows that the conditions are computationally infeasible to verify, and
raises some questions about their practical applications.
However, by taking an average-case perspective instead of the worst-case view
of NP-hardness, we show that a particular condition, sensitivity, has
certain desirable properties. This condition is weaker and more general than
the others. We show that it holds with high probability in models where the
parent population is well behaved, and that it is robust to certain data
processing steps. These results are desirable, as they provide guidance about
when the condition, and more generally the theory of sparse regression, may be
relevant in the analysis of high-dimensional correlated observational data.Comment: Manuscript shortened and more motivation added. To appear in
Communications in Mathematics and Statistic
- …