41,986 research outputs found
Efficient Estimation of Mutual Information for Strongly Dependent Variables
We demonstrate that a popular class of nonparametric mutual information (MI)
estimators based on k-nearest-neighbor graphs requires number of samples that
scales exponentially with the true MI. Consequently, accurate estimation of MI
between two strongly dependent variables is possible only for prohibitively
large sample size. This important yet overlooked shortcoming of the existing
estimators is due to their implicit reliance on local uniformity of the
underlying joint distribution. We introduce a new estimator that is robust to
local non-uniformity, works well with limited data, and is able to capture
relationship strengths over many orders of magnitude. We demonstrate the
superior performance of the proposed estimator on both synthetic and real-world
data.Comment: 13 pages, to appear in International Conference on Artificial
Intelligence and Statistics (AISTATS) 201
Quantifying dependencies for sensitivity analysis with multivariate input sample data
We present a novel method for quantifying dependencies in multivariate
datasets, based on estimating the R\'{e}nyi entropy by minimum spanning trees
(MSTs). The length of the MSTs can be used to order pairs of variables from
strongly to weakly dependent, making it a useful tool for sensitivity analysis
with dependent input variables. It is well-suited for cases where the input
distribution is unknown and only a sample of the inputs is available. We
introduce an estimator to quantify dependency based on the MST length, and
investigate its properties with several numerical examples. To reduce the
computational cost of constructing the exact MST for large datasets, we explore
methods to compute approximations to the exact MST, and find the multilevel
approach introduced recently by Zhong et al. (2015) to be the most accurate. We
apply our proposed method to an artificial testcase based on the Ishigami
function, as well as to a real-world testcase involving sediment transport in
the North Sea. The results are consistent with prior knowledge and heuristic
understanding, as well as with variance-based analysis using Sobol indices in
the case where these indices can be computed
Advances in Feature Selection with Mutual Information
The selection of features that are relevant for a prediction or
classification problem is an important problem in many domains involving
high-dimensional data. Selecting features helps fighting the curse of
dimensionality, improving the performances of prediction or classification
methods, and interpreting the application. In a nonlinear context, the mutual
information is widely used as relevance criterion for features and sets of
features. Nevertheless, it suffers from at least three major limitations:
mutual information estimators depend on smoothing parameters, there is no
theoretically justified stopping criterion in the feature selection greedy
procedure, and the estimation itself suffers from the curse of dimensionality.
This chapter shows how to deal with these problems. The two first ones are
addressed by using resampling techniques that provide a statistical basis to
select the estimator parameters and to stop the search procedure. The third one
is addressed by modifying the mutual information criterion into a measure of
how features are complementary (and not only informative) for the problem at
hand
- …