20,339 research outputs found
Information Theoretical Estimators Toolbox
We present ITE (information theoretical estimators) a free and open source,
multi-platform, Matlab/Octave toolbox that is capable of estimating many
different variants of entropy, mutual information, divergence, association
measures, cross quantities, and kernels on distributions. Thanks to its highly
modular design, ITE supports additionally (i) the combinations of the
estimation techniques, (ii) the easy construction and embedding of novel
information theoretical estimators, and (iii) their immediate application in
information theoretical optimization problems. ITE also includes a prototype
application in a central problem class of signal processing, independent
subspace analysis and its extensions.Comment: 5 pages; ITE toolbox: https://bitbucket.org/szzoli/ite
Resampling methods for parameter-free and robust feature selection with mutual information
Combining the mutual information criterion with a forward feature selection
strategy offers a good trade-off between optimality of the selected feature
subset and computation time. However, it requires to set the parameter(s) of
the mutual information estimator and to determine when to halt the forward
procedure. These two choices are difficult to make because, as the
dimensionality of the subset increases, the estimation of the mutual
information becomes less and less reliable. This paper proposes to use
resampling methods, a K-fold cross-validation and the permutation test, to
address both issues. The resampling methods bring information about the
variance of the estimator, information which can then be used to automatically
set the parameter and to calculate a threshold to stop the forward procedure.
The procedure is illustrated on a synthetic dataset as well as on real-world
examples
Information Theoretic Structure Learning with Confidence
Information theoretic measures (e.g. the Kullback Liebler divergence and
Shannon mutual information) have been used for exploring possibly nonlinear
multivariate dependencies in high dimension. If these dependencies are assumed
to follow a Markov factor graph model, this exploration process is called
structure discovery. For discrete-valued samples, estimates of the information
divergence over the parametric class of multinomial models lead to structure
discovery methods whose mean squared error achieves parametric convergence
rates as the sample size grows. However, a naive application of this method to
continuous nonparametric multivariate models converges much more slowly. In
this paper we introduce a new method for nonparametric structure discovery that
uses weighted ensemble divergence estimators that achieve parametric
convergence rates and obey an asymptotic central limit theorem that facilitates
hypothesis testing and other types of statistical validation.Comment: 10 pages, 3 figure
Scalable Bayesian nonparametric measures for exploring pairwise dependence via Dirichlet Process Mixtures
In this article we propose novel Bayesian nonparametric methods using
Dirichlet Process Mixture (DPM) models for detecting pairwise dependence
between random variables while accounting for uncertainty in the form of the
underlying distributions. A key criteria is that the procedures should scale to
large data sets. In this regard we find that the formal calculation of the
Bayes factor for a dependent-vs.-independent DPM joint probability measure is
not feasible computationally. To address this we present Bayesian diagnostic
measures for characterising evidence against a "null model" of pairwise
independence. In simulation studies, as well as for a real data analysis, we
show that our approach provides a useful tool for the exploratory nonparametric
Bayesian analysis of large multivariate data sets
On a Nonparametric Notion of Residual and its Applications
Let be a continuous random vector in , . In this paper, we define the notion of a
nonparametric residual of on that is always independent of the
predictor . We study its properties and show that the proposed
notion of residual matches with the usual residual (error) in a multivariate
normal regression model. Given a random vector in
, we use this notion of
residual to show that the conditional independence between and , given
, is equivalent to the mutual independence of the residuals (of
on and on ) and . This result is used
to develop a test for conditional independence. We propose a bootstrap scheme
to approximate the critical value of this test. We compare the proposed test,
which is easily implementable, with some of the existing procedures through a
simulation study.Comment: 19 pages, 2 figure
- …