14 research outputs found
Direct Ensemble Estimation of Density Functionals
Estimating density functionals of analog sources is an important problem in
statistical signal processing and information theory. Traditionally, estimating
these quantities requires either making parametric assumptions about the
underlying distributions or using non-parametric density estimation followed by
integration. In this paper we introduce a direct nonparametric approach which
bypasses the need for density estimation by using the error rates of k-NN
classifiers asdata-driven basis functions that can be combined to estimate a
range of density functionals. However, this method is subject to a non-trivial
bias that dramatically slows the rate of convergence in higher dimensions. To
overcome this limitation, we develop an ensemble method for estimating the
value of the basis function which, under some minor constraints on the
smoothness of the underlying distributions, achieves the parametric rate of
convergence regardless of data dimension.Comment: 5 page
Understanding confounding effects in linguistic coordination: an information-theoretic approach
We suggest an information-theoretic approach for measuring stylistic
coordination in dialogues. The proposed measure has a simple predictive
interpretation and can account for various confounding factors through proper
conditioning. We revisit some of the previous studies that reported strong
signatures of stylistic accommodation, and find that a significant part of the
observed coordination can be attributed to a simple confounding effect - length
coordination. Specifically, longer utterances tend to be followed by longer
responses, which gives rise to spurious correlations in the other stylistic
features. We propose a test to distinguish correlations in length due to
contextual factors (topic of conversation, user verbosity, etc.) and
turn-by-turn coordination. We also suggest a test to identify whether stylistic
coordination persists even after accounting for length coordination and
contextual factors
Conditional Mutual Information Neural Estimator
Several recent works in communication systems have proposed to leverage the
power of neural networks in the design of encoders and decoders. In this
approach, these blocks can be tailored to maximize the transmission rate based
on aggregated samples from the channel. Motivated by the fact that, in many
communication schemes, the achievable transmission rate is determined by a
conditional mutual information term, this paper focuses on neural-based
estimators for this information-theoretic quantity. Our results are based on
variational bounds for the KL-divergence and, in contrast to some previous
works, we provide a mathematically rigorous lower bound. However, additional
challenges with respect to the unconditional mutual information emerge due to
the presence of a conditional density function which we address here.Comment: To be presented at ICASSP 202
Squared-loss mutual information via high-dimension coherence matrix estimation
Squared-loss mutual information (SMI) is a surro- gate of Shannon mutual information that is more advantageous for estimation. On the other hand, the coherence matrix of a pair of random vectors, a power-normalized version of the sample cross-covariance matrix, is a well-known second-order statistic found in the core of fundamental signal processing problems, such as canonical correlation analysis (CCA). This paper shows that SMI can be estimated from a pair of independent and identically distributed (i.i.d.) samples as a squared Frobenius norm of a coherence matrix estimated after mapping the data onto some fixed feature space. Moreover, low computation complexity is achieved through the fast Fourier transform (FFT) by exploiting the Toeplitz structure of the involved autocorrelation matrices in that space. The performance of the method is analyzed via computer simulations using Gaussian mixture models.This work is supported by projects TEC2016-76409-C2-1-R (WINTER), Ministerio de Economia y Competividad, Spanish National Research Plan, and 2017 SGR 578 - AGAUR, Catalan Government.Peer ReviewedPostprint (published version
Mutual Information in Frequency and its Application to Measure Cross-Frequency Coupling in Epilepsy
We define a metric, mutual information in frequency (MI-in-frequency), to
detect and quantify the statistical dependence between different frequency
components in the data, referred to as cross-frequency coupling and apply it to
electrophysiological recordings from the brain to infer cross-frequency
coupling. The current metrics used to quantify the cross-frequency coupling in
neuroscience cannot detect if two frequency components in non-Gaussian brain
recordings are statistically independent or not. Our MI-in-frequency metric,
based on Shannon's mutual information between the Cramer's representation of
stochastic processes, overcomes this shortcoming and can detect statistical
dependence in frequency between non-Gaussian signals. We then describe two
data-driven estimators of MI-in-frequency: one based on kernel density
estimation and the other based on the nearest neighbor algorithm and validate
their performance on simulated data. We then use MI-in-frequency to estimate
mutual information between two data streams that are dependent across time,
without making any parametric model assumptions. Finally, we use the MI-in-
frequency metric to investigate the cross-frequency coupling in seizure onset
zone from electrocorticographic recordings during seizures. The inferred
cross-frequency coupling characteristics are essential to optimize the spatial
and spectral parameters of electrical stimulation based treatments of epilepsy.Comment: This paper is accepted for publication in IEEE Transactions on Signal
Processing and contains 15 pages, 9 figures and 1 tabl