63,830 research outputs found
Regularized Mutual Information Neural Estimation
With the variational lower bound of mutual information (MI), the estimation
of MI can be understood as an optimization task via stochastic gradient
descent. In this work, we start by showing how Mutual Information Neural
Estimator (MINE) searches for the optimal function that maximizes the
Donsker-Varadhan representation. With our synthetic dataset, we directly
observe the neural network outputs during the optimization to investigate why
MINE succeeds or fails: We discover the drifting phenomenon, where the constant
term of is shifting through the optimization process, and analyze the
instability caused by the interaction between the and the
insufficient batch size. Next, through theoretical and experimental evidence,
we propose a novel lower bound that effectively regularizes the neural network
to alleviate the problems of MINE. We also introduce an averaging strategy that
produces an unbiased estimate by utilizing multiple batches to mitigate the
batch size limitation. Finally, we show that regularization achieves
significant improvements in both discrete and continuous settings.Comment: 18 pages, 15 figur
MINDE: Mutual Information Neural Diffusion Estimation
In this work we present a new method for the estimation of Mutual Information
(MI) between random variables. Our approach is based on an original
interpretation of the Girsanov theorem, which allows us to use score-based
diffusion models to estimate the Kullback Leibler divergence between two
densities as a difference between their score functions. As a by-product, our
method also enables the estimation of the entropy of random variables. Armed
with such building blocks, we present a general recipe to measure MI, which
unfolds in two directions: one uses conditional diffusion process, whereas the
other uses joint diffusion processes that allow simultaneous modelling of two
random variables. Our results, which derive from a thorough experimental
protocol over all the variants of our approach, indicate that our method is
more accurate than the main alternatives from the literature, especially for
challenging distributions. Furthermore, our methods pass MI self-consistency
tests, including data processing and additivity under independence, which
instead are a pain-point of existing methods
CNN-AIDED FACTOR GRAPHS WITH ESTIMATED MUTUAL INFORMATION FEATURES FOR SEIZURE DETECTION
We propose a convolutional neural network (CNN) aided factor graphs assisted by mutual information features estimated by a neural network for seizure detection. Specifically, we use neural mutual information estimation to evaluate the correlation between different electroencephalogram (EEG) channels as features. We then use a 1D-CNN to extract extra features from the EEG signals and use both features to estimate the probability of a seizure event. Finally, learned factor graphs are employed to capture the temporal correlation in the signal. Both sets of features from the neural mutual estimation and the 1D-CNN are used to learn the factor nodes. We show that the proposed method achieves state-of-the-art performance using 6-fold leave-four-patients-out cross-validation
Estimating mutual information and multi--information in large networks
We address the practical problems of estimating the information relations
that characterize large networks. Building on methods developed for analysis of
the neural code, we show that reliable estimates of mutual information can be
obtained with manageable computational effort. The same methods allow
estimation of higher order, multi--information terms. These ideas are
illustrated by analyses of gene expression, financial markets, and consumer
preferences. In each case, information theoretic measures correlate with
independent, intuitive measures of the underlying structures in the system
CRYPTO-MINE: Cryptanalysis via Mutual Information Neural Estimation
The use of Mutual Information (MI) as a measure to evaluate the efficiency of
cryptosystems has an extensive history. However, estimating MI between unknown
random variables in a high-dimensional space is challenging. Recent advances in
machine learning have enabled progress in estimating MI using neural networks.
This work presents a novel application of MI estimation in the field of
cryptography. We propose applying this methodology directly to estimate the MI
between plaintext and ciphertext in a chosen plaintext attack. The leaked
information, if any, from the encryption could potentially be exploited by
adversaries to compromise the computational security of the cryptosystem. We
evaluate the efficiency of our approach by empirically analyzing multiple
encryption schemes and baseline approaches. Furthermore, we extend the analysis
to novel network coding-based cryptosystems that provide individual secrecy and
study the relationship between information leakage and input distribution
- …