29 research outputs found
Equality in the Matrix Entropy-Power Inequality and Blind Separation of Real and Complex sources
The matrix version of the entropy-power inequality for real or complex
coefficients and variables is proved using a transportation argument that
easily settles the equality case. An application to blind source extraction is
given.Comment: 5 pages, in Proc. 2019 IEEE International Symposium on Information
Theory (ISIT 2019), Paris, France, July 7-12, 201
On Codes for the Noisy Substring Channel
We consider the problem of coding for the substring channel, in which
information strings are observed only through their (multisets of) substrings.
Because of applications to DNA-based data storage, due to DNA sequencing
techniques, interest in this channel has renewed in recent years. In contrast
to existing literature, we consider a noisy channel model, where information is
subject to noise \emph{before} its substrings are sampled, motivated by in-vivo
storage.
We study two separate noise models, substitutions or deletions. In both
cases, we examine families of codes which may be utilized for error-correction
and present combinatorial bounds. Through a generalization of the concept of
repeat-free strings, we show that the added required redundancy due to this
imperfect observation assumption is sublinear, either when the fraction of
errors in the observed substring length is sufficiently small, or when that
length is sufficiently long. This suggests that no asymptotic cost in rate is
incurred by this channel model in these cases.Comment: ISIT 2021 version (including all proofs
Slope and generalization properties of neural networks
Neural networks are very successful tools in for example advanced
classification. From a statistical point of view, fitting a neural network may
be seen as a kind of regression, where we seek a function from the input space
to a space of classification probabilities that follows the "general" shape of
the data, but avoids overfitting by avoiding memorization of individual data
points. In statistics, this can be done by controlling the geometric complexity
of the regression function. We propose to do something similar when fitting
neural networks by controlling the slope of the network.
After defining the slope and discussing some of its theoretical properties,
we go on to show empirically in examples, using ReLU networks, that the
distribution of the slope of a well-trained neural network classifier is
generally independent of the width of the layers in a fully connected network,
and that the mean of the distribution only has a weak dependence on the model
architecture in general. The slope is of similar size throughout the relevant
volume, and varies smoothly. It also behaves as predicted in rescaling
examples. We discuss possible applications of the slope concept, such as using
it as a part of the loss function or stopping criterion during network
training, or ranking data sets in terms of their complexity
Convergence of Smoothed Empirical Measures with Applications to Entropy Estimation
This paper studies convergence of empirical measures smoothed by a Gaussian
kernel. Specifically, consider approximating , for
, by
, where is the empirical measure,
under different statistical distances. The convergence is examined in terms of
the Wasserstein distance, total variation (TV), Kullback-Leibler (KL)
divergence, and -divergence. We show that the approximation error under
the TV distance and 1-Wasserstein distance () converges at rate
in remarkable contrast to a typical
rate for unsmoothed (and ). For the
KL divergence, squared 2-Wasserstein distance (), and
-divergence, the convergence rate is , but only if
achieves finite input-output mutual information across the additive
white Gaussian noise channel. If the latter condition is not met, the rate
changes to for the KL divergence and , while
the -divergence becomes infinite - a curious dichotomy. As a main
application we consider estimating the differential entropy
in the high-dimensional regime. The distribution
is unknown but i.i.d samples from it are available. We first show that
any good estimator of must have sample complexity
that is exponential in . Using the empirical approximation results we then
show that the absolute-error risk of the plug-in estimator converges at the
parametric rate , thus establishing the minimax
rate-optimality of the plug-in. Numerical results that demonstrate a
significant empirical superiority of the plug-in approach to general-purpose
differential entropy estimators are provided.Comment: arXiv admin note: substantial text overlap with arXiv:1810.1158
Analytical calculation formulas for capacities of classical and classical-quantum channels
We derive an analytical calculation formula for the channel capacity of a
classical channel without any iteration while its existing algorithms require
iterations and the number of iteration depends on the required precision level.
Hence, our formula is its first analytical formula without any iteration. We
apply the obtained formula to examples and see how the obtained formula works
in these examples. Then, we extend it to the channel capacity of a
classical-quantum (cq-) channel. Many existing studies proposed algorithms for
a cq-channel and all of them require iterations. Our extended analytical
algorithm have also no iteration and output the exactly optimum values