6,521 research outputs found
Invariance entropy, quasi-stationary measures and control sets
For control systems in discrete time, this paper discusses measure-theoretic
invariance entropy for a subset Q of the state space with respect to a
quasi-stationary measure obtained by endowing the control range with a
probability measure. The main results show that this entropy is invariant under
measurable transformations and that it is already determined by certain subsets
of Q which are characterized by controllability properties.Comment: 30 page
Local Intrinsic Dimensional Entropy
Most entropy measures depend on the spread of the probability distribution
over the sample space X, and the maximum entropy achievable scales
proportionately with the sample space cardinality |X|. For a finite |X|, this
yields robust entropy measures which satisfy many important properties, such as
invariance to bijections, while the same is not true for continuous spaces
(where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same
cardinality (from Cantor's correspondence argument), cardinality-dependent
entropy measures cannot encode the data dimensionality. In this work, we
question the role of cardinality and distribution spread in defining entropy
measures for continuous spaces, which can undergo multiple rounds of
transformations and distortions, e.g., in neural networks. We find that the
average value of the local intrinsic dimension of a distribution, denoted as
ID-Entropy, can serve as a robust entropy measure for continuous spaces, while
capturing the data dimensionality. We find that ID-Entropy satisfies many
desirable properties and can be extended to conditional entropy, joint entropy
and mutual-information variants. ID-Entropy also yields new information
bottleneck principles and also links to causality. In the context of deep
learning, for feedforward architectures, we show, theoretically and
empirically, that the ID-Entropy of a hidden layer directly controls the
generalization gap for both classifiers and auto-encoders, when the target
function is Lipschitz continuous. Our work primarily shows that, for continuous
spaces, taking a structural rather than a statistical approach yields entropy
measures which preserve intrinsic data dimensionality, while being relevant for
studying various architectures.Comment: Proceedings of the AAAI Conference on Artificial Intelligence 202
Fundamental Limitations of Disturbance Attenuation in the Presence of Side Information
In this paper, we study fundamental limitations of disturbance attenuation of feedback systems, under the assumption that the controller has a finite horizon preview of the disturbance. In contrast with prior work, we extend Bode's integral equation for the case where the preview is made available to the controller via a general, finite capacity, communication system. Under asymptotic stationarity assumptions, our results show that the new fundamental limitation differs from Bode's only by a constant, which quantifies the information rate through the communication system. In the absence of asymptotic stationarity, we derive a universal lower bound which uses Shannon's entropy rate as a measure of performance. By means of a case-study, we show that our main bounds may be achieved
Full Resolution Image Compression with Recurrent Neural Networks
This paper presents a set of full-resolution lossy image compression methods
based on neural networks. Each of the architectures we describe can provide
variable compression rates during deployment without requiring retraining of
the network: each network need only be trained once. All of our architectures
consist of a recurrent neural network (RNN)-based encoder and decoder, a
binarizer, and a neural network for entropy coding. We compare RNN types (LSTM,
associative LSTM) and introduce a new hybrid of GRU and ResNet. We also study
"one-shot" versus additive reconstruction architectures and introduce a new
scaled-additive framework. We compare to previous work, showing improvements of
4.3%-8.8% AUC (area under the rate-distortion curve), depending on the
perceptual metric used. As far as we know, this is the first neural network
architecture that is able to outperform JPEG at image compression across most
bitrates on the rate-distortion curve on the Kodak dataset images, with and
without the aid of entropy coding.Comment: Updated with content for CVPR and removed supplemental material to an
external link for size limitation
A Note on the Shannon Entropy of Short Sequences
For source sequences of length L symbols we proposed to use a more realistic
value to the usual benchmark of number of code letters by source letters. Our
idea is based on a quantifier of information fluctuation of a source, F(U),
which corresponds to the second central moment of the random variable that
measures the information content of a source symbol. An alternative
interpretation of typical sequences is additionally provided through this
approach.Comment: 3 figure
- …