26,414 research outputs found
Infinite Shannon entropy
Even if a probability distribution is properly normalizable, its associated
Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze
conditions under which this phenomenon can occur. Roughly speaking, this
happens when arbitrarily small amounts of probability are dispersed into an
infinite number of states; we shall quantify this observation and make it
precise. We develop several particularly simple, elementary, and useful bounds,
and also provide some asymptotic estimates, leading to necessary and sufficient
conditions for the occurrence of infinite Shannon entropy. We go to some effort
to keep technical computations as simple and conceptually clear as possible. In
particular, we shall see that large entropies cannot be localized in state
space; large entropies can only be supported on an exponentially large number
of states. We are for the time being interested in single-channel Shannon
entropy in the information theoretic sense, not entropy in a stochastic field
theory or QFT defined over some configuration space, on the grounds that this
simple problem is a necessary precursor to understanding infinite entropy in a
field theoretic context.Comment: 13 pages; V2: 4 references adde
Estimating Entropy of Data Streams Using Compressed Counting
The Shannon entropy is a widely used summary statistic, for example, network
traffic measurement, anomaly detection, neural computations, spike trains, etc.
This study focuses on estimating Shannon entropy of data streams. It is known
that Shannon entropy can be approximated by Reenyi entropy or Tsallis entropy,
which are both functions of the p-th frequency moments and approach Shannon
entropy as p->1.
Compressed Counting (CC) is a new method for approximating the p-th frequency
moments of data streams. Our contributions include:
1) We prove that Renyi entropy is (much) better than Tsallis entropy for
approximating Shannon entropy.
2) We propose the optimal quantile estimator for CC, which considerably
improves the previous estimators.
3) Our experiments demonstrate that CC is indeed highly effective
approximating the moments and entropies. We also demonstrate the crucial
importance of utilizing the variance-bias trade-off
- …