17,774 research outputs found
Estimating Renyi Entropy of Discrete Distributions
It was recently shown that estimating the Shannon entropy of a
discrete -symbol distribution requires samples,
a number that grows near-linearly in the support size. In many applications
can be replaced by the more general R\'enyi entropy of order
, . We determine the number of samples needed to
estimate for all , showing that
requires a super-linear, roughly samples, noninteger
requires a near-linear samples, but, perhaps surprisingly, integer
requires only samples. Furthermore,
developing on a recently established connection between polynomial
approximation and estimation of additive functions of the form , we reduce the sample complexity for noninteger values of by a
factor of compared to the empirical estimator. The estimators
achieving these bounds are simple and run in time linear in the number of
samples. Our lower bounds provide explicit constructions of distributions with
different R\'enyi entropies that are hard to distinguish
Universal geometric approach to uncertainty, entropy and information
It is shown that for any ensemble, whether classical or quantum, continuous
or discrete, there is only one measure of the "volume" of the ensemble that is
compatible with several basic geometric postulates. This volume measure is thus
a preferred and universal choice for characterising the inherent spread,
dispersion, localisation, etc, of the ensemble. Remarkably, this unique
"ensemble volume" is a simple function of the ensemble entropy, and hence
provides a new geometric characterisation of the latter quantity. Applications
include unified, volume-based derivations of the Holevo and Shannon bounds in
quantum and classical information theory; a precise geometric interpretation of
thermodynamic entropy for equilibrium ensembles; a geometric derivation of
semi-classical uncertainty relations; a new means for defining classical and
quantum localization for arbitrary evolution processes; a geometric
interpretation of relative entropy; and a new proposed definition for the
spot-size of an optical beam. Advantages of the ensemble volume over other
measures of localization (root-mean-square deviation, Renyi entropies, and
inverse participation ratio) are discussed.Comment: Latex, 38 pages + 2 figures; p(\alpha)->1/|T| in Eq. (72) [Eq. (A10)
of published version
On observability of Renyi's entropy
Despite recent claims we argue that Renyi's entropy is an observable
quantity. It is shown that, contrary to popular belief, the reported domain of
instability for Renyi entropies has zero measure (Bhattacharyya measure). In
addition, we show the instabilities can be easily emended by introducing a
coarse graining into an actual measurement. We also clear up doubts regarding
the observability of Renyi's entropy in (multi--)fractal systems and in systems
with absolutely continuous PDF's.Comment: 18 pages, 1 EPS figure, REVTeX, minor changes, accepted to Phys. Rev.
- …