2 research outputs found
Phase Transitions for the Uniform Distribution in the PML Problem and its Bethe Approximation
The pattern maximum likelihood (PML) estimate, introduced by Orlitsky et al.,
is an estimate of the multiset of probabilities in an unknown probability
distribution , the estimate being obtained from i.i.d. samples
drawn from . The PML estimate involves solving a difficult
optimization problem over the set of all probability mass functions (pmfs) of
finite support. In this paper, we describe an interesting phase transition
phenomenon in the PML estimate: at a certain sharp threshold, the uniform
distribution goes from being a local maximum to being a local minimum for the
optimization problem in the estimate. We go on to consider the question of
whether a similar phase transition phenomenon also exists in the Bethe
approximation of the PML estimate, the latter being an approximation method
with origins in statistical physics. We show that the answer to this question
is a qualified "Yes". Our analysis involves the computation of the mean and
variance of the th entry, , in a random
non-negative integer matrix with row and column sums all equal to ,
drawn according to a distribution that assigns to a probability
proportional to
Profile Entropy: A Fundamental Measure for the Learnability and Compressibility of Discrete Distributions
The profile of a sample is the multiset of its symbol frequencies. We show
that for samples of discrete distributions, profile entropy is a fundamental
measure unifying the concepts of estimation, inference, and compression.
Specifically, profile entropy a) determines the speed of estimating the
distribution relative to the best natural estimator; b) characterizes the rate
of inferring all symmetric properties compared with the best estimator over any
label-invariant distribution collection; c) serves as the limit of profile
compression, for which we derive optimal near-linear-time block and sequential
algorithms. To further our understanding of profile entropy, we investigate its
attributes, provide algorithms for approximating its value, and determine its
magnitude for numerous structural distribution families.Comment: 56 page