69 research outputs found
Information and Entropy
What is information? Is it physical? We argue that in a Bayesian theory the
notion of information must be defined in terms of its effects on the beliefs of
rational agents. Information is whatever constrains rational beliefs and
therefore it is the force that induces us to change our minds. This problem of
updating from a prior to a posterior probability distribution is tackled
through an eliminative induction process that singles out the logarithmic
relative entropy as the unique tool for inference. The resulting method of
Maximum relative Entropy (ME), which is designed for updating from arbitrary
priors given information in the form of arbitrary constraints, includes as
special cases both MaxEnt (which allows arbitrary constraints) and Bayes' rule
(which allows arbitrary priors). Thus, ME unifies the two themes of these
workshops -- the Maximum Entropy and the Bayesian methods -- into a single
general inference scheme that allows us to handle problems that lie beyond the
reach of either of the two methods separately. I conclude with a couple of
simple illustrative examples.Comment: Presented at MaxEnt 2007, the 27th International Workshop on Bayesian
Inference and Maximum Entropy Methods (July 8-13, 2007, Saratoga Springs, New
York, USA
Origins of the Combinatorial Basis of Entropy
The combinatorial basis of entropy, given by Boltzmann, can be written , where is the dimensionless entropy, is the
number of entities and is number of ways in which a given
realization of a system can occur (its statistical weight). This can be
broadened to give generalized combinatorial (or probabilistic) definitions of
entropy and cross-entropy: and , where is the probability of a given
realization, is a convenient transformation function, is a
scaling parameter and an arbitrary constant. If or
satisfy the multinomial weight or distribution, then using
and , and asymptotically
converge to the Shannon and Kullback-Leibler functions. In general, however,
or need not be multinomial, nor may they approach an
asymptotic limit. In such cases, the entropy or cross-entropy function can be
{\it defined} so that its extremization ("MaxEnt'' or "MinXEnt"), subject to
the constraints, gives the ``most probable'' (``MaxProb'') realization of the
system. This gives a probabilistic basis for MaxEnt and MinXEnt, independent of
any information-theoretic justification.
This work examines the origins of the governing distribution ....
(truncated)Comment: MaxEnt07 manuscript, version 4 revise
The marginalization paradox and the formal Bayes' law
It has recently been shown that the marginalization paradox (MP) can be
resolved by interpreting improper inferences as probability limits. The key to
the resolution is that probability limits need not satisfy the formal Bayes'
law, which is used in the MP to deduce an inconsistency. In this paper, I
explore the differences between probability limits and the more familiar
pointwise limits, which do imply the formal Bayes' law, and show how these
differences underlie some key differences in the interpretation of the MP.Comment: Presented at Maxent 2007, Saratoga Springs, NY, July 200
From Information Geometry to Newtonian Dynamics
Newtonian dynamics is derived from prior information codified into an
appropriate statistical model. The basic assumption is that there is an
irreducible uncertainty in the location of particles so that the state of a
particle is defined by a probability distribution. The corresponding
configuration space is a statistical manifold the geometry of which is defined
by the information metric. The trajectory follows from a principle of
inference, the method of Maximum Entropy. No additional "physical" postulates
such as an equation of motion, or an action principle, nor the concepts of
momentum and of phase space, not even the notion of time, need to be
postulated. The resulting entropic dynamics reproduces the Newtonian dynamics
of any number of particles interacting among themselves and with external
fields. Both the mass of the particles and their interactions are explained as
a consequence of the underlying statistical manifold.Comment: Presented at MaxEnt 2007, the 27th International Workshop on Bayesian
Inference and Maximum Entropy Methods (July 8-13, 2007, Saratoga Springs, New
York, USA
Updating Probabilities with Data and Moments
We use the method of Maximum (relative) Entropy to process information in the
form of observed data and moment constraints. The generic "canonical" form of
the posterior distribution for the problem of simultaneous updating with data
and moments is obtained. We discuss the general problem of non-commuting
constraints, when they should be processed sequentially and when
simultaneously. As an illustration, the multinomial example of die tosses is
solved in detail for two superficially similar but actually very different
problems.Comment: Presented at the 27th International Workshop on Bayesian Inference
and Maximum Entropy Methods in Science and Engineering, Saratoga Springs, NY,
July 8-13, 2007. 10 pages, 1 figure V2 has a small typo in the end of the
appendix that was fixed. aj=mj+1 is now aj=m(k-j)+
Information Geometry and Chaos on Negatively Curved Statistical Manifolds
A novel information-geometric approach to chaotic dynamics on curved
statistical manifolds based on Entropic Dynamics (ED) is suggested.
Furthermore, an information-geometric analogue of the Zurek-Paz quantum chaos
criterion is proposed. It is shown that the hyperbolicity of a non-maximally
symmetric 6N-dimensional statistical manifold M_{s} underlying an ED Gaussian
model describing an arbitrary system of 3N non-interacting degrees of freedom
leads to linear information-geometric entropy growth and to exponential
divergence of the Jacobi vector field intensity, quantum and classical features
of chaos respectively.Comment: 7 pages, presented at MaxEnt2007, the 27th International Workshop on
Bayesian Inference and Maximum Entropy Methods, Saratoga (NY, USA)
(July-2007
On Shannon-Jaynes Entropy and Fisher Information
The fundamentals of the Maximum Entropy principle as a rule for assigning and
updating probabilities are revisited. The Shannon-Jaynes relative entropy is
vindicated as the optimal criterion for use with an updating rule. A
constructive rule is justified which assigns the probabilities least sensitive
to coarse-graining. The implications of these developments for interpreting
physics laws as rules of inference upon incomplete information are briefly
discussed.Comment: Presented at the 27th International Workshop on Bayesian Inference
and Maximum Entropy Methods in Science and Engineering, Saratoga Springs, NY,
July 8-13, 200
Lessons about likelihood functions from nuclear physics
Least-squares data analysis is based on the assumption that the normal
(Gaussian) distribution appropriately characterizes the likelihood, that is,
the conditional probability of each measurement d, given a measured quantity y,
p(d | y). On the other hand, there is ample evidence in nuclear physics of
significant disagreements among measurements, which are inconsistent with the
normal distribution, given their stated uncertainties. In this study the
histories of 99 measurements of the lifetimes of five elementary particles are
examined to determine what can be inferred about the distribution of their
values relative to their stated uncertainties. Taken as a whole, the variations
in the data are somewhat larger than their quoted uncertainties would indicate.
These data strongly support using a Student t distribution for the likelihood
function instead of a normal. The most probable value for the order of the t
distribution is 2.6 +/- 0.9. It is shown that analyses based on long-tailed
t-distribution likelihoods gracefully cope with outlying data.Comment: presented at 27th International Workshop on Bayesian Inference and
Maximum Entropy Methods in Science and Engineering (Maxent 2007), 10 pages,
12 figure
- …