The classical Bayesian posterior arises naturally as the unique solution of several different optimization problems, without the necessity of interpreting data as conditional probabilities and then using Bayes' Theorem. For example, the classical Bayesian posterior is the unique posterior that minimizes the loss of Shannon information in combining the prior and the likelihood distributions. These results, direct corollaries of recent results about conflations of probability distributions, reinforce the use of Bayesian posteriors, and may help partially reconcile some of the differences between classical and Bayesian statistics

Dall'Aglio, Marco

Theodore P., Hill

DALL'AGLIO, MARCO

Theodore P. HILL

Archivio della ricerca- LUISS Libera Università Internazionale degli Studi Sociali Guido Carli di Roma

arXiv:1203.0251v1  [math.ST]  1 Mar 2012Bayesian Posteriors Without Bayes’ TheoremT. P. Hill and Marco Dall’ AglioAbstractThe classical Bayesian posterior arises naturally as the unique solution of several dif-ferent optimization problems, without the necessity of interpreting data as conditionalprobabilities and then using Bayes’ Theorem. For example, the classical Bayesianposterior is the unique posterior that minimizes the loss of Shannon information incombining the prior and the likelihood distributions. These results, direct corollar-ies of recent results about conflations of probability distributions, reinforce the use ofBayesian posteriors, and may help partially reconcile some of the differences betweenclassical and Bayesian statistics.1 IntroductionIn statistics, prior belief about the value of an unknown parameter, θ ∈ Θ ⊆ Rn obtainedfrom experiments or other methods, is often expressed as a Borel probability distribution P0on Θ ⊆ Rn called the prior distribution. New evidence or information about the value of θ,based on an independent experiment or survey, is recorded as a likelihood distribution L. Hereand throughout it will be assumed that the likelihood function has finite positive total mass,and that L has been normalized, so that in fact L is also a Borel probability distribution onΘ. Given the prior distribution P0 and the likelihood distribution L, a posterior distributionP1 = P1(P0, L) for θ incorporates the new likelihood information about θ into the informationfrom the prior, thus updating the prior. The posterior distribution P1 is typically viewed asthe conditional distribution of θ given the new likelihood information, often expressed as arandom variable X .The first main goal of this note is to use recent results for conflations of probabilitydistributions [3, 4] to show that the Bayesian posterior is the unique posterior that minimizesthe loss of Shannon information in combining the prior and likelihood distributions. TheBayesian posterior is also the unique posterior that attains the minimax likelihood ratioof the prior and likelihood distributions, and the unique posterior that is a proportionalconsolidation of the prior and likelihood distributions. Thus, the classical Bayesian posteriorappears naturally as the solution of several different optimization problems, without thenecessity of interpreting likelihood as a conditional probability and then invoking BayesTheorem. These results reinforce the use of Bayesian posteriors, and may help partiallyreconcile some of the differences between classical statistics and Bayesian statistics.The second main goal of this note, another direct corollary of recent results for conflationsof probability distributions [4], is to identify the best posterior when the prior and likeli-hood distributions are not weighted equally, such as in cases when the prior distribution is1given more weight than the likelihood distribution. This new weighted posterior, the uniquedistribution that minimizes the loss of weighted Shannon information, coincides with theclassical Bayesian posterior if the prior and likelihood are weighted equally, but in general isdifferent.2 Combining Priors and Likelihoods into PosteriorsThere are many different methods for combining several probability distributions (e.g., see [1,3]), and in particular, for combining the prior distribution P0 and the likelihood distributionL into a single posterior distribution P1 = P1(P0, L). For example, the prior and likelihoodscould simply be averaged, i.e. P1 =P0+L2, or the data underlying the prior and the likelihoodcould be averaged, in which case the posterior P1 would be the distribution ofX0+XL2, whereX0 and XL are independent random variables with distributions P0 and L, respectively.In Bayesian statistics, the likelihood function L is usually interpreted as L(θ) = αP (X |θ), where X is the independent experiment or random variable yielding new informationabout θ, and α is the normalizing constant for L to have mass one (cf. [2]). The Bayesianposterior distribution PB is then calculated using Bayes Theorem: for example, if P0 andL are discrete with probability mass functions (p.m.f.’s) p0 and pL respectively, then PB isdiscrete with p.m.f.pB(θ) =p0(θ)pL(θ)∑θˆ∈Θ p0(θ̂)pL(θ̂);and if P0 and L are absolutely continuous with probability density functions (p.d.f.’s) f0 andfL respectively, then PB is absolutely continuous with p.d.f.fB(θ) =f0(θ)fL(θ)∫Θf0(θ̂)fL(θ̂)dθ̂(provided the denominators are positive and finite).3 Minimizing Loss of Shannon InformationWhen the goal is to consolidate information from a prior distribution and a likelihood dis-tribution into a (posterior) distribution, replacing those two distributions by a single distri-bution will clearly result in some loss of information, however that is defined. Recall thatthe classical Shannon information (also called the self-information or surprisal) associatedwith the event A from a probability distribution P , SP (A), is given by SP (A) = − log2 P (A)(so the smaller the value of P (A), the greater the information or surprise). The numericalvalue of the Shannon information of a given probability is simply the number of binary bitsof information reflected in that probability.Example 3.1. If P is uniformly distributed on (0, 1) and A = (0, 0.25) ∪ (0.5, 0.75), thenSP (A) = − log2(P (A)) = − log2(0.5) = 1, so if X is a random variable with distribution P ,then exactly one binary bit of information is obtained by observing that X ∈ A, in this casethat the value of the second binary digit of X is 0.2Definition 3.2. The combined Shannon Information associated with the event A from theprior distribution P0 and the likelihood distribution L isS{P0,L}(A) = SP0(A) + SL(A) = − log2 P0(A)L(A),and the maximum loss between the Shannon Information of a posterior distribution P1and the combined Shannon information of the prior and likelihood distributions P0 andL, M(P1;P0, L), isM(P1;P0, L) = maxA{S{P0,L}(A)− SP1(A)}= maxA{log2P1(A)P0(A)L(A)}.Note that the definition of combined Shannon information implicitly assumes indepen-dence of the prior and likelihood distributions. Note also that no information is obtained byobserving an event that is certain to occur, so for instance S[P0,L](Θ) = SP1(Θ) = 0. Thisimplies that M(P1;P0, L) is never negative.Definition 3.3. A prior distribution P0 and a likelihood distribution L are compatible if P0and L are both discrete with p.m.f’s p0 and pL satisfying∑θ∈Θ p0(θ)pL(θ) > 0, or are bothabsolutely continuous with p.d.f.’s f0 and fL satisfying 0 <∫Θf0(θ)fL(θ)dθ <∞.Example 3.4. Every two geometric distributions are compatible, every two normal distri-butions are compatible, and every exponential distribution is compatible with every normaldistribution. Distributions with disjoint support, discrete or continuous, are not compatible.Remark. In practice, compatibility is not problematic. Any two distributions may be easilytransformed into two new distributions, arbitrarily close to the original distributions, so thatthe two new distributions are compatible, for instance by convolving each with a U(−ǫ, ǫ)distribution.Theorem 3.5. Let P0 and L be discrete compatible prior and likelihood distributions. Thenthe Bayesian posterior PB is the unique posterior distribution that minimizes the maximumloss of Shannon information from the prior and likelihood distributions, i.e., that minimizesM(P1;P0, L) among all posterior distributions P1. Moreover,M(P1;P0, L) ≥ log2(∑θ∈Θp0(θ)pL(θ))−1 for all posterior distributions P1,and equality is uniquely attained by the Bayesian posterior P1 = PB.The conclusion of Theorem 3.5 follows immediately as a special case of [3, Corollary 4.4];analogous conclusions for the case of compatible absolutely continuous distributions followfrom [3, Theorem 4.5]. For the benefit of the reader, a sketch of the proof of Theorem 3.5similar to that in [4] is included.Sketch of proof. First observe that for an event A, the difference between the combinedShannon information obtained from a prior distribution P0 and a likelihood distribution L,and the Shannon information obtained from the posterior P1, isS{P0,L}(A)− SP1(A) = SP0(A) + SL(A)− SP1(A) = log2P1(A)P0(A)L(A).3Since log2(x) is strictly increasing, the maximum (loss) thus occurs for an event A whereP1(A)P0(A)L(A)is maximized.Next note that the largest loss of Shannon information occurs for small sets A, since fordisjoint sets A and B,P1(A ∪ B)P0(A ∪ B)L(A ∪B)≤P1(A) + P1(B)P0(A)L(A) + P0(B)L(B)≤ max{P1(A)P0(A)L(A),P1(B)P0(B)L(B)},where the inequalities follow from the inequalities (a + b)(c + d) ≥ ac + bd and a+bc+d≤max{ac, bd}for positive numbers a, b, c, d. Thus the problem reduces to finding the proba-bility mass function p that makes the maximum, over all real values θ, of the ratio p(θ)p0(θ)pL(θ)as small as possible. But the minimum over all nonnegative q1, . . . , qn with q1 + · · ·+ qn = 1of the maximum of q1r1, . . . , qnrnoccurs when q1r1= · · · = qnrn(if they are not equal, reducing thenumerator of the largest ratio, and increasing that of the smallest, will make the maximumsmaller). Thus the p that makes the maximum of p(θ)p0(θ)pL(θ)as small as possible is whenp(θ) = cp0(θ)pL(θ), where c is chosen to make p a probability mass function, i.e., to makep(θ) sum to 1. But this is exactly the definition of the Bayesian posterior PB in the discretecase. 4 Minimax Likelihood RatiosIn classical hypotheses testing, a standard technique to decide from which of several knowndistributions given data actually came is to maximize the likelihood ratios, that is, the ratiosof the p.m.f.’s or p.d.f.’s. Analogously, when the objective is to decide how best to consolidatea prior distribution P0 and a likelihood distribution L into a single (posterior) distributionP1 = P1(P0, L), one natural criterion is to choose P1 so as to make the ratios of the likelihoodof observing θ under P1 as close as possible to the likelihood of observing θ under both theprior distribution P0 and the likelihood distribution L. This motivates the following notionof minimax likelihood ratio posterior.Definition 4.1. A discrete probability distribution P ∗ (with p.m.f. p∗) is the minimaxlikelihood ratio (MLR) posterior of a discrete prior distribution P0 with p.m.f. p0 and adiscrete likelihood distribution L with p.m.f. pL ifminp.m.f.’s p{maxθ∈Θp(θ)p0(θ)pL(θ)−minθ∈Θp(θ)p0(θ)pL(θ)}is attained by p = p∗ (where 0/0 := 1).Similarly, an a.c. distribution P ∗ with p.d.f. f ∗ is the MLR posterior of an a.c. priordistribution P0 with p.d.f. f0 and an a.c. likelihood distribution L with p.d.f. fL ifminp.m.f.’s f{ess supθ∈Θf(θ)f0(θ)fL(θ)− ess infθ∈Θf(θ)f0(θ)fL(θ)}is attained by f ∗.4The min-max terms in Definition 4.1 are similar to the min-max criterion for loss ofShannon Information (Theorem 3.5), whereas the others are dual max-min criteria. Just asthe Bayesian posterior minimizes the loss of Shannon information, the Bayesian posterior isalso the MLR posterior of the prior and likelihood distributions.Theorem 4.2. Let P0 and L be compatible discrete or compatible absolutely continuous priorand likelihood distributions, respectively. Then the unique MLR posterior for P0 and L isthe Bayesian posterior distribution PB.Proof. Immediate from [3, Theorem 5.2]. 5 Proportional PosteriorsA criterion similar to likelihood ratios is to require that the posterior distribution P1 reflectthe relative likelihoods of identical individual outcomes under both P0 and L. For example,if the probability that the prior and the (independent) likelihood are both θa is twice thatof the probability both are θb, then P1(θa) should also be twice as large as P1(θb).Definition 5.1. A discrete (posterior) probability distribution P ∗ with p.m.f. p∗ is a pro-portional posterior of a discrete prior distribution P0 with p.m.f. p0 and a compatible discretelikelihood distribution L with p.m.f. pL ifp∗(θa)p∗(θb)=p0(θa)pL(θa)p0(θb)pL(θb)for all θa, θb ∈ Θ.Similarly, a posterior a.c. distribution P ∗ with p.d.f. f ∗ is a proportional posterior of ana.c. prior distribution P0 with p.d.f. f0 and a compatible likelihood distribution L with p.d.f.fL iff ∗(θa)f ∗(θb)=f0(θa)fL(θa)f0(θb)fL(θb)for (Lebesgue) almost all θa, θb ∈ Θ.Theorem 5.2. Let P0 and L be compatible discrete or compatible absolutely continuous priorand likelihood distributions, respectively. Then the Bayesian posterior distribution PB is aproportional consolidation for P0 and L.Proof. Immediate from [3, Theorem 5.5]. 6 Optimal Posteriors for Weighted Prior and Likeli-hood DistributionsDefinition 6.1. Given a prior distribution P0 with weight w0 > 0 and a likelihood distri-bution L with weight wL > 0, the combined weighted Shannon information associated withthe event A, S(P0,w0;L,wL)(A), isS(P0,w0;L,wL)(A) =w0max{w0, wL}SP0(A) +wLmax{w0, wL}SL(A).5This definition ensures that only the relative weights are important, so for instance ifw0 = wL, the combined weighted Shannon information of the prior and likelihood alwayscoincides with the (unweighted) combined Shannon information of the prior and likelihood.Note again that no information is attained by observing any event that is certain to occur, nomatter what the distributions and weights, since SP0(Θ) = SL(Θ) = 0. The next theorem, aspecial case of [4, (8)], identifies the posterior distribution that minimizes the loss of weightedShannon information in the case the prior and likelihood distributions are compatible discretedistributions; the case for compatible absolutely continuous distributions is analogous.Theorem 6.2. Let P0 and L be compatible discrete prior and likelihood distributions withp.m.f.’s p0 and pL and weights w0 > 0 and wL > 0, respectively. Then the unique posteriordistribution that minimizes the maximum loss of Shannon information from the weightedprior and likelihood distributions, i.e., that minimizes, among all posterior distributions P1,maxA{S(P0,w0;L,wL)(A)− SP1(A)},is the posterior distribution Pw1 with p.m.f.pw1 (θ) =(p0(θ))w0max[w0,wL] (pL(θ))wLmax[w0,wL]∑θˆ∈Θ(p0(θ̂))w0max[w0,wL] (pL(θ̂))wLmax[w0,wL].Remark. If both the prior and likelihood distributions are normally distributed, the Bayesianposterior is also a best linear unbiased estimator (BLUE) and a maximum likelihood esti-mator (MLE); e.g. see [3].References[1] Genest, C. and Zidek, J. (1986), Combining probability distributions: a critique andan annotated bibliography, Statist. Sci. 1, 114–148.[2] Gelman, A., Rubin, D. B., Carlin, J., and Stern, H. (2004), Bayesian data analysis,2nd Ed. London: Chapman & Hall.[3] Hill, T. (2011), Conflations of Probability Distributions, Transactions of the AmericanMathematical Society 363(6), pp. 3351–3372.[4] Hill, T. and Miller, J. (2011), How to combine independent data sets for the samequantity, Chaos 21, doi:10.1063/1.3593373.6

Bayesian Posteriors Without Bayes' Theorem

Abstract

Similar works

Full text

Available Versions

Archivio della ricerca- LUISS Libera Università Internazionale degli Studi Sociali Guido Carli di Roma