The Intrinsic Bayes Factor (IBF) has been recently introduced by Berger and Pericchi (1996) for
automatic model selection and hypothesis testing in a Bayesian framework. A major result is the
existence, in hypothesis testing problems, of an Intrinsic Proper Prior (JPP) that can be obtained
from the IBF in an automatic way. In this article we describe the IBF and compute the IPP in a
simple example. It is the hope that the present article will help in making Bayesian methods more
widely used for Testing Hypothesis

Fiteni, I.

Pericchi, L. R.

Presa, E.

Name not available

The intrinsic bayes factor described by an example

Universidad Carlos III de Madrid e-Archivo

THE INTRINSIC BAVES FACTOR DESCRIBED BV AN EXAMPLE L. R. Pericchi, I. Fiteni and E. Presa 96-44 Universidad Carlos III de Madrid en 0::: w 0.. « 0.. Working Paper 96-44 Statistics and Econometrics Series 15 July 1996 Departamento de Estadrstica y Econometrfa Universidad Carlos III de Madrid Calle Madrid, 126 28903 Getafe (Spain) Fax (341) 624-9849 THE INTRINSIC BAYES FACTOR DESCRIBED BY AN EXAMPLE L.R. Pericchi, 1. Fiteni and E. Presa* Abstract _____________________________ _ The Intrinsic Bayes Factor (IBF) has been recently introduced by Berger and Pericchi (1996) for automatic model selection and hypothesis testing in a Bayesian framework. A major result is the existence, in hypothesis testing problems, of an Intrinsic Proper Prior (JPP) that can be obtained from the IBF in an automatic way. In this article we describe the IBF and compute the IPP in a simple example. It is the hope that the present article will help in making Bayesian methods more widely used for Testing Hypothesis. Key Words Bayes Factor; Intrinsic Proper Prior. ·Pericchi, Departamento de Matematicas, Universidad Sim6n BoHvar, Caracas; Fiteni, Departamento de Estadfstica y Econometrfa, Universidad Carlos III de Madrid; Presa, Departamento de EconomCa, Universidad Carlos III de Madrid. 1 Introduction For ease of exposition, asume that the data x is a random sample from one of the simplest (and still useful) model: the exponential model, with likelihood, 1.1 Estimation. Suppose first that the problem is one of the estimation. If we do not have or do not wish to use subjective prior information, there are several automatic methods to assign a default prior for (). The most widely used methods seems to be the Jeffreys' rule and the Berger and Bernardo (1992) reference prior algorithm. In this case both methods coincide and yield, calling J( ()) the Fisher's information, or equivalently where c is an unspecified arbitrary positive constant. 1 () (1) Equivalently 7rN (log()) = c in the Real line. Note that 7rN (()) is improper, that is, it integrates infinity in 0 < () < 00. This does not prevent us from calculating the posterior density for (), 7r(()I.;r.), which is proper (if n 2: 1) and does not depend on the arbitrary constant c. To see this, note that application of Bayes Rule, yields ()n-1 e-92: X i r(n) (2: Xi)" (2) Thus, 7r( ()I.;r.) is a Gamma distribution with parameter nand 2:i=l Xi. This is clearly proper and does not involve the arbitrary constant c, which cancels out. As the sample size grows, the 2 posterior distribution above is closely approximated by a Normal distribution with the posterior mean and variance of () , namely 1/x and 1/(nx2 ) respectively. On the other hand, the Maximum Likelihood inference based on the MLE (Maximum Likeli-hood Estimator) {) = ~ yields approximately that, {) '" N ((), i(;r.t l ), where i(;r.) is the observed Fisher Information evaluated at {). In this case it gives, (3) For moderate to large samples, in view of the Central Limit Theorem, estimation infer-ences based on (2) and (3) are numerically quite similar. Still the interpretation of (2) is more satisfiying, in our opinion. However, it may be argued that the differences between automatic Bayes and Likelihood inference, is more academic than practical, for point estimation in regular low dimensional likelihoods and for not too small sample sizes, as often encountered in prac-tice. This numerical similarity is typical in one dimensional estimation problem with regular likelihoods and reference priors. 1.2 Hypothesis Testing Suppose now that we are told that there is reason to test the hypothesis, AIo : () = (}o vs How the Bayesian analysis is going to proceed? Formally, there is no doubt about it. Define the Bayes Factor of AIl vs Mo as, where, J f (;r.I(})1r ((})d() J(ifj(}o) mn;r.) = J f(;r.I(})1r((})d(}, is the marginal or predictive of the observations under Ml . (4) Then after the assessment of Po = P(lvlo) and PI = P(MI ), po + PI - 1, for example Po = PI = 1/2 for a default analysis, it follows that (5) 3 and P(M1 1J:.) = 1 - P(Mol~). Equation (5) appears to solve the testing problem in a most satisfying manner. It gives the posterior probability of the alternative models, based on the relative adequacy of predicting the data actually observed, that is the Bayes Factor (4). Furthermore to perform predictions of future observations we are not forced to decide for either one model or the other, but we may keep both under consideration weighting individual model predicitions by their posterior probabilities. However, in the quest for automatic default methods, that we can actually be compared with frequentist testing on equal footing, (4) is a formal sot'ution to the problem not a definite one. To see this assume the automatic choice (l),r.N(O) ~. Then, which depends upon the arbitrary constant c. Thus, automatic improper choice of r. N (0), leaves the Bayes Factor undetermined. On the other hand, there is a sense in which an automatic or default choice in Testing is drastically different from that in Estimation. It can be forcefully argued in a Testing scenario, that the fact that the null model Mo is seriously considered gives a definite piece of information, and now 00 is a distinguished point. Therefore it is needed a general automatic method to assign a proper default prior r.(0) under the alternative model, in view that the null model !vIa: 0 = 00 has been definetely suggested. The surprising fact is that up to now, there was no general method for assessing automatic proper priors for testing, equivalent to the Jeffrey's Rule or the Berger-Bernardo algorithm that assess improper priors for estimation. In words, although there is a formally flawless Bayesian methods for testing, it is not attainable, unless substantial prior information exist. This fact may explain a paradox encountered in the statistical practice. Bayesian methods are much more developed for estimation problems than for testing. But as explained previously, for estimation in regular univariate likelihoods, Bayesian and Likelihood methods tipically rapidly converge numerically, although not in interpretation. On the other hand in testing problems, Bayesian and frequientist methods differ dramatically, and this difference tipically grows with the sample size, see for example Berger and Sellke (1987). And it is frequentist methods for testing which seems to be at fault. Among other things frequentist measure of evidence for models have a difficulty in incorporating the well accepted scientific principle of "Ockham's Razon", i.e. the notion that if two models predicts the data at hand approximately equally well, the simpler model is to be preferred. In fact, for large sample sizes and fixed levels of type one error, the simpler hypothesis is typically rejected in practice, see for example Allenby (1990). To 4 solve this paradox in practice, and to make cohere the statistical measures of evidence with well established scientific principles, an strategic aim is to develop and disseminate automatic general Bayesian methods for testing hypothesis. In Section 2, we describe the automatic and general Bayesian method for testing hypothesis and selecting models put forward by Berger and Pericchi (1996), the Intrinsic Bayes Factor (IBF). A key result of the IBF theory, is the existence of a Intrinsic Proper Prior (IPP), for which constructive equations exist. For the simple testing problem described in the introduction, we compute the IBF and find the IPP, which is argued to be appropiate to the problem. In the last Section we advance some conclusions. It is the hope of the present article, to help to disseminate in practice, default Bayesian methods for hypothesis testing. 2 The Intrinsic Bayes Factor for the Exponential Model. Assume that Xl, X2, •• • , Xn is a sample from the Exponential distribution, as in the Introduction and suppose the framework and notation as in Section (1.2). Hence the Bayes Factor based on 7rN(O) is, (6) From (6) it is concluded, as in the Introduction that B~ is undetermined since it depends on the arbitrary constant c. A solution to this problem is to use a subset at the observations, say !!l.(l) to make the prior for () proper, and perform the discrimination with the remaining observations z.( -I). In this example it follows that it is enough to take training samples of size 1 to make the prior proper. Thus, z..(l) = Xl > 0 and x( -I) is the original data set taking away the data Xl. For such a training sample, (7) Thus, 7r(Olx/) is exponential with parameter XI, wich is obviously proper. 5 Using 7r(Olx/) to define a proper Bayes Factor leads to (using Bayes Rule), foOO f(x(-l)IO)7r(Olx/)dO f (x( -1)100 ) mf(~) f(x/IOo) N N f(~IOo) mf(x/) = BlOB01 (x/). Note that in (8), the arbitrary constant c, has cancel out. (8) Notice that the more complex model has been placed in the numerator in the Bayes Factor, which is not the usual practice. For explanation of this see Berger and Pericchi (1996) and below. The Bayes Factor, B lO (l), is well defined, but it depends on the arbitrary choice of training sample Xl. To eliminate such a dependence and to increase stability, Berger and Pericchi (1996) propose to average all the posible BlO(l). Two averages are put forward: a) The Aritmetic IBF (AIBF), is defined as (9) Computation gives, BA! [ r (n) 1 [00 2:n -0 xl - - x/e 0 I 10 - -00 ",n_ Xi (",n .)n n . e L...._l L...i=l X, /=1 (10) b) The Geometric IBF (GIBF), is defined as, (11) 6 Notice that Bfl ::::; Bfl, since the Geometric mean is less or equal than the Aritmetic mean, so that Bfl will favor more the null hypothesis than Bfl. For a justification of the GIBF see Smith (1995) and Berger and Pericchi (1994). This justification is based on a decision theoretic argument, without assuming that one of the models, Mo or M1 , is necessarely the sampling model. This is what Bernardo and Smith (1993) call the "Open Model Perspective". To see this, consider in a decision framework the Kullback-Leibler loss funtion predicting the observed data, under the true sampling model mT(x). and choose Ml iff the expectation is grater than O. For improper priors this can not be computed since the marginals are not proper. Take a training sample of size s in order to make all the marginals proper. The problem is of course that we do not know the true sampling model mT. But as the sampling size grows this can be approximated by where For Minimal Training Samples this turns out to be the Geometric IBF. In the present paper we concentrate mainly in explorating and justifying the AIBF, because they generate proper Intrinsic priors, see below. However the Geometric IBF has also an important role to play. Appart from the justification in the Open Model Perspective, they are automatically coherent across models in the sense that considering a further model M 2 , which is not the the case for the Arithmetic IBF which needs an adjustment, Berger and Pericchi (1996). Unfortunately, the Geometric IBF generates Intrinsic Measures that do not integrate up to one, although they do not tend to be far off, see below. c) The Expected AIBF. (BfoAI). For very small samples the arithmetic mean of the AIBF's may have a large variability. On the other hand for very large samples the computation of the AIBF may be computationally expensive. An attractive alternative is then to substitute, the average of correction factor by its expectation under the larger model: 7 Note that quite generally, the law of large numbers gives that, both when Ml or Mo are the sampling model, provided the limit exists. Coming back to our example we find that Now since B is unknown, we estimate it by the M.L.E. estimator B = ~, and we find, (12) (13) Notice that for Bfo(Xl), the expectation does not exist for all B. This is why the more complex model is placed above. d) The Expected Geometric IBF. Similarly the Expected IBF is defined as BfoGI = Bfoexp [E~l (ZogBf::t(xt))] In the present example computation yields EGI N Bo ( Bo) BlO = BlOOexp ~(1) - B ' where ~ is the digamma function. Anagously, i3EG1 is obtained replacing above B by B. 8 2.1 Intrinsic Priors The Intrinsic Priors are defined as those measures that gives approximately (as n grows) the same answer as the IBF's. Assuming as in the leiv motiff example of this article, a simple null model, we are led via Laplace expansions of the integrals involved to the following equation. First, assume a proper prior 7r( 0). Then, Thus in order that 7r(0) would yield approximately the same value as the IBF's we have, from (9) and (ll), and, 7r(o! ~ exp [~ t log (B01(X1))] , 7rN(O) n 1=1 for the Arithmetic or the Geometric IBF, repectively. The asymptotic solution of this equation is I N EO [N ] 7r (0) = 7r (0) Ml B01(Xl) or 7rI(O) = 7rN(O)exp [E~l (logB6;.(x1))] for the Arithmetic and Geometric averages, respectively. These are called Intrinsic Priors. a) The Arithmetic Intrinsic Prior. I N EO [N] 00 7r (0)=7r (0) M B01(XI) = 2' 1 (00 +0) (14) This prior is quite appealing in more than one sense. First of all it integrates up to one as it is easy to check. Secondly, its median is precisely 00 , the distinguished point specified by the null model. Thirdly it is quite flat over the whole range with heavy tails. In Figure 1, it is depicted this prior for 00 = 5 .. It is seen how different it is with respect to the original "non informative" prior. In fact it may be argued that the Arithmetic Intrinsic Prior is the appropriate "default or authomatic" prior given the information that the null model is seriously considered. 9 The fact that this Intrinsic Prior is proper, is not a fortunate fact particular to this example as the following argument shows. Consider any simple Hypothesis Mo. Then integrating the Intrinsic Prior we find, J 7r I (B)dB J 7rN(B) J ~~(~~~)? J(X(l)IB)dX(l)dB = J ~~(~i~)? J 7rN(B)J(X(l)IB)dBdX(l) = J J(X(l)IBo)dX(l) = 1 This can be seen as a procedure for obtaining priors appropriate for Testing Hypothesis, and might be thought as the equivalent procedure for obtaining reference or default priors for testing hypothesis than Jeffreys or Berger-Bernardo algorithm for obtaining reference priors for estimation problems. Once obtained they can be used as a prior and the perform tha c,orresponding integration (often numerically) to obtain the proper Intrinsic Bayes Factors. Alternatively, approximations can be performed, obtaining in this example, the following simple approximation b) The Geometric Intrinsic Prior. The same method as above, but now using the average of the Geometric IBF, leads in this example to, I Bo [ Bo] 7r (B) = B2 exp 7jJ(1) - B . Integration of this prior gives exp(7jJ(l)) = 0.561, that is it is integrable but not proper as anticipated above. This prior, after being normalized for comparison, is also shown in Figure 1. Except close to B = 0, the Geometric and Arithmetic priors are quite close. 3 Discussion For ease of exposition we have restricted ourselves to describe some of the ideas behind the Intrinsic Strategy for Bayesian model comparison in a simple example. Far more complex situations are addressed in Berger and Pericchi (1994, 1996). 10 When the expectations involved in the Intrinsic Prior Equations are feasable this strategy gives a procedure for obtaining prior measures suitable for automatic analyses in the comparison of models in a proper Bayesian way. More generally, Arithmetic and Geometric IBF's, are often easely computable, and this paves the way for practical Bayesian Hypothesis Testing. For large data sets, and a large number of models computations might be expensive, but recently several approximating inexpensive procedures are being put forward (Varshavsky, 1995). 1 0.8 0.6 0.4 0.2 o o References Intrinsic arithmetic prior ISF I 1 2 3 Non informative 4 5 6 Figure 1. Intrinsic geometric prior ISF 7 8 9 10 [1] Allenby G.M. (1990), "Hypothesis testing with scanner data: the advantage at Bayesian Methods", Journal of Marketing Research, 27, 379-389. [2] Berger J.O. and Sellke T. (1987), "Testing a point null hypothesis: the irrecon-ciabilityof p-values and evidence", Journal of American Statistical Association, 82, 112-122. 11 [3] Berger J.O. and Bernardo J.M. (1992), "On the development of the reference prior method", Bayesian Statistics IV, (eds. J.M. Bernardo et al.), London: Oxford University Press. [4] Berger J.O. and Pericchi L.R. (1994), "The intrinsic Bayes Factor for Linear Models", To appear in Bayesian Statistics V, (eds. J.M. Bernardo et al.), Lon-don: Oxford University Press. [5] Berger J.O. and Pericchi L.R. (1996), "The intrinsic'Bayes Factor for Model Selection and Prediction", Journal of American Statistical Association, in press. [6] Bernardo J.M. and Smith (1993), "Bayesian Theory", J. Wiley and Sons, New York. [7] Smith, A.F.M. (1995), Contribution on the discussion of: O'Hagan A. "Frac-tional Bayes factors for model comparisons", J. Royal Statistical Society B, 57, 1, 120-122. [8] Varshavsky, J.A. (1995), "On the development of Intrinsic Bayes Factors", Tech-nical Report, Statistics Department, Purdue University, Indiana, U.S.A .. 12 

http://creativecommons.org/licenses/by-nc-nd/3.0/es/

The intrinsic bayes factor described by an example

Abstract

Similar works

Full text

Available Versions

Name not available

Name not available

Universidad Carlos III de Madrid e-Archivo