This paper is concerned with the Bayesian estimation of a Multivariate Probit

model. In particular, this paper provides a method to sample the restricted variancecovariance

matrix directly from its conditional posterior density. The method allows

the application of a standard Gibbs sampling algorithm to sample from the posterior

density of the parameters, and hence it avoids the use of a Metropolis step. The method

uses a decomposition of the Inverted Wishart density and alternative identification

restrictions

Leon-Gonzalez, R.

English

White Rose Research Online

This is a repository copy of Sampling the Variance-Covariance Matrix in the Bayesian Multivariate Probit Model.White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/9879/Monograph:Leon-Gonzalez, R. (2003) Sampling the Variance-Covariance Matrix in the Bayesian Multivariate Probit Model. Working Paper. Department of Economics, University of Sheffield ISSN 1749-8368 Sheffield Economic Research Paper Series 2003008eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/Reuse Unless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy solely for the purpose of non-commercial research or private study within the limits of fair dealing. The publisher or other rights-holder may allow further reproduction and re-use of this version - refer to the White Rose Research Online record for this item. Where records identify the publisher as the copyright holder, users can verify any specific terms of use on the publisher’s website. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request.   Sheffield Economic Research Paper Series  SERP Number: 2003008     Roberto León González  Sampling the Variance-Covariance Matrix  in the Bayesian Multivariate Probit Model  September 2003     Department of Economics University of Sheffield 9 Mappin Street Sheffield S1 4DT  United Kingdom www.shef.ac.uk/economics  Sampling the Variance-Covariance Matrix in theBayesian Multivariate Probit Model∗Roberto León GonzálezDepartment of EconomicsUniversity of SheffieldSeptember 2003AbstractThis paper is concerned with the Bayesian estimation of a Multivariate Probitmodel. In particular, this paper provides a method to sample the restricted variance-covariance matrix directly from its conditional posterior density. The method allowsthe application of a standard Gibbs sampling algorithm to sample from the posteriordensity of the parameters, and hence it avoids the use of a Metropolis step. The methoduses a decomposition of the Inverted Wishart density and alternative identificationrestrictions.∗This paper was circulated before as part of the discussion paper 09/02 of the Department of Economicsat the University of York11 IntroductionA common feature in qualitative-dependent variable models is that the scale of thelatent variable is not identified. As a normalization, restrictions are usually placedupon the variance parameters. In the context of the Bayesian multinomial probit,just one of the elements in the variance-covariance matrix is restricted to be one,and several algorithms to sample the variance-covariance matrix directly are available(Cowles 1996, McCulloch et al. 2000 and Nobile 2000).In the multivariate probit model, a larger number of normalization restrictions arenecessary in order to make the model identifiable. In a Bayesian context, Chib andGreenberg (1998) propose to restrict the diagonal elements of the variance-covariancematrix (Σ) to be equal to one. However, there is not any method to obtain a drawfrom the posterior conditional density of Σ subject to such restriction. In order toovercome this difficulty, Chib and Greenberg (1998) use a Metropolis step (Metropoliset al. 1953). However, a Metropolis step requires a proposal density that is agood approximation of the posterior density. For this reason, Chib and Greenberg(1998) proposed to incorporate a maximization algorithm within the MCMC algorithmin order to calculate the mode and hessian of the conditional posterior density ofΣ. However, even with this additional computational and programming burden, aMetropolis step will probably not work well when the number of variables is large.Lui (2001) proposes an algorithm to sample Σ directly from its conditional posterior.However, the algorithm relies on the specification of an improper prior for thecovariance matrix and restricts each equation to contain the same set of regressors.Unfortunately, there are no results that ensure that the posterior density would existif an improper prior is specified, and hence the specification of a proper prior seems tobe necessary. In addition, proper priors are often required for Bayes factors to be welldefined, specially when testing hypothesis about model specific parameters.This paper proposes a simple method to sample Σ directly from its conditionalposterior density, using a proper prior. The method relies on an alternativenormalization and on a decomposition of the inverted Wishart density. Despite ofthe use of an alternative normalization, the method can also yield results according to2the normalization used by Chib and Greenberg (1998).The plan of the paper is as follows. Section 2 describes the notation for the model,and for simplicity in the exposition concentrates on the bivariate probit model (i.e.two equations). Section 3 looks at the more general case. Section 4 describes how thealgorithm can be adapted when the normalization of Chib and Greenberg (1998) isused. Section 5 summarizes the paper.2 Sampling the Variance-Covariance Matrix inthe Bivariate Probit ModelLet Yi be a (T × 1) vector of zeros and ones. In the multivariate probit model,each component yit of Yi is determined by a continuous unobserved latent variabley∗it generated according to the following process,y∗it = Xitβt + eit i = 1, ..., N t = 1, ..., T (1)the vector ei = (ei1, ..., eiT )T is normally distributed with zero mean and covariancematrix Σ = (σjk). The binary variable yit is equal to one if and only if y∗it ≥ 0, andis equal to zero otherwise. Xit is a 1 × kt vector of regressors and βt is a vector ofparameters.2.1 Identification in the Bivariate Probit ModelThe likelihood contribution of an observation (0, 1) is,Pr {yi1 = 0, yi2 = 1} = PrXi1β1 + ei1 ≤ 0Xi2β2 + ei2 > 0(2)Let ∆ = (δjk) be the lower triangular Cholesky decomposition of Σ, so thatΣ = ∆∆T .∆ =√σ11 0σ12/√σ11√σ22 − (σ12)2 /σ11(3)Then the vector ei can be seen as a transformation of a random vector εi that containsindependent standard normal variables. That is, ei = ∆εi, where εi follows a N(0, I).3The probability in (2) can be rewritten as,Pr {yi1 = 0, yi2 = 1} = PrXi1β1 +√σ11εi1 ≤ 0,Xi2β2 +σ12√σ11εi1 +√σ22 − (σ12)2 /σ11εi2 > 0(4)From expression (4), different values for (β,Σ) give the same value for theprobability. In particular, for two arbitrary positive constants (c, d), the value ofthe parameters {c (β1, δ11) , d (β2, δ21, δ22)} give the same value for the probability as{β1, δ11, β2, δ21, δ22}. Hence, the model is not identified.The most common normalization in the literature is to fix σ11 = σ22 = 1(e.g. Chib and Greenberg 1998). However, the following section shows that itis more convenient from the point of view of computational tractability to chooseσ11 = σ22 − (σ12)2 /σ11 = 1. From expression (4), both normalizations make the modelidentified without imposing any unnecessary restrictions upon the parameters.2.2 The Sampling MethodIn the Bayesian approach, model specification is completed by providing a priordistribution for the parameters. Let the prior for Σ be an Inverted WishartIW2 (df0,K0) distribution conditional to the restriction that σ11 = σ22 − (σ12)2 /σ11 =1. That is, given a matrix Σ that satisfies the restriction, the kernel of the prior is:|Σ|−df0/2 exp(−1/2tr(Σ−1K0))(5)The expected value of the unrestricted prior is 1df0−2T−2K0, where T is the dimensionof Σ. The definition of inverted Wishart distribution used here is the one described inPress (1986, pp. 117).The conditional posterior of Σ given parameters β and latent data {y∗it : t =1, ... , T}Ti=1 is an inverted Wishart IW2 (df, K) with the restriction that σ11 =σ22 − (σ12)2 /σ11 = 1. The parameters of this inverted Wishart are df = df0 + Nand K = K0 +∑Ni=1 eieTi .The following Theorem is useful to sample Σ conditional on the normalisationδ11 = δ22 = 1. Cowles et al (1996) sample from an Inverted Wishart distribution with4the restriction that one element in the diagonal is equal to one. They made use of thefollowing theorem, which can be found in Bauwens et al. (1999, pages 305-306).Theorem 1 Let Σ be distributed as an IWd (n, G), and be partitioned as Σ = (Σij) ,i,j=1,2, where Σ11 is a q × q matrix. Define Σ22·1 = Σ22 − Σ21Σ−111 Σ12, then1) Σ22·1|Σ11 ∼ IWd−q (n − q, G22·1)2) Σ12| (Σ22·1,Σ11) ∼ MN(Σ11 (G11)−1 G12,Σ11 (G11)−1 Σ22·1ΣT11)where MN refers to a matrix normal distribution.Theorem 1 states that the conditional posterior of σ12 given β and all latent datay∗it is a normal distribution and hence it can be sampled directly.3 Dimension larger than 2The proposed normalization in the bivariate probit is to set the variance of ei1 (σ11)and the conditional variance of ei2 given ei1(σ22 − (σ12)2 /σe11)both equal to one. Orin other words, to fix the elements in the diagonal of the cholesky decomposition equalto one. Similarly, in the case of T ≥ 2 the proposed normalization isV ar (ei1) = V ar (ei2|ei1) = V ar (ei3|ei2, ei1) = ... = V ar(eiT |ei(T−1), ..., ei1)= 1 (6)This normalization can also be shown to be the same as fixing the elements in thediagonal of the Cholesky decomposition equal to one. The following lemma gives anstatistical interpretation to each of the elements of the Cholesky decomposition.Lemma 1 If ei follows a N (0,Σ), and ∆ is the lower triangular choleskydecomposition, then∆ =√σ11 0 0 0 0σ12√σ11√σ22·1 0 0 0σ13√σ11σ23·1√σ22·1√σ33·12 0 0σ14√σ11σ24·1√σ22·1σ34·12√σ33·12√σ44·123 0...σ1T√σ11σ2T ·1√σ22·1σ3T ·12√σ33·12σ4T ·123√σ44·123...√σTT ·123...(T−1)where σtt·vwz = V ar (eit|eiv, eiw, eiz) and σth·vwz = Cov (eit, eih|eiv, eiw, eiz).5Proof. If εi follows a N (0, I), then the linear combination ∆εi also follows anormal N (0,Σ). Hence, ei can be expressed as ei = ∆εi, that is:ei1 = δ11εi1ei2 = δ21εi1 + δ22εi2ei3 = δ31εi1 + δ32εi2 + δ33εi3ei4 = δ41εi1 + δ42εi2 + δ43εi3 + δ44εi4...eiT = δT1εi1 + δT2εi2 + δT3εi3 + δT4εi4 + ... + δTT εiT(7)Using these equations, Σ can be related to ∆:V ar (ei1) = σ11 = V ar (δ11εi1) = (δ11)2Cov (ei1, ei2) = σ12 = Cov (δ11εi1, δ21εi1 + δ22εi2) = δ11δ21V ar (ei2|ei1) = σ22·1 = V ar (δ21εi1 + δ22εi2|δ11εi1) = (δ22)2Cov (ei3, ei1) = δ31δ11,Cov (ei3, ei2|ei1) = δ32δ22,V ar (ei3|ei1, ei2) = (δ33)2Cov (ei4, ei1) = δ41δ11,Cov (ei4, ei2|ei1) = δ42δ22,Cov (ei4, ei3|ei1, ei2) = δ43δ33,V ar (ei4|ei1, ei2, ei3) = (δ44)2Cov (eiT |ei1) = δT1δ11,Cov (eiT , ei2|ei1) = δT2δ22,Cov (eiT , ei3|ei1, ei2) = δT3δ33, ...,V ar(eiT |ei1, ei2, ei3, ..., ei(T−1))= (δTT )2Hence, restriction (6) is equivalent to fixing the diagonal elements of the Choleskydecomposition equal to one. In addition, from (7) this restriction identifies the model,not imposing any unnecessary restrictions upon the parameters.Let the prior for the elements out of the diagonal of Σ have a kernel equal toexpression (5). The elements in the diagonal are fixed by normalisation (6), whichensures that Σ is positive definite. Note that since normalization (6) ensures that Σis positive definite, the non-diagonal elements of Σ are not subject to any restrictions.From the Appendix, the prior variance of the free elements in Σ can be made arbitrarilylarge by choosing small values in the diagonal of K0.Let Σjj be the sub-matrix of Σ containing the first j rows and the first j columns ofΣ and Kjj the corresponding sub-matrix of K. Let Σj be the vertical vector containing6the first (j − 1) rows in the jth column and let Kj be the corresponding sub-matrix ofK. Following Theorem 1, if Σ follows an unrestricted IWT (df, K), then• ΣT conditional on Σ(T−1)(T−1) and σTT ·12...(T−1) follows a normal distribution• σTT ·12...(T−1) is independent of Σ(T−1)(T−1) and follows an inverted Wishartdistribution.• Σ(T−1)(T−1) follows an inverted Wishart distribution.The third property holds because the marginal distribution of sub-matrices centeredin the diagonal is also an inverted Wishart distribution (Press 1986, pp. 118-119). SinceσTT ·12...(T−1) is independent of Σ(T−1)(T−1), conditioning on σTT ·12...(T−1) = 1 does notchange the marginal distribution of Σ(T−1)(T−1).Consider now that Σ follows an IWT (df,K) with the restriction that σ11 =σ22·1 = ... = σTT ·12...(T−1) = 1. By applying the above argument recursively, themarginal distribution of Σ2 given the restriction is a normal distribution. In addition,the distribution of Σn conditional on Σ(n−1)(n−1) and σnn·12...(n−1) is also a normaldistribution, for 2 ≤ n ≤ T . The appendix gives full detail of the distributions involvedin this decomposition of the inverted Wishart density.The following algorithm describes how Σ can be sampled from its conditionalposterior distribution:Algorithm 1 Step 0: Fixσ11 = σ22·1 = σ33·12 = ... = σTT ·12...(T−1) = 1Step 1: Sample σ12 conditional on σ11 and σ22·1 from aN(σ11K−111 K2, (σ11)2 K−111 σ22·1)Step 2: Fix σ22 = σ22·1 + (σ12)2 σ−111Step 3: Sample Σ3 conditional on Σ22 and σ33·12 from aN(Σ22K−122 K3, σ33·12Σ22K−122 Σ22)Step 4: Fix σ33 = σ33·12+ ΣT3 Σ−122 Σ3.7Step 2(n–1)-1: Sample Σn conditional on Σ(n−1)(n−1) and σnn·12...(n−1) from aN(Σ(n−1)(n−1)K−1(n−1)(n−1)Kn, σnn·12...(n−1)Σ(n−1)(n−1)K−1(n−1)(n−1)Σ(n−1)(n−1))Step 2(n-1): Fix σnn = σTT ·12...(T−1)+ ΣTnΣ−1(n−1)(n−1)Σn.4 An alternative normalization.This section describes how to transform the estimated values for (β,Σ) if anothernormalization is chosen. In particular, instead of normalization (6), one might beinterested in choosing the more widely used normalization:σ11 = σ22 = ... = σTT = 1 (8)A sample from the posterior of (β,Σ) given normalization (8) can be obtained bysimply transforming the values sampled using Algorithm 1. Let C1 be the diagonalmatrix of dimension T with diagonal equal to (1/√σ11, 1/√σ22·1, 1/√σ33·12 , ...,1/√σTT ·12...(T−1)) and let C2 be the diagonal matrix of dimension T with diagonalequal to (1/√σ11, 1/√σ22, 1/√σ33 , ..., 1/√σTT ). The parameters that are identifiedwith normalization (6) are(1/√σ11β1, ..., 1/√σTT ·12...(T−1)βT , C1ΣC1)(9)When normalization (8) is used, the identified parameters are(1/√σ11β1, ..., 1/√σTT βT , C2ΣC2) (10)Let(βk,Σk)1be the kth value in the chain when normalization (6) is used. And let(βk,Σk)2be the kth value in a chain in which normalization (8) is chosen. To obtaina sample from the posterior when normalization (8) is used, transform(βk,Σk)1inthe following way:• Construct Ck2 as the diagonal matrix with diagonal equal to(1/√σk11, 1/√σk22, 1/√σk33, ..., 1/√σkTT)• Fix(βk, Σk)2=(1/√σk11β1, ..., 1/√σkTT βT , , Ck2 ΣkCk2)85 ConclusionThis paper has described a simple method (Algorithm 1) to sample Σ directly from itsposterior conditional density. This method makes it possible to use a standard GibbsSampling algorithm (e.g. Gelfand and Smith 1990 ) and hence avoids the use of aMetropolis step.9Appendix: Decomposition of an invertedWishartLet fN (x;µ,Φ) denote the density function of a N (µ,Φ), evaluated at x. And letfIW (x; p, df, K) denote the density function of an inverted Wishart of dimension p,degrees of freedom df , and expected value K 1df−2p−2 .Proposition 1 If Σ follows an unrestricted inverted Wishart IWT (df, K), then thedensity function of(σ11, σ22·1, σ12, σ33·12,Σ3, ..., σTT ·12...(T−1),ΣT)can be expressed as:fIW (σ11; 1, df − 2T + 2, k11) ×fIW (σ22·1; 1, df − 2T + 3, k22·1) ×fN(σ12;σ11 (k11)−1 K2, σ11 (k11)−1 σ11σ22·1)×fIW (σ33·12; 1, df − 2T + 4, k33·12) ×fN(Σ3; Σ22 (K22)−1 K3,Σ22 (K22)−1 Σ22σ33·12)×... ×... ×fIW(σTT ·12...(T−1); 1, df − 2T + (T + 1) , kTT ·12...(T−1))×fN (ΣT ; Σ(T−1)(T−1)(K(T−1)(T−1))−1KT ,Σ(T−1)(T−1)(K(T−1)(T−1))−1Σ(T−1)(T−1)σTT ·12...(T−1))Proof. This decomposition results from applying recursively Theorem 1, in theway described in Section 3.10ReferencesBauwens, L., Lubrano, M. and Richard, J. (1999) Bayesian Inference in DynamicEconometrics Models. Oxford University Press, Oxford.Chib., S and Greenberg, E. (1998) ”Analysis of multivariate probit models”Biometrika 85:347-361Cowles, K., Carlin, B. P., and Connett, J. E. (1996) ”Bayesian Tobit Modelligof Longitudinal Ordinal Clinical Trial Compliance Data with NonignorableMissingness” Journal of the American Statistical Association, 91, 86-98.Gelfand, A. E., and Smith, A. F. M. (1990) ”Sampling Based Approaches toCalculating Marginal Densities” Journal of the American Statistical Association,85, 398-409.Liu, C., (2001) ”The Art of Data Augmentation: Discussion” Journal ofComputational and Graphical Statistics 10, 75-81McCulloch, R. E., Polson, N. G. and Rossi, P. E. (2000) ”A Bayesian Analysisof the Multinomial Probit Model with Fully Identified Parameters” Journal ofEconometrics 99, 173-193Metropolis, N., Rosenbluth, A. W. , Rosenbluth, M. N. , Teller, A. H. and Teller,E. (1953) ”Equation of state calculations by fast compuing machine” Journal ofChemical Physics 21, 1087-91Nobile, A. (2000) ”Comment: Bayesian Multinomial Probit Models with aNormalization Constraint” Journal of Econometrics 99, 2, 335-45Press, S. J. (1982) Applied Multivariate Analysis: Using Bayesian and FrequentistMethods of Inference, Florida: Krieger.11

Sampling the Variance-Covariance Matrix

in the Bayesian Multivariate Probit Model

https://eprints.whiterose.ac.uk/9879/1/SERP20030081.pdf

Sampling the Variance-Covariance Matrix in the Bayesian Multivariate Probit Model

Abstract

Similar works

Full text

Available Versions

White Rose Research Online