[Abstract]: Prediction distribution is a basis for predictive inferences applied in many real world situations. It is a distribution of the unobserved future response(s) conditional on a set of realized responses from an informative experiment. Various statistical approaches can be used to obtain prediction distributions for different models. This study derives the prediction distribution(s) for multiple linear regression model using the Bayesian method when the error components of both the performed and future models have a multivariate Student-t distribution. The study observes that the prediction distribution(s) of future response(s) has a multivariate Student-t distribution whose degrees of freedom depends on the size of the realized sample and the dimension of the regression parameters’ vector but does not depend on the degrees of freedom of the errors distribution

Khan, Shahjahan

Rahman, Azizur

English

University of Southern Queensland ePrints

Proceedings of The Third Conferences On Research And Education In Mathematics 2007 Rahman A and Khan S  1 PREDICTION DISTRIBUTION FOR LINEAR REGRESSION MODEL WITH MULTIVARIATE STUDENT-T ERRORS UNDER THE BAYESIAN APPROACH  Azizur Rahman, Shahjahan Khan 	 Abstract:  Prediction distribution is a basis for predictive inferences applied in many real world situations. It is a distribution of the unobserved future response(s) conditional on a set of realized responses from an informative experiment. Various statistical approaches can be used to obtain prediction distributions for different models. This study derives the prediction distribution(s) for multiple linear regression model using the Bayesian method when the error components of both the performed and future models have a multivariate Student-t distribution. The study observes that the prediction distribution(s) of future response(s) has a multivariate Student-t distribution whose degrees of freedom depends on the size of the realized sample and the dimension of the regression parameters’ vector but does not depend on the degrees of freedom of the errors distribution.  Introduction  The prediction distribution of future response(s) can be derived from the regression model for statistical predictive inferences. Predictive inference uses the observations from a realized experiment to make inference about the performance of the future observation(s) of a future experiment. Many authors have considered the linear regression model in prediction problems and they have been used different methods to derive the prediction distribution. General prediction problems have been discussed by Jeffreys (1961). Fraser and Haq (1970) used the structural distribution approach, Aitchison and Dunsmore (1975) and Geisser (1993) used the Bayesian approach, and  Haq (1982) and Haq and Khan (1990) used the structural relations approach to obtain the prediction distribution from the linear model to mention a few. For details of predictive inferences and applications of prediction distribution interested readers may refer to Geisser (1993) and Khan (2004), and references therein.   Most of the authors have contributed to solving the prediction problem by using linear models with independent and normal errors. Unlike others Haq and Khan (1990) obtained prediction distribution for the linear regression model with multivariate Student-t error terms by using the structural relation approach. In real life situations when the underlying distributions have heavier tails, linear models with multivariate Student-t errors have been emphasized and used by Zellner (1976), and Sutradhar and Ali (1989) among others. This study assumes that the error terms of the performed as well as the future multiple regression models have a joint multivariate Student-t distribution, and obtains the prediction distribution(s) of future response(s) by the Bayesian method under a uniform prior distribution.  The Multiple Regression model  Consider the linear regression model for an 1×n dimensional responses vector y    y = X β  + e   (1) where X is the design matrix of order an kn ×  ( kn > ); /110 ),...,,( −= kββββ  is a vector of k  regression parameters; and e is the  errors vector associated with the responses vector y. Assume that each elements of e is uncorrelated but not independent with others and has the same univariate Student-t distribution with location 0, scale 0>σ and ν  degrees of freedom (d.f.). Thus the joint probability density function (p.d.f.) of e is  f(e) ∝  222 [)( −− + σνσne / e] 2n+−ν  (2) It is noted that E(e) = 0, a vector of 0’s and Cov(e) = 21)2( νσν −− I n  for >ν  2 in which I n is an identity matrix .  Hence the p.d.f. of the realized responses vector y is y ~ t n (X β , νσ , ) with density  Proceedings of The Third Conferences On Research And Education In Mathematics 2007 Rahman A and Khan S  2  f(y| 2,σβ ) ∝  222 [)( −− + σνσn(y - X β ) / (y - X β )] 2n+−ν (3) The posterior distribution of parameters for a set of sample observations is typically the major objective of the Bayesian statistical analysis. To obtain a posterior distribution using the Bayes’s Theorem a prior distribution of unknown parameters is essential. Adopting the invariance theory (Jeffreys, 1961), the joint uniform prior density of parameters can be written as g( 2,σβ ) ∝  2−σ . Under this uniform prior the joint posterior density of β  and 2σ for the realized responses y can be obtained as   f( 2,σβ | y) ∝  2222 [)( −+−+ σνσn(y - X β ) / (y - X β )] 2n+−ν  (4) Inference about unknown parameters β  and 2σ from the above linear model has been considered in other studies (Zellner, 1976; Fraser and Ng, 1980). In this case, the study is concerned to derive the prediction distribution of future response(s) from the future model, conditional on the observed responses y from the realized model.  The Bayesian Prediction rule  If z*  be an unobserved future response from a future regression model with the same regression parameters and assumption of the realized model but with different design matrix, then under the Bayesian approach the prediction distribution of z*  given y can be obtained by solving the following integral  f(z* | y) ∝    >β σ 02  f(2,σβ | y) f(z*) 2σd βd  (5) where f( 2,σβ | y) is the joint posterior density of unknown parameters β  and 2σ that is provided in (4) and  f(z*) is a probability density of the future response  z*  from the future model. This principle is appropriate when the future response z* is independently distributed from the observed responses y, that means z* and y are not dependent to each other. However, in this study the responses from the realized as well as the future models are dependent but uncorrelated.           Prediction distribution of a set of future responses  Let y f  be a set of  fn  future responses from the model in (1) corresponding to the kn f ×  order design matrix X f  and 1×fn  dimensional errors vector e f .  Thus the future model can be expressed as  y f  = X f β  + e f  (6) where,  e f ~ t fn (0, νσ , ), and hence  y f ~ t fn ( X f β , νσ , ).  According to the assumption, the observed errors vector e and the unobserved future errors vector e f  are uncorrelated but not independent then their respective observed responses y from the realized model and unobserved future responses y f  from the future model are also dependent but uncorrelated. Thus the combined joint p.d.f. of y and y f  is   f(y, y f | 2,σβ ) ∝  222 [)( −+−+ σνσfnnQ] 2fnn++−ν (7) where Q = (y - X β ) / (y - X β ) + (y f  - X f β ) / (y f  - X f β ) and ν  is the d.f. of the errors distribution.  Using the prior density g( 2,σβ ) ∝  2−σ  and the joint p.d.f. of the combined responses y f and y  in (7), the joint posterior density of β  and 2σ for the combined responses y f and y  can be easily obtained by the Bayes’s Theorem, as   Proceedings of The Third Conferences On Research And Education In Mathematics 2007 Rahman A and Khan S  3  f( 2,σβ | y, y f ) ∝   2222 [)( −++−+ σνσfnnQ] 2fnn++−ν (8) As y f and y are not independent so they are not independently distributed. Thus the usual idea in (5) is not appropriate here. In this situation since the density function of a set of future responses y f from the future model is linked with the density function of the set of observed responses y from the realized model within the combined joint p.d.f. in equation (7), the prediction distribution of y f  can be obtained by solving the following integral  f(y f  | y) ∝    >β σ 02  f(y, y f |2,σβ )  g( 2,σβ ) 2σd βd       ∝    >β σ 02  f(2,σβ | y, y f )  2σd βd  (9) That means, in this case the prediction distribution of future responses can be derived from the joint posterior density function of the parameters for the combined responses y and y f . Now equation (8) can be expressed as the following convenient form  f( 2,σβ | y, y f ) ∝  (Q) 2fnn+−  2121]1[fnnCCQ++−−−+νν  (10) where C = 21νσ−Q  .  Considering the transformation r = 21 )( σfnnQ +−  in (10) and then after using the results, equation (9) can be written as  f(y f  | y) ∝   β r (Q)2fnn+−  2121fnnfrnnr++−−++ννν        dr βd   (11)   It is clear that r has an F distribution with ν  and fnn +  degrees of freedom that is, r ~ F fnn+,ν .  Employing the F integral in (11) to integrating over r, the prediction density of future responses becomes  f(y f  | y) ∝  β (Q)2fnn+−  βd   (12) Now Q = (y - X β ) / (y - X β ) + (y f  - X f β ) / (y f  - X f β ) can be expressed as the following form    Q = A + ( β -P) / M ( β -P)   (13) where A = y / y  +  y /f  y f  - (y / X +  y /f  X f ) M 1− ( X / y + X /f  y f ) is free from the parameters’ vector β , M = X / X + X /f X f  and P =  M1− ( X / y + X /f  y f ). Using the expression of Q in (13) to equation (12) and then integrating over β  by using the multivariate Student-t integral, it is easy to obtain the following p.d.f. of y f  given y   f(y f  | y) ∝  [y / y  +  y /f  y f  - (y / X +  y /f  X f ) M 1− ( X / y + X /f  y f )] 2fnkn +−− Hence the prediction distribution of a set of future responses y f , conditional on a set of realized responses y, is obtained as  Proceedings of The Third Conferences On Research And Education In Mathematics 2007 Rahman A and Khan S  4  f(y f  | y) = fΦ  [(n - k) + (y f - X f βˆ ) / H (y f - X f βˆ )] 2fnkn +−−  (14) where H = s 2− (Ifn- X f M1− X /f ), βˆ  = (X / X) 1− X / y is the OLS estimator of  the regression vector β , s2= 1)( −− kn [(y - X βˆ ) / (y - X βˆ )] = y / [ I n  - X(X / X) 1− X /  ]y, and the normalizing constant of the prediction distribution is given by  fΦ = [ ]21221/1)(22sknknXMXInknffnffnf−	 −Γ−	 +−Γ −pi.  Here, it is clear that y f , the vector of a set of future responses, has an fn -dimensional multivariate Student-t distribution with the location X f βˆ , scale [s 2− (I fn - X f M 1− X /f )] 21−, and the shape parameter kn − . This result is identical with the results obtained for the same model by Haq and Khan (1990), and for the multiple regression model with independent and normal errors by Zellner (1971) and Geisser (1993) among others.  Therefore, it is noted that the prediction distribution is unaffected by departures from the model with indepedent and normal errors to multivariate Student-t errors distribution.   Prediction distribution of a single future response  For  fn = 1 the set of future responses vector  y f  becomes a single future response, and hence if  y f  denotes a single future response, then the future regression model in (6) becomes the following form     y f  = x f β  + e f   (15) where, x f is a k×1  order design vector, β  is the same regression coefficients vector of order 1×k  and e f is the error term associated with y f and e f has a univariate Student-t distribution as e f ~ t 1  (0, νσ , ). By the same operations as used in previous section for the derivation of prediction distribution of a set of future responses, it can be easily obtained the joint posterior density of parameters for a single response y f and the realized responses vector y under the same prior distribution g( 2,σβ ) ∝  2−σ , as  f( 2,σβ | y, y f ) ∝   2232 [)( −+−+ σνσnQ] 21++−nν  (16) where Q = (y - X β ) / (y - X β ) + (y f  - x f β ) 2 in which y f is scalar.   The prediction distribution of a single future response y f can be derived from the joint posterior density function for the combined responses y f  and y in (16) by integrating over the parameters β  and 2σ . For completing the derivation of prediction distribution of a single future response, the same operational steps is used as considered in the previous section. At first the joint posterior density can be expressed as its convenient form as like in equation (10), and then using an appropriate transformation r = 21 )1( σ+− nQ , the parameter 2σ can be eliminated by the F 1, +nν  integral. After that, Q can be expressed as a quadratic form of β  and then the properties of the multivariate Student-t distribution can be used to complete the integration over β . Finally, the prediction distribution of a single future response y f , conditional on a set of realized responses y, is obtained as Proceedings of The Third Conferences On Research And Education In Mathematics 2007 Rahman A and Khan S  5  f(y f  | y) = fΨ  [(n - k) + (y f - x f βˆ ) / H (y f -  x f βˆ )] 21+−−kn  (17) where H = s 2− (1 - x f M 1− x /f ) in which M = X / X + x /f x f  , βˆ  = (X / X) 1− X / y is the OLS estimator of  the regression vector β , s 2 = 1)( −− kn [(y - X βˆ ) / (y - X βˆ )] = y / [ I n  - X(X / X) 1− X /  ]y, and fΨ  represents the normalizing constant which is given by  fΨ = [ ]21221/1)(2121sknknxMxkn ff−	 −Γ−	 +−Γ −pi.  Thus, the prediction distribution of a single future response for the multiple regression model with multivariate Student-t error terms is a univariate Student-t distribution with appropriate parameters.  Conclusion  The prediction distribution of future response(s), conditional on a set of observed responses has been derived for the multiple linear regression model having multivariate Student-t errors by the improper Bayesian method. Results reveal that the prediction distribution of a single future response and a set of future responses are a univariate Student-t distribution and a multivariate Student-t distribution respectively. It has been shown that the prediction distributions for the multiple regression model remains identical by a change in the error distribution from normal to multivariate Student-t distribution. Furthermore, the prediction distribution depends on the observed responses and the design matrices of the realized model as well as the future model. The shape parameter of the prediction distribution depends on the size of the realized sample and the dimension of parameters vector of the model. However, the shape parameter of the prediction distribution does not depend on the d.f. of the errors distribution.   References:   Aitchison, J and Dunsmore, I R (1975). Statistical prediction analysis. Cambridge University Press, Cambridge. Fraser, D A S and Haq, M S (1970). Inference and prediction for multilinear model. Journal of Statistical Research, University of Dhaka, Dhaka 4: 93-109. Fraser, D A S and Ng, K W (1980). Multivariate regression analysis with spherical error. In V.P.R. Krishnaiah, (ed.): Multivariate Analysis, North-Holland Publishing Co., New York, 369-386. Geisser, S (1993). Predictive inference: An introduction. Chapman and Hall, London. Haq, M S (1982). Structural relation and prediction for multivariate models. Statistische Hefte, 23: 218-228. Haq, M S and Khan, S (1990). Prediction distribution for a linear regression model with multivariate Student-t error distribution. Communications in Statistics -Theory and Methods, 19(12): 4705-4712. Jeffreys, H (1961). Theory of probability. 3rd Edn., Oxford University Press, Oxford. Khan, S (2004). Predictive distribution of regression vector and residual sum of squares for normal multiple regression model. Communications in Statistics: Theory and Methods, 33(10): 2423-2441. Sutradhar, B C and Ali, M M (1989). A generalization of the Wishart distribution for the elliptical model and its moments for the multivariate-t model. Journal of Maltivariate Analysis, 29: 155-162. Zellner, A (1971). An introduction to Bayesian inference in econometrics. John Wiley and Sons, NY. Zellner, A (1976). Bayesian and non-Bayesian analysis of the regression model with multivariate Student-t error terms. Journal of the American Statistical Association, 71(354):  400-405. 

A generalization of the Wishart distribution for the elliptical model and its moments for the multivariate-t model.

An introduction to Bayesian inference in econometrics.

Bayesian and non-Bayesian analysis of the regression model with multivariate Student-t error terms.

Inference and prediction for multilinear model.

Multivariate regression analysis with spherical error.

Prediction distribution for a linear regression model with multivariate Student-t error distribution.

Predictive distribution of regression vector and residual sum of squares for normal multiple regression model.

Predictive inference: An introduction.

Statistical prediction analysis.

Structural relation and prediction for multivariate models.

Theory of probability. 3rd Edn.,

Prediction distribution for linear regression model with multivariate Student-t errors under the Bayesian approach

http://eprints.usq.edu.au/2446/1/Rahman-Khan-Malayis-Apr07.pdf

Prediction distribution for linear regression model with multivariate Student-t errors under the Bayesian approach

Abstract

Similar works

Full text

Available Versions

University of Southern Queensland ePrints