We consider an approach to prediction in linear model when values of the future explanatory variables are unavailable, we predict a future response y f at a future sample point x f when some components of x f are unavailable. We consider both the cases where x f are dependent and independent but normally distributed. A Taylor expansion is used to derive an approximation to the predictive density, and the influence of missing future explanatory variables (the loss or discrepancy) is assessed using the Kullback-Leibler measure of divergence. This discrepancy is compared in different scenarios including the situation where the missing variables are dropped entirely. © 2011 Copyright Taylor and Francis Group, LLC

Bhattacharjee, SK

Jammalamadaka, SR

Sabiruzzaman, M

Shamiri, A

Shamiri, Ahmed

Sabiruzzaman

Jammalamadaka, S Rao

eScholarship - University of California

UC Santa BarbaraUC Santa Barbara Previously Published WorksTitlePredictive Influence of Unavailable Values of Future Explanatory Variables in a Linear ModelPermalinkhttps://escholarship.org/uc/item/3nm4h12jJournalCommunication in Statistics- Theory and Methods, 40(24)ISSN0361-0926AuthorsBhattacharjee, SKShamiri, AhmedSabiruzzamanet al.Publication Date2011-12-15DOI10.1080/03610926.2010.513794 Peer reviewedeScholarship.org Powered by the California Digital LibraryUniversity of CaliforniaThis article was downloaded by: [University of California Santa Barbara]On: 17 November 2011, At: 13:00Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UKCommunications in Statistics - Theory and MethodsPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/lsta20Predictive Influence of Unavailable Values of FutureExplanatory Variables in a Linear ModelS. K. Bhattacharjee a , Ahmed Shamiri a , Md. Sabiruzzaman a & S. Rao Jammalamadaka ba Institute of Mathematical Sciences, University of Malaya, Kuala Lumpur, Malaysiab Department of Statistics and Applied Probability, University of California, Santa Barbara,California, USAAvailable online: 10 Nov 2011To cite this article: S. K. Bhattacharjee, Ahmed Shamiri, Md. Sabiruzzaman & S. Rao Jammalamadaka (2011): PredictiveInfluence of Unavailable Values of Future Explanatory Variables in a Linear Model, Communications in Statistics - Theory andMethods, 40:24, 4458-4466To link to this article:  http://dx.doi.org/10.1080/03610926.2010.513794PLEASE SCROLL DOWN FOR ARTICLEFull terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditionsThis article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form toanyone is expressly forbidden.The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses shouldbe independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims,proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly inconnection with or arising out of the use of this material.Communications in Statistics—Theory and Methods, 40: 4458–4466, 2011Copyright © Taylor & Francis Group, LLCISSN: 0361-0926 print/1532-415X onlineDOI: 10.1080/03610926.2010.513794Predictive Influence of Unavailable Valuesof Future Explanatory Variables in a LinearModelS. K. BHATTACHARJEE1, AHMED SHAMIRI1,MD. SABIRUZZAMAN1, ANDS. RAO JAMMALAMADAKA21Institute of Mathematical Sciences, University of Malaya,Kuala Lumpur, Malaysia2Department of Statistics and Applied Probability,University of California, Santa Barbara, California, USAWe consider an approach to prediction in linear model when values of the futureexplanatory variables are unavailable, we predict a future response yf at a futuresample point xf when some components of xf are unavailable. We consider boththe cases where xf are dependent and independent but normally distributed. ATaylor expansion is used to derive an approximation to the predictive density, andthe influence of missing future explanatory variables (the loss or discrepancy) isassessed using the Kullback–Leibler measure of divergence. This discrepancy iscompared in different scenarios including the situation where the missing variablesare dropped entirely.Keywords Discrepancy; Influence of variables; Kullback–Leibler divergence;Missing variable; Predictive density; Prior density; Taylor expansion.Mathematics Subject Classification 62F15; 62J05; 62B10; 62M20.1. IntroductionPredictive inference has been rightly called the central focus of statistical analysesand one may refer to Bjornstad (1990) for a review of various approaches. Inrecent years, much attention has been given to the influence of variables in bothclassical and Bayesian predictions. Bhattacharjee and Dunsmore (1991) consideredthe problem of the influence of variables in a logistic model in Bayesian predictiveapproach. In the logistic model, Zellner et al. (2004) compared the performanceof stepwise selection procedure with a bagging method. The influence of variableselection in Bayesian diagnostic perspective in logistic model is considered by Weiss(1995). Predictive influence of variables in normal linear regression model has beenstudied by Bhattacharjee and Dunsmore (1995). Mollah and Bhattacharjee (2008)Received June 8, 2010; Accepted July 27, 2010Address correspondence to S. K. Bhattacharjee, Institute of Mathematical Sciences,University of Malaya, Kuala Lumpur, Malaysia; E-mail: skbhattacharjee01@yahoo.com4458Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 Influence of Missing Variables in Linear Model 4459considered the problem of the influence of variables in general linear regressionmodel in Bayesian predictive approach in the presence of perfect multicollinearity.Our aim here is to detect the influence of missing future explanatory variablesin a normal linear model. We consider an approach to prediction analysis in generallinear model when the values of some or all of the future explanatory variablesare not available. We assume that in the observed data, the past records of all theexplanatory variables are available. We want to predict a single future response yfat a future sample point xf when some or all components of xf are unavailable.We assume that the future variables xf are normally distributed but both the casesare considered where xf ’s are dependent or independent. Improper prior densitiesare considered to derive the predictive density to assess the influence of the missingvariables. Since the predictive density is not mathematically tractable for missingfuture explanatory variables, the Taylor expansion is used to derive the approximatepredictive density. We then employ the Kullback–Leibler (K–L) directed measure ofdivergence (1951) to assess the influence of the missing future variables.2. Bayes Predictive Density in Linear ModelLet us consider the general linear model y = 0 + 1x1 + · · · + kxk + , where’s are the random errors normally distributed with mean zero and commonvariance 2. Our aim is to predict a future response yf when some or all of thefuture explanatory variables xf are unavailable. We denote the density of futureexplanatory variables by fxf . Let us suppose that r future explanatory variables,denoted by xfr, are not available. For convenience and without loss of generality,we assume that the last r future variables are unavailable.The density of an observed y is given by py  x   = Nx −1. The densityof a future response yf is pyf  xf    = Nxf −1. Then the predictive density ofa future response yf is given by pyf  xf  = ∫ pyf  xf   p   dd, wherep    is the posterior density of  and , and  is the observed data.We assume that the conditional density of xfr given xf∗ is independent of and , i.e., pxfr  xf∗   = pxfr  xf∗, where xf∗ denotes the future explanatoryvariables without variables xfr. First, we assume that xfi ’s are dependent and thedistribution of xf is k-dimensional multivariate normal, i.e., fxf  = MNk 	. Theconditional density of xfr given xf∗ is given byfxfr  xf∗ = MNr(∗r 	∗r) where  = (∗r)′  xf = (xf∗xfr)′  	 =[	11 	12	21 	22]∗r = r + 	21	−111(xf∗ − ∗) and 	∗r = 	22 − 	21	−111 	12Then the density of future response when xfr is missing is given byp(yf  xf∗  ) = ∫ p (yf  xf   )f (xfr  xf∗)dxfr= N(k−r∑0xfi i +k∑k−r+1∗i ik∑k−r+1ij	∗ij + −1)where ∗i is the ith component of ∗r and 	∗ij is the i jth component of 	∗r.Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 4460 Bhattacharjee et al.Using improper prior density for both  and , the approximate predictivedensity of yf when xfr is missing is given byPr(yf  xf∗) ≈ N(k−r∑0xfi bi +k∑k−r+1∗i bik∑k−r+1bibj	∗ij + s2)×{1+ 1/2k∑0Q∗ij   cov(i j)+ 1/2Q2   var }(1)evaluated at b and s2, where Q∗ij is the multiplicative factor for the second-orderapproximation.If xf ’s are independent the corresponding approximate predictive density isPr(yf  xf∗) ≈ N(k−r∑0xfi bi +k∑k−r+1ibik∑k−r+1b2i 	2i + s2)×{1+ 1/2k∑0Qij   cov(i j)+ 1/2Q2   var }evaluated at b and s2, where i and 	2i are mean and variance of the ith missingvariable.If no observation is missing then the corresponding predictive density based onall explanatory variables is given byP(yf  xf ) = St (n− p xfb s2 (1+ xf X′X−1 xf ′)) (2)Remark 2.1. Instead of considering normal linear regression model, the problemmay be extended for more general regression model where the errors follow aspherically symmetric distribution. The predictive density is completely unaffectedby departures of the normality assumption in the direction of the sphericallysymmetric family (Jammalamadaka et al., 1987). Therefore, the predictive density(2) based on all explanatory variables and no missing future variables will beunaltered if any spherically symmetric distribution is considered. The predictivedensity (1) for missing xfr will be changed due to different forms of the distributionand approximation may be required to derive the predictive density.3. Measure of Influence of the Missing VariablesTo assess the influence of the missing future explanatory variables xfr, we employthe Kullback–Leibler (K–L) directed measure of divergence DKL between the twopredictive densities (1) and (2). The form of the (K–L) divergence used here isgiven byDKL =∫p(yf  xf ) log{p (yf  xf )/pr (yf  xf∗)}dyf There are certainly many other measures of divergence between two distributions,e.g., a very general divergence measure is given by Csiszar (1963). He introducedDownloaded by [University of California Santa Barbara] at 13:00 17 November 2011 Influence of Missing Variables in Linear Model 4461the following class of divergence measures, called “h-divergence”, between twoprobability distributions F1· and F2·:Ih F1 F2 =∫Rh(dF1xdF2x)dF2xwhere h  0 →  is a convex function with h1 = 0. However, we use thespecial case of the K–L measure (corresponding to hx = − logx which ismore practical and easy to calculate in our case. Also, we can use informationmeasure I = ∫ P·  · logP·  ·− ∫ Pr·  · logPr·  ·, ratio measure R = DKL ÷I, and predictive interval to assess the influence of the missing future explanatoryvariables, these measures give similar conclusion to K–L directed measure ofdivergence. Details of these measures may be found in Bhattacharjee (1987), the firstauthor’s unpublished Ph.D. thesis.An explicit expression for DKL with Student distribution is difficult to obtain,so we derive approximate DKL by substituting approximate normal form of (2) asN(xfbn− pn− p− 2 s2(1+ xf X′X−1 xf ′)) (3)Since it is difficult to derive the distributional form of DKL, as in Bhattacharjee andDunsmore (1995), any discrepancy due to missing variables xfr is less than 1% ofthe largest discrepancy would be considered as negligible at 1% error. Where largestdiscrepancy occurs between the predictive density based on all variables and thepredictive density based on no variable.If xfi ’s are not independent then the approximate DKLbetween the predictivedensities (1) and (3) is given byDKL ≈12T 22r+ 12[22r− log{22r}− 1]− 12k∑ij=0E{Q∗ij   cov(i j)}+ 12E{Q2   var } (4)evaluated at b and s2, where2 = n− p s2n− p− 2{1+ xf X′X−1 xf ′}2r =k∑i=k−r+1b2i 	∗ij + s2T =k∑i=k−r+1(xfi − ∗i)biProof.DKL =∫p(yf  xf ) log{p (yf  xf )/pr (yf  xf∗)}dyf=∫p(yf  xf ) log p (yf  xf )dyf− ∫ p (yf  xf ) log pr (yf  xf )dyfDownloaded by [University of California Santa Barbara] at 13:00 17 November 2011 4462 Bhattacharjee et al.= log(22)−1/2 − 1/2−∫p(yf  xf ) log[N(k−r∑0xfi bi+k∑k−r+1∗i bik∑k−r+1bibj	∗ij + s2)]dyf+∫p(yf  xf ) log{1+k∑0Q∗ij   cov(i j)/2+Q2   var /2}dyf= log(22)−1/2 − 1/2−log(22r)−1/2 − 2/22r −[k∑0xfi bi −k−r∑0xfi bi −k∑k−r+1ibi]2−∫p(yf  xf ){k∑0Qij   cov(i j)/2+Q2   var /2}dyf neglecting O1/n terms in logarithmic series= T 2/22r +[2/22r − log{2/22r}− 1]/2−k∑ij=0E{Q∗ij   cov(i j)/2}+ E {Q2   var /2}evaluated at b and s2If xfi ’s are independent then the corresponding approximate DKL is same as (4) butreplacing 	∗ij by 	2i  ∗i by i and EQ∗ij  by EQij .4. IllustrationWe use the data in a four-variable problem given by Hald (1952). The responsey is the amount of heat evolved in calories per gram of cement. The explanatoryvariables x1 x2 x3, and x4 are the amount of tricalcium aluminate, tricalciumsilicate, calcium aluminium ferrate and dicalcium silicate, respectively. For theanalysis of the data, 20 future sample points xfi are chosen within the region ofprevious experience.We assume that in a natural informative experiment the values of the variablesx arise randomly from the same distribution as for the future variables xfi . Wealso assume that the marginal densities of the future variables xfi ’s are normal withmeans and variances are their sample means and sample variances, respectively. Wewant to observe here how much information is lost due to missing future variablesand to test whether the loss of information is negligible or not. We also wantto compare this lose of information with the variable deletion case where missingvariables will be deleted from the data set and a reduced model will be constructed.Then the predictive density will be derived based on the reduced model. Thediscrepancy between the predictive density based on full model and the predictivedensity based on reduced model is considered as lose of information of the variabledeletion case.Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 Influence of Missing Variables in Linear Model 4463Figure 1. K–L Divergence for single variable case.In this example, the single most influential variable is x1 and the least influentialvariable is x3. The best pair is x1 x2, and x1 x4 is the second best pair(see Bhattacharjee, 1987; Draper and Smith, 1966). The curves of DKL for missingxfi when the predictive density based on any single variable xi i = 1     4 are givenin Fig. 1. We see from the figure that the discrepancies are large for missing xf1 andnegligible for missing xf3 . We also see that discrepancies are minimum around themean of the missing variable.The plots of DKL for missing any single variable when the predictive densitybased on x1x2 x1x3, and x1x4 are given in Figs. 2(a)–(c). Here, we see that DKL islarge due to missing xf1 in any combination with the other variables and negligiblediscrepancies occurred when variable xf3 is missing. We also observe that DKL isminimum at the appropriate combination with the other explanatory variables.The discrepancies are larger toward the ends of the missing variables.Box plots of DKL for missing a single variable when predictive density is basedon all four variables are given in Fig. 3. From this figure it is clear that if a singlevariable is missing among the four variables, then the summary measures of thediscrepancies DKL due to missing xf1 is larger than the others and summary measuresof DKL due to missing xf3 are negligible.In Fig. 4, only two variables x1 and x2 are considered. Then box plots are shownto compare the discrepancies due to missing variable xi and due to deleting variablexi from the data set, i = 1 2. We see that discrepancies for missing variables aresmaller than the variable deletion case.Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 4464 Bhattacharjee et al.Figure 2. (a) DKL for missing a single variable when predictive density based on x1 andx2, (b) DKL for missing a single variable when predictive density based on x1 and x3, and(c) DKL for missing a single variable when predictive density based on x1 and x4. (colorfigure available online.)Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 Influence of Missing Variables in Linear Model 4465Figure 3. Box plots of DKL for missing any single variable when predictive density basedon four variables.Figure 4. Box plots of DKL for comparison of the two methods.5. ConclusionThe minimum discrepancies occur around the mean of the missing variable. We alsosee that the discrepancy depends on the influence of the missing variable. Larger theDownloaded by [University of California Santa Barbara] at 13:00 17 November 2011 4466 Bhattacharjee et al.influence of the missing variable, more the discrepancy, and less influential variablewill produce small discrepancy. Minimum discrepancies occur at the appropriatecombination of the explanatory variables. The discrepancies due to missing variablesare less than the discrepancies due to deleting the missing variables from the dataset. As in Bhattacharjee and Dunsmore (1995), one can test whether any discrepancyis negligible or not.ReferencesBhattacharjee, S. K. (1987). Influence of variables in Bayesian prediction. Unpublished Ph.D.Thesis, Department of Probability and Statistics, University of Sheffield, Sheffield,England.Bhattacharjee, S. K., Dunsmore, I. R. (1995). The predictive influence of variables in anormal regression model. J. Inform. Optimiz. Sci. 16(2):327–334.Bhattacharjee, S. K., Dunsmore, I. R. (1991). The influence of variables in a logistic model.Biometrika 78:851–856.Bjornstad, B. J. F. (1990). Predictive likelihood: a review. Statist. Sci. 5:242–265.Csiszar, I. (1963). Eine Informationstheoretische Ungleichung und ihre Anwendung auf denBeweiz der Ergodizitat von Markoffschen Ketten. Magyar Tud. Akad. Mat. Kutato Int.Kozl 8:85–108.Draper, N. R., Smith, H. (1966). Applied Regression Analysis. New York: John Wiley.Hald, A. (1952). Statistical Theory and Engineering Applications. New York: John Wiley.Jammalamadaka, S. R., Tiwari, R. C., Chib, S. (1987). Bayes prediction in the linear modelwith spherically symmetric errors. Econ. Lett. 24:39–44.Kullback, S., Leibler, R. A. (1951). On information and sufficiency. Ann. Math. Statist.22:79–86.Mollah, N. H., Bhattacharjee, S. K. (2008). Predictive influence of variables in a multivariatedistribution in presence of perfect multicollinearity. Commun. Statist. Theor. Meth.37(1):121–136.Weiss, R. E. (1995). The influence of variable selection: a Bayesian diagnostic perspective.J. Amer. Statist. Assoc. 90(430):619–625.Zellner, D., Keller, F., Zellner, G. (2004). Variable selection in logistic regression models.Commun. Statist. Simul. Computat. 33(3):787–805.Downloaded by [University of California Santa Barbara] at 13:00 17 November 2011 

English

Predictive Influence of Unavailable Values of Future Explanatory Variables in a Linear Model

https://escholarship.org/uc/item/3nm4h12j

Predictive influence of unavailable values of future explanatory variables in a linear model

Abstract

Similar works

Full text

Available Versions

eScholarship - University of California