Generalized linear modelling (GLM) is a versatile technique, which may be viewed as a generalization of well-known techniques such as least squares regression, analysis of variance, loglinear modelling, and logistic regression. In may applications, low-order interaction (such as bivariate interaction) terms are included in the model. However, as the number of categorical variables increases, the total number of low-order interactions also increases dramatically. In this papaer, we propose to constrain bivariate interactions by a bi-additive model which allows a simple graphical representation in which each category of every variable is represented by a vector

Groenen, P.J.F. (Patrick)

Koning, A.J. (Alex)

Groenen, Patrick

Koning, Alex

EUR Research Repository

Generalized Bi-additive modelling for categorical data

Erasmus University Digital Repository

Generalized Bi-additive Modelling forCategorical DataPatrick J.F. Groenen1 and Alex J. Koning1Econometric Institute,Erasmus University Rotterdam, The Netherlandsgroenen@few.eur.nl, koning@few.eur.nlEconometric Institute Report EI 2004-05Abstract. Generalized linear modelling (GLM) is a versatile statistical technique,which may be viewed as a generalization of well-known techniques such as leastsquares regression, analysis of variance, loglinear modelling, and logistic regression.In many applications, low-order interaction (such as bivariate interaction) terms areincluded in the model. However, as the number of categorical variables increases,the total number of low-order interactions also increases dramatically. In this pa-per, we propose to constrain bivariate interactions by a bi-additive model whichallows a simple graphical representation in which each category of every variable isrepresented by a vector.1 IntroductionGeneralized linear modelling (GLM) is a versatile statistical technique, whichmay be viewed as a generalization of well-known techniques such as leastsquares regression, analysis of variance, loglinear modelling, logistic regression(Nelder andWedderburn (1972); McCullagh and Nelder(1989)). In this paper,we limit ourselves to categorical predictor variables. Then, GLMs may consistof main effects, bivariate, and higher-order interactions. Since higher orderinteractions are generally difficult to interpret, we consider only bivariateinteractions here. Note that as the number of categorical variables increases,the total number of bivariate interactions may also increase dramatically. Inaddition, if there are many categories per variable, it becomes increasinglydifficult to interpret the estimated interaction parameters because there areso many of them. Our aim here is to provide a simple graphical representationto facilitate the interpretation of all bivariate interactions. To reach this goal,we impose rank restrictions on the bivariate interactions, thus leading to abi-additive model.For two categorical variables, van Eeuwijk (1995), De Falguerolles andFrancis (1992) and Gabriel (1996) have provided algorithms for a bi-additivemodel within GLM. Here we propose a bi-additive model for more than twocategorical predictors. To some extent, the proposed model can be seen as ageneralization of multiple correspondence analysis to GLM.2 Groenen and Koning2 Generalized Bi-Additive ModellingLet us introduce some notation needed for generalized linear modelling (GLM).Let y be the dependent vector of n objects that needs to be predicted by mcategorical variables. Also, let the categorical variable j be represented by theindicator matrix Gj with a zero-one variable for each category with gijk = 1if observation i falls in category k of variable j and gijk = 0 otherwise. Letthe number of categories of variable j be denoted by Kj and g′ij be row i ofGj .The central idea behind GLM is that the distribution of the dependentvariable belongs to a given family of distributions (popular choices are theNormal, Poisson, binomial, gamma, and inverse Gaussian families). Thisleaves some freedom, which allows the distribution to vary from object toobject. Especially, the expectation µi of the dependent variable may differfrom object to object, and is assumed to depend on the values taken bythe predictor variables through the linear predictor ηi (examples are givenbelow). Finally, the inverse of the link function g(µi) = ηi relates the lin-ear predictor to µi. Some standard link functions are the logarithm, power,logistic, identity, and probit (McCullagh and Nelder(1989)).A simple linear predictor may be specified by ηi = c+∑mj=1 g′ijaj , wherec is an overall mean and aj is a vector of main effect for variable j. However,we are interested in bivariate interactions as well, so that we need the linearpredictorηi = c+m∑j=1g′ijaj +m∑j=1m∑l=j+1g′ijBjlgil,where Bjl is the Kj ×Kl matrix of bivariate interactions between variablesj and l. It is easily verified that g′ijBjlgil selects the appropriate row andcolumn element that corresponds to the categories of the variables j and lof object i. Note that summation over j > l or j < l gives the same results,because one can always choose Bjl = B′lj so that g′ijBjlgil = g′ilBljgij .We can obtain more insight and compact notation by joining the effectsof all variables. Let G be the super indicator matrix with all m variables nextto each other, that is, G = [G1,G2, . . . ,Gm], g′i be row i of G, and a be thevector of all main effects. Finally, all bivariate interaction effects are joinedinto the symmetric partitioned block matrixB =0 B′12 . . . B′1mB12 0 . . . B′2m....... . ....B1m B2m . . . 0 .The diagonal blocks are zero because j 6= l. Then, we may writeηi = c+m∑j=1g′ijaj +m∑j=1m∑l=j+1g′ijBjlgil = c+ g′ia+12g′iBgi. (1)Generalized Bi-additive Modelling 3The basic idea of this paper is to impose constraints on the interactionterms Bjl. The type of constraint that we consider is the one of commonrank-reduction, that is, to require thatBjl = YjY′l (2)withYj a Kj×p matrix andYl a Kl×p matrix being of rank not higher thanp > 0. Such rank constraints are similar to the ones used in multiple corre-spondence analysis, joint correspondence analysis, or homogeneity analysis.This rank constrained bi-additive model can be expressed asηi = c+m∑j=1g′ijaj +m∑j=1m∑l=j+1g′ijYjY′lgil. (3)To avoid that Yj also estimates main effects, we impose the restriction thatYj has column mean zero. This restriction also implies that Bjl = YjY′l haszero row and column mean, which is a restriction that is usually imposed onbivariate interactions to ensure uniqueness. We shall refer to the matrix Yas the interaction generating matrix, and to the kth column of Yj as the kthdimension of interaction generators belonging to the categorical variable j.To fit this model, we have developed a prototype in MatLab that optimizesthe likelihood by iterated weighted least squares and iterative majorization.Note that standard likelihood theory applies to model (3), and hence wemay employ the likelihood ratio test to determine the rank p. From thisperspective, it is relevant to know the degrees of freedom associated to therank p model. Observe that the parameters in the rank p model are theconstant term c, the main effect vectors aj and the elements of the interactiongenerating matrix Y. Hence, the number of parameters in this model equals1 +m∑j=1Kj + pm∑j=1Kj .However, we have also imposed several restrictions on these parameters. Eachof them main effect vectors and each of the p dimensions of them interactiongenerators should add up to zero. Moreover, the interaction generating matrixY can be rotated by any orthonormal rotation matrix T (with T′T = TT′ =I) without affecting Bjl since Bjl = YjY′l = YjTT′Y′l for all j and l.Therefore, without loss of generality, we impose the restriction that∑jYjis orthogonal thereby making the Yj ’s unique. This restriction implies fixingp(p − 1)/2 of the elements of the the Yj ’s. Summarizing, the number ofrestrictions in the rank p model is equal tom+mp+p(p− 1)24 Groenen and KoningTable 1. Frequencies of occurrence of lung cancer for (non) smokers in eight Chi-nese cities (taken from Liu (1992), see also Tabel 3.3 in Agresti (1996), p. 60).CityLung Bei- Shang- Shen- Nan- Har- Zeng- Tai- Nan-cancer Smoker jing hai yang jing bin zhou yuan chang Totalyes yes 126 908 913 235 402 182 60 104 2930yes no 35 497 336 58 121 72 11 21 1121no yes 100 688 747 172 308 156 99 89 2359no no 61 807 598 121 215 98 43 36 1979Total 322 2900 2594 586 1046 508 213 250 8419The difference in number of parameters and number of restrictionsdfmodel =1 + m∑j=1Kj + pm∑j=1Kj− [m+mp+ p(p− 1)2]= 1 + (1 + p)m∑j=1(Kj − 1)− p(p− 1)2 .is the degrees of freedom associated to the rank p model. The residual de-grees of freedom dfres is obtained by subtracting dfmodel from the degreesof freedom n (the number of objects) associated to the so-called saturatedmodel.3 Application: lung cancer in ChinaLung cancer is one of the leading causes of death in the People’s Republic ofChina. It has been estimated that in the year 2025, the number of new cancercases in China will reach three million, of which two million are associatedwith smoking, and the remaining one million attributable to other causes(Peto (1994); Peto, Chen and Boreham (1996)).A number of epidemiological studies have investigated the associationbetween lung cancer and smoking in China. In Liu (1992), a meta-analysisis presented of eight case-control studies conducted in Beijing, Shanghai,Shenyang, Nanjing, Harbin, Zhengzhou, Taiyuan, and Nanchang. Table 1cross-classifies a total of 4081 lung cancer cases and 4338 controls accordingto smoking behaviour and city. In this paragraph, we investigate the relationbetween smoking and lung cancer in China by applying the generalized bi-additive model introduced in the previous section to the data in Table 1. Notethat m = 3, K1 = 2− 1, K2 = 2− 1, and K3 − 1, and hence∑mj (Kj − 1) =1 + 1 + 7 = 9.Generalized Bi-additive Modelling 5Table 2. Summary of fit for different models, using the Poisson family of distribu-tions and the log link function on the data of Table 1.Model Deviance dfres pMain effects 457.1 22 .0000Bi-additive interaction model, rank 1 35.7 13 .0007Bi-additive interaction model, rank 2 5.4 5 .3690Table 2 summarizes the fit of three models when using the Poisson familyof distributions and a log link function. The main effects model may beregarded as a special case of model (3) with rank p = 0. Next, adding a firstdimension of bilinear interactions to each variable yields model (3) with rankp = 1. Finally, adding a second dimension of bilinear interactions to eachvariable yields model (3) with rank p = 2. Adding further dimensions wouldlead to models with degrees of freedom exceeding 32, the degrees of freedomof the saturated model; such models are unidentified, and have no statisticaluse.Note that the three models in Table 2 are nested, and thus may be com-pared as usual by means of the likelihood ratio test (that is, by relatingdifferences in deviance to the chi-square distribution corresponding to thedifference in degrees of freedom). Obviously, the rank 2 model is to be pre-ferred, as it is the only model that fits the data.Table 3. Estimation results obtained by applying the rank 2 bi-additive decompo-sition model to the Chinese lung cancer-smoking data.Main Interaction generatorsVariable Category effects Dim 1 Dim 2Overall mean c 5.005Lung cancer Yes 0.410 -0.494 -0.169No -0.410 0.494 0.169Smoking Yes -0.124 -0.055 -0.413No 0.124 0.055 0.413City Beijing -0.727 -0.015 -0.052Shanghai 1.557 0.348 -0.133Shenyang 1.406 0.124 -0.053Nanjing -0.121 0.004 -0.057Harbin 0.472 0.047 -0.071Zengzhou -0.239 0.081 -0.080Taiyuan -1.270 -0.342 0.421Nanchang -1.078 -0.248 0.0256 Groenen and Koning-.5 0 .5-.6-.4-.20.2.4.6Beijin gShanghaiShenyangNanjin gHarbi nZengzhouTaiyuanNanchangYesSmokingNo SmokingYes LungcanNo LungcanDimension 1 Dimension 2 -1.0-.50.51.01.5City BeijingCity ShanghaiCity ShenyangCity NanjingCity HarbinCity ZengzhouCity TaiyuanCity NanchangSmoker YesSmoker NoLungcan YesLungcan NoVariable   Categorya. Maineffects b. Plot of interaction effectsFig. 1. Representation of the bi-additive decomposition model of the Chinese lungcancer-smoking data. Panel (a) shows the main effects and Panel (b) the decompo-sition of the interactions in two dimensions.Applying the rank 2 bi-additive decomposition model to the Chinese lungcancer-smoking data yields the estimation results listed in Table 3. Theseresults are visualized in Figure 1, where Panel (a) shows the main effects andPanel (b) gives the vectors for the interaction generators Yj .Figure 1 may be interpreted as follows. To see the size of the interac-tion effect, project the vector of one category onto a category of a differentvariable. For example, Taiyuan is characterized by having more nonsmokersthan smokers, when corrected for the main effects of smoking. The reasonis the nonsmoking vector projects highly on the vector of Taiyuan. Longvectors lead to longer projections and thus to a stronger interaction effect.Conversely, short vectors have short projections, indicating a small inter-action effect. Therefore, the cities Beijing, Harbin, Nanjing, Zengzhou, andShenyang will only have small interaction effects with the other variables.In Figure 1, three Chinese cities relatively stand out: Nanchang, Taiyuanand Shanghai. Nanchang was badly battered after the Communist takeover,but reinvented itself as a centre of modern steel and chemical industry (Leff-man, Lewis and Atiyah (2000), p. 52). Taiyuan’s extensive coal mines wereconstructed by the Japanese in 1940; serious industrialization began after theCommunist takeover and today it is the factories that dominate, relentlesslyprocessing the region’s coal and mineral deposits (Leffman, et al. (2000), p.225). After forty years of stagnation, Shanghai seems certain to recaptureits position as East Asia’s leading business city, a status it last held beforeWorld War II (Leffman, et al. (2000), p. 337).Generalized Bi-additive Modelling 7-.5 0 .5-.6-.4-.20.2.4.6Beijin gShanghaiShenyangNanjin gHarbi nZengzhouTaiyuanNanchangYesSmokerNoSmokerYesLungcanNoLungcanDimension 1 Dimension 2 .25.30.40.50.35.45City BeijingCity ShanghaiCity ShenyangCity NanjingCity HarbinCity ZengzhouCity TaiyuanCity NanchangSmoker YesSmoker NoLungcan Yes a. Plot of interaction effects b. Interaction effects conditioned     on having lungcancer Fig. 2. Interaction terms conditioned on category ‘yes’ of the variable ‘lung cancer’.Panel (a) shows the projections of all other catgories on category ‘yes’ of the variable‘lung cancer’ and Panel (b) the values of the interactions.Recall that Table 1 was compiled by Liu (1992) from the results of eightdifferent case-control studies investigating lung cancer in China. In this re-spect, the interactions between the variable Lung Cancer on one hand andthe variables Smoking and City on the other hand are of primary interest.Thus, to gain insight into the causes of lung cancer in China, we shouldproject each category of the variables Smoking and City onto the categoryYes of the variable Lung Cancer, as indicated in Panel (a) of Figure 2. Panel(b) visualizes the interaction values according to the bi-additive interactionmodel for the categories of Smoking and City conditioned on having lungcancer.As to be expected, smoking is clearly a risk factor for obtaining lungcancer. However, Panel (b) of Figure 2 seems to suggest that there is also anenvironmental risk factor, as the interaction between the industrial cities ofNanchang and Taiyuan on one hand and the presence of lung cancer on theother hand is much higher than the interaction between the leading businesscity Shanghai and the presence of lung cancer.4 ConclusionsWe have proposed a new model for representing two-way interactions for aGLM with more than two categorical predictor variables, where we constrain8 Groenen and Koningthe two-way interactions to have reduced rank. Each category is representedby a vector in a plot. The interaction effect between two categories of differ-ent predictor variables is obtained by projecting the vector of one categoryonto the vector of another. Categories of the same variable should not becompared within the plot, but only by looking at the main effects. The mainadvantage of the bi-additive interaction model is that interactions betweenseveral variables can be visualized in a relatively simple display, even whenthe total number of categories is large.ReferencesAGRESTI, A. (1996): An introduction to categorical data analysis. New York, Wi-ley.DE FALGUEROLLES, A. and FRANCIS, B. (1992): Algorithmic approaches forfitting bilinear models. In: Y. Dodge and J. Whittaker (Eds.): Compstat 1992.Heidelberg, Physica-Verlag, 77–82.EEUWIJK, F. A. (1995): Multiplicative interaction in generalized linear models.Biometrics, 85, 1017-1032.GABRIEL, K. R. (1996): Generalised bilinear regression. Biometrika, 85, 689-700.LEFFMAN, D., LEWIS, S. and ATIYAH, J. (2000): China. London, Rough GuidesLtd.LIU, Z. (1992): Smoking and lung cancer in China. Internal Journal of Epidemiol-ogy, 21, 197-201.MCCULLAGH, P. and NELDER, J. A. (1989): Generalized linear models. London,Chapman and Hall.NELDER, J. A. and WEDDERBURN, R. W. M. (1972): Generalized linear models.Journal of the Royal Statistical Society A, 135, 370-384.PETO R (1994): Smoking and death - the past 40 years and the next 40. BritishMedical Journal 309, 937–939.PETO, R., CHEN, Z. M. and BOREHAM. J. (1996): Tobacco - The growing epi-demic in China. Journal of the American Medical Association, 275, 1683–1684.

Generalized bi-additive modelling for categorical data

Abstract

Similar works

Full text

Available Versions

EUR Research Repository

EUR Research Repository

Erasmus University Digital Repository