Distance-based regression allows for a neat implementation of the Partial Least Squares recurrence. In this paper we address practical issues arising when dealing with moderately large datasets (n ~ 10^4) such as those typical of automobile insurance premium calculations

Boj, Eva

Claramunt, M. Merce

Fortiana, Josep

Grané, Aurea

English

Name not available

Implementing PLS for distance-based regression: computational issues

Universidad Carlos III de Madrid e-Archivo

  Working Paper 06-35 Statistics and Econometrics Series 14 May 2006  Departamento de EstadísticaUniversidad Carlos III de MadridCalle Madrid, 12628903 Getafe (Spain)Fax (34) 91 624-98-49  IMPLEMENTING PLS FOR DISTANCE-BASED  REGRESSION: COMPUTATIONAL ISSUES Eva Boj(1), Aurea Grané(2), Josep Fortiana(3)  and M. Mercè Claramunt(1)Abstract Distance-based regression allows for a neat implementation of the Partial Least Squares recurrence. In this paper we address practical issues arising when dealing with moderately large datasets (n ~ 104) such as those typical of automobile insurance premium calculations.   Keywords: Distance-based prediction; PLS regression; Large datasets;  Automobile insurance data.               (1) Dept. de Matemàtica Econòmica, Financera i Actuarial, Universitat de Barcelona, Avda. Diagonal, 690, 08034 Barcelona, Spain, e-mail evaboj@ub.edu . (2) Dpto. de Estadística, Universidad Carlos III de Madrid, C/ Madrid 126, 28903 Getafe (Madrid), Spain, e-mail: agrane@est-econ.uc3m.es . (3) Dept. de Probabilitat, Lògica i Estadística, Universitat de Barcelona, C/ Gran Via de les Corts Catalanes, 585, 08007 Barcelona, Spain, e-mail: fortiana@ub.edu .  Work partially supported by Spanish grant MTM2005-02370 (Ministerio de Educación, Ciencia y Tecnología and FEDER). Implementing PLS for distance-basedregression: computational issuesEva Boj1, Aurea Grane´,2 Josep Fortiana3and M. Merce` Claramunt11Dept. de Matema`tica Econo`mica, Financera i Actuarial,Fac. de Cie`ncies Econo`miques i Empresarials, Univ. deBarcelona, Av. Diagonal, 690, 08034 Barcelona, Spain.2Dept. de Estad´ıstica, Univ. Carlos III, Calle Madrid, 126, 28903Getafe (Madrid), Spain.3Dept. de Probabilitat, Lo`gica i Estad´ıstica, Fac. deMatema`tiques, Univ. de Barcelona, Gran Via de les CortsCatalanes, 585, 08007 Barcelona, Spain.SummaryDistance-based regression allows for a neat implementation of the PartialLeast Squares recurrence. In this paper we address practical issues arisingwhen dealing with moderately large datasets (n ∼ 104) such as those typicalof automobile insurance premium calculations.Keywords: Distance-based prediction; PLS regression; Large datasets;Automobile insurance data.11 IntroductionDistance-Based Regression (DBR) (see Cuadras 1989, Cuadras & Arenas1990, Cuadras et al. 1996) is a method for predicting a numerical responsey from a set z of both numerical and categorical predictors. The name ofthe procedure originates in the fact that it involves a metric in the space ofpredictors, d( · , · ), which must be Euclidean in the sense of MultidimensionalScaling (see Section 2).In this paper we adapt PLS regression to the DBR context, with an empha-sis on computational issues arising in treating moderately large (n ≤ 105)datasets. Since DBR for such datasets entails a linear regression on a largenumber of predictors, it seems particularly adequate for PLS. It is so in-deed, but huge matrices appearing at intermediate phases impose a carefulout-of-core treatment.A motivation for this study is the analysis of automobile insurance data,more precisely in rate-making, i.e., predicting total claim amounts from a setof a priori risk factors, whose results will be used to determine risk premiafor new policy holders (Boj et al. 2004, 2005). Such data usually consist ofa number of observations ranging from a moderately large to a very largesize (> 106). In this paper we will concentrate on the former, which canbe directly input to DBR, whereas the very large case requires a modifiedapproach with additional processing, such as subsampling or stratification.The paper is structured as follows: in Section 2 we outline the main char-acteristics of Distance Based Regression. In Section 3 we derive the DBRversion of the PLS recurrence and in Section 4 we give some details on itsimplementation. As an illustration, in Section 5 we apply the method to anexample with real data.2 Distance Based RegressionThe DBR procedure is as follows: Given n observed pairs {(yi, z i), 1 ≤ i ≤n}, we compute the matrix D , with entries (d2(z i, z j)), and the doubly-centered inner products matrix,G = −12J ·D · J ,where J = I − 1 · 1 ′/n is the n × n centering matrix. The Euclideanrequirement is equivalent to the positive semidefiniteness of G, hence to theexistence of an X such that G = X ·X ′, called in this context a centeredEuclidean configuration of D , meaning that 1 ′ ·X = 0 and that the squaredEuclidean interdistances‖x i − x j‖22between the rows x 1, . . . ,xn of X coincide with the corresponding entriesin D . The verification of this equivalence involves some simple algebra forwhich we refer to any standard textbook on Multidimensional Scaling, suchas Borg and Groenen (2005).DBR of y = (yi) on the predictor distance matrix D is defined as a LeastSquares regression of y on a Euclidean configuration X of D . The rationalesupporting this definition is that DBR contains Ordinary Least Squares re-gression as a particular case. Specifically, with the Pythagorean metric (ℓ2)on numerical z predictors DBR reduces to OLS.For a givenG there are many Euclidean configurations: Applying any nonsin-gular linear transformation or a translation to one of them generates another.DBR predictions, however, are independent of the choice. The reason is thatboth the hat matrix giving the fitted responses,ŷ = H · ythat is, the orthogonal projectorH = X · (X ′ ·X )−1 ·X ′on the subspace G = 〈X 〉 ⊂ Rn generated by the columns of X , and G itselfcan be expressed directly in terms of G or D , hence they are independent ofX . It can be checked thatH = G+ ·G = G ·G+,where G+ is the Moore-Penrose generalized inverse of G.When implementing DBR a possible method is to actually compute an ex-plicit Euclidean configuration X . For instance, Cuadras et al. (1996) use thePrincipal Coordinates Euclidean configuration, X = U ·Λ, obtained from theeigendecomposition G = U · Λ2 ·U ′. This version of DBR amounts to per-forming a Principal Components Regression (PCR) (see, e.g., Jolliffe 2002).One of the pitfalls of PCR (Cuadras 1993, 1998, Hadi & Ling 1998)is the factthat the first few Principal Axes, with a greater variance, are not necesarilyhighly correlated with the response y . Furthermore, diagonalization of largen× n matrices presents substantial computational problems.The choice of a metric is an important component in DBR model-building:it has points of contact with the choice of a link function for a GeneralizedLinear Model. In principle it is possible to tailor a metric to reflect specificinformation on predictors and on how their proximity relates to the particularprediction under study. Most of the times, however, it suffices to utilizean omnibus metric function which satisfies the Euclidean condition referredto in the Introduction. One very popular metric for mixtures of numericalcontinuous, nominal categorical and binary predictor variables is the one3based on Gower’s general similarity coefficient, which for two p-dimensionalvectors z i and z j is equal tosij =∑p1h=1 (1− |zih − zjh| /Rh ) + a+ αp1 + (p2 − d) + p3 , (1)where p = p1 + p2+ p3, p1 is the number of continuous variables, a and d arethe number of positive and negative matches, respectively, for the p2 binaryvariables, and α is the number of matches for the p3 multi-state categoricalvariables. Rh is the range of the h-th continuous variable. The squareddistance is computed as:d2(z i, z j) = 1− sij . (2)It can be proved (Gower 1971) that (2) satisfies the Euclidean condition.3 PLS for DB regressionPLS regression provides us with a sensible alternative. The standard proce-dure (see, e.g., Helland 1988, Hoskuldsson 1988) can be described as follows:Given an n× p centered predictor matrix X and an n× 1 centered responsey , for each k, 1 ≤ k ≤ p, we obtain sets:{uj ; 0 ≤ j ≤ k}, of n× 1 vectors,{E j ; 0 ≤ j ≤ k}, of n× p matrices,{f j ; 1 ≤ j ≤ k}, of n× 1 orthogonal vectors,{aj ; 1 ≤ j ≤ k}, of p× 1 vectors,{bj ; 1 ≤ j ≤ k}, of scalars,for which the following decompositions hold:X = f 1 · a ′1 + · · ·+ f k · a ′k +Ek, (3)y = f 1 b1 + · · ·+ f k bk + uk. (4)These sets are generated sequentially, through the following recursion: LetE0 ≡ X and u0 ≡ y . Then, for each k ≥ 1, ak and bk are the regressioncoefficientsak = (E′k−1 · f k)/(f ′k · f k),bk = (u′k−1 · f k)/(f ′k · f k),and f k is the linear combinationf k = Ek−1 ·wk, (5)4with weightswk = E′k−1 · uk−1. (6)The next recurrence step starts with the residualsEk = Ek−1 − f k · a ′k, (7)uk = uk−1 − f k bk. (8)In the DBR context, X is assumed to be a centered Euclidean configurationof the square distance matrix D , but its explicit computation is not requiredfor the PLS recursion, since all the steps are invariant under a change of Xand may be performed by operating directly with G. Indeed,f 1 =X ·X ′ · y = G · y ,b1 =y ′ · f 1f ′1 · f 1=y ′ ·G · yy ′ ·G2 · y ,ŷ1 = f 1 b1 =G · y (y ′ ·G · y)y ′ ·G2 · y ,which can be written more compactly in terms of f 1 = G · y :ŷ1 =(f 1 · f ′1‖f 1‖2)· y = P1 · y ,where P1 = f 1 ·f ′1/‖f 1‖2 is the orthogonal projector on the one-dimensionallinear space spanned by f 1. The residualy˜1 = y − ŷ1 = Q1 · y ,where Q1 = I −P1 is the complementary orthogonal projector, also dependson X through G. Similarly,a1 =X ′ · f 1f ′1 · f 1=X ′ ·G · yy ′ ·G2 · y =X ′ · f 1‖f 1‖2,X̂ 1 = f 1 · a ′1 = P1 ·X ,X˜ 1 =X − X̂ 1 = Q1 ·X .Thus we see that it is possible to define the following recursive procedure:Given an n×n doubly centered positive semidefinite matrix G and an n× 1centered response y , for each k, 1 ≤ k ≤ rank(G), we obtain sets:5{uj ; 0 ≤ j ≤ k}, of n× 1 vectors,{Gj ; 0 ≤ j ≤ k}, of n×n doubly centered positivesemidefinite matrices,{f j ; 1 ≤ j ≤ k}, of n× 1 orthogonal vectors,{(Pj ,Qj); 1 ≤ j ≤ k}, of pairs of complementary n× northogonal projectors,{bj; 1 ≤ j ≤ k}, of scalars,starting with u0 ≡ y and G0 ≡ G. The recursion, for k ≥ 1, follows by:f k = Gk−1 · uk−1, bk =1‖f k‖2f ′k · uk−1,Pk =1‖f k‖2f k · f ′k, Qk = I −Pk,uk = uk−1 − f k bk = Qk · uk−1,Gk = Qk ·Gk−1 ·Qk.showing that each recurrence step, hence the whole sequence of computations,involves only the distances D . The fitted response vector at step k is givenby the sumŷk = f 1 b1 + · · ·+ f k bk.It is worth noting that, for 1 ≤ j ≤ k, f j bj = P j · uj−1 and, by recurrence,f j bj = Pj · y ,hence the k-th hat matrix, i.e., the orthogonal projector H k such that ŷk =H k · y , is given by the sum of mutually orthogonal orthogonal projectors:H k = P1 + · · ·+Pk.4 ImplementationVery large datasets, with a number of observations greater than 105, requirespecific treatments with the goal of reducing the computational effort to afeasible range. Such an endeavor would involve a displacement of the theo-retical emphasis to designing and analyzing suitable subsampling strategies,a problem which will not be treated here. Instead, our current targets arethose datasets referred to in the Introduction as moderately large, consistingof n ∼ 104 observations. Even for this size, both the distance matrix D andthe inner products matrix G are too large to fit comfortably into the main6memory of a standard computer. For instance, if n = 104, double precisionstorage requires about 0.8GB. The PLS-DB scheme outlined above can beeasily adapted to this situation. To this end, we implement the followingpieces of software:1. calcd: Computes D from the observed predictors z i and writes itsequentially (one row at a time) to the disk. Several matrices along thecomputations will be stored out-of-core in this way and henceforth wename them big matrices. We are not taking advantage of the symmetryof D for storage since the resulting code complexity and the increasedaccess time override the intended saving.2. bigproduct: An auxiliary function for multiplying an n×n big matrix,M , times an n×1 in-core vector, resulting another n×1 in-core vector.3. bigproject: An auxiliary function to compute(I − v · v ′) ·M · (I − v · v ′)for an n×1 in-core unit vector v and an n×n big matrix M , resultinganother big matrix, the orthogonal projection ofM onto the hyperplane〈v〉⊥, the orthogonal complement of v . We need bigproject both toobtain G from D (with v = 1/√n) and in each PLS step, to obtainthe residual.4. PLSstep: From an n × 1 in-core y˜ and a big G˜, computes the new(ŷ , y˜) and G˜.A main function controls the sequence and measures convergence. The PLS-DBR procedure can integrate formal stopping rules for the iterative algo-rithm, by adapting usual devices such as the Akaike or the Bayes Informa-tion Criteria or a Crossvalidation Statistic, which can be implemented in astraightforward manner. As a final observation, it is not necessary to allocatedisk space for several Gigabyte-sized matrices. Three of them will suffice, onefor D and two more to flip-flop between the two G˜’s of consecutive steps.A package with the set of MATLAB functions implementing DBR-PLS isavailable from the authors upon request.5 An illustrative case in insuranceWe consider the problem of rate-making in automobile insurance, that is,prediction of total claim amounts from a set of a priori risk factors (for ter-minology and context see Boj et al. 2004). The results are used to determine7Table 1: Set of predictors for the insurance dataset• Continuous predictors:– Power of the vehicle (in HP)– Vehicle age– Price (Original list price)– Age of the main driver– Driving license age• Categorical predictors:– Sex of the main driver– Geographical zone (10 levels)risk premia for new policy holders. As an illustration we use a real datasetwith n = 11028 cases, obtained from a portfolio from a Spanish insurer inthe period 1996–1997 corresponding to compulsory civil liability insurance.We set up a DB regression model, where the response is the total claimamount per policyholder and the set of predictors is of a mixed type, com-prising both continuous and categorical predictors (Table 1).The metric we use is Gower’s distance (2), based on the general similaritycoefficient (1). In addition to being one of the most popular measures ofproximity for mixed data types, as mentioned above, it appears to be ade-Table 2: R2 as a function of the number of PLS stepsk R21 0.76982 0.80343 0.821410 0.863320 0.880630 0.886449 0.89348Table 3: Comparing PLS and PCR: R2 as a function of the number of stepsR2 (PLS-DBR) R2 (PCR-DBR)1 0.1601 0.08252 0.2072 0.10723 0.2274 0.10754 0.2471 0.10809 0.3091 0.151624 0.4610 0.227649 0.7132 0.277174 0.8531 0.290899 0.9254 0.3294149 0.9736 0.3640199 0.9904 0.4003642 1.0000 0.7202quate for the type of actuarial prediction we are currently dealing with (Bojet al. 2002).Table 2 shows the successive determination coefficients obtained as a functionof the number of PLS steps. We observe a quite steep ascent, as it is to beexpected for the PLS algorithm. The purpose of this table is to demonstratethe comparatively effortless computation required by PLS-DBR to attaina given predictive quality, neatly defeating other methods that require anexplicit Euclidean configuration.For instance, computation of DBR through the Principal Coordinates Eu-clidean configuration (PCR-DBR) for datasets in the range of sizes we aredealing with here, is at least impractical, perchance utterly impossible, sinceit would involve obtaining eigenpairs of a huge out-of-core matrix. We haveextracted a relatively small subset (n = 1000) from the dataset used abovein order to compare both methods. Table 3 shows the results as pairs of R2for equal number of steps (number of latent variables taken as predictors) inboth methods.6 ConclusionWe have presented an implementation of the PLS recurrence for DB regres-sion. The fact that it does not need explicit matrix decompositions or eigen-9value extraction is a crucial property that enables the method to handlemoderately large datasets, such as those found in automobile insurance pre-diction.AcknowledgmentsWork supported in part by the Spanish Ministerio de Ciencia y Tecnolog´ıaand FEDER grant MTM2005-02370.ReferencesBoj, E., Claramunt, M. M., Fortiana, J. & Vidiella, A. (2002), ‘The use ofdistance-based regression and generalized linear models in the rate-makingprocess’. Mathematics Preprint Series, Institut de Matema`tica de la Uni-versitat de Barcelona, Num. 305.Boj, E., Claramunt, M. M. & Fortiana, J. (2004), Ana´lisis multivarianteaplicado a la seleccio´n de factores de riesgo en la tarificacio´n. Madrid:Fundacio´n MAPFRE Estudios. Cuadernos de la Fundacio´n MAPFRE Es-tudios, Num. 88Boj, E., Claramunt, M. M., Fortiana, J. & Vegas, A. (2005), ‘Bases de datosy estad´ısticas del seguro de automo´viles en Espan˜a: influencia en el ca´lculode primas’, Estad´ıstica Espan˜ola 47, 539–566.Borg, I. & Groenen, P. J. F. (2005), Modern Multidimensional Scaling: The-ory and Applications. 2nd Edition, Springer-Verlag, New York.Cuadras, C. M. (1989), ‘Distance Analysis in discrimination and classificationusing both continuous and categorical variables’, in Yadolah Dodge (ed.),’Statistical Data Analysis and Inference’, North-Holland Publishing Co.,Amsterdam, pp. 459–473.Cuadras, C. M. (1993), ‘Intepreting and inequality in multiple regression’,The American Statistician 47, 256–258.Cuadras, C. M. (1998), ‘Some cautionary notes on the use of principal compo-nents regression (Revisited)’, The American Statistician 52, 371 (Lettersto the Editor).Cuadras, C. M. & Arenas, C. (1990), ‘A Distance Based Regression Modelfor Prediction with Mixed Data’, Communications in Statistics A. Theoryand Methods 19, 2261–2279.10Cuadras, C. M., Arenas, C. & Fortiana, J. (1996), ‘Some computationalaspects of a Distance-Based model for Prediction’, Communications inStatistics B. Simulation and Computation 25, 593–609.Gower, J. C. (1971), ‘A general coefficient of similarity and some of its prop-erties’, Biometrics 27, 857–874.Hadi, A. S. & Ling, R. F. (1998), ‘Some cautionary notes on the use ofprincipal components regression’, The American Statistician 52, 15–19.Helland, I. S. (1988), ‘On the structure of partial least squares regression’,Communications in Statistics - Simulation and Computation 17, 581–607.Hoskuldsson, A. (1988), ‘PLS Regression Methods’ Journal of Chemometrics2, 211–228.Jolliffe, I. T. (2002), Principal Component Analysis. 2nd Edition, Springer-Verlag, New York.

A Distance Based Regression Model for Prediction with Mixed Data’,

A general coefficient of similarity and some of its properties’,

Ana´lisis multivariante aplicado a la seleccio´n de factores de riesgo en la tarificacio´n. Madrid: Fundacio´n MAPFRE Estudios. Cuadernos de la Fundacio´n

Distance Analysis in discrimination and classification using both continuous and categorical variables’,

Intepreting and inequality in multiple regression’,

On the structure of partial least squares regression’,

Principal Component Analysis. 2nd Edition,

Some cautionary notes on the use of principal components regression (Revisited)’,

Some cautionary notes on the use of principal components regression’,

Some computational aspects of a Distance-Based model for Prediction’,

The use of distance-based regression and generalized linear models in the rate-making process’.

Implementing PLS for distance-based regression: computational issues

Abstract

Similar works

Full text

Available Versions

Name not available

Name not available

Universidad Carlos III de Madrid e-Archivo