Cadima, Jorge

Cerdeira, Jorge Orestes

Duarte Silva, A. P.

Minhoto, Manuel

English

Repositório Institucional da Universidade Católica Portuguesa

Selecting variables in multivariate linear models with Subselect (1) FEG/CEGE – UNIV. CATÓLICA PORTUGUESA – C.R. PORTO(2) I.S. AGRONOMIA – UNIV. TÉCNICA DE LISBOA(3) DEP. MATEMÁTICA – UNIVERSIDADE DE ÉVORAA. PEDRO DUARTE SILVA(1)(*)JORGE CADIMA(2)MANUEL MINHOTO(3)JORGE ORESTES CERDEIRA(2)(*) Supported by: FEDER / POCI 2010The      Subselect PackageOVERVIEW1. GOALS AND ISSUES OF VARIABLE SELECTION 2. VARIABLE SELECTION IN LINEAR REGRESSION3. VARIABLE SELECTION IN LINEAR MODELSWITH MULTIPLE RESPONSES4. VARIABLE SELECTION IN GENERALIZEDLINEAR MODELS5. The        Subselect PACKAGE6. FINAL REMARKSThe      Subselect PackageGOALS OF VARIABLE SELECTION PARCIMONY / INTERPETABILITY  BETTER PREDICTIONS EFFICIENT ESTIMATION / INFERENCE ISSUES IN VARIABLE SELECTION EFFECTS OF SELECTION BIAS  CHOICE OF SELECTION CRITERIA  EFFICIENCY / EFECTIVNESS OF SELECTIONALGORITHMS The      Subselect PackageVARIABLE SELECTION IN LINEAR REGRESSION  RSS (OR R2) - BASED   SELECTION CRITERIA : MALLOWS CP   )( )1k(2nRSSC2k  (PRED.CRITERION)ADJUSTED R2  (DESCRIP.INDEX)AKAIKE ANDSCHWARZ(INFORMATION CRITERIA)k2n1knRSS2lnnAIC  )nln(kn1knRSS2lnn21BICy =  X b + e RSS = eT e 2y2s)1n(RSS1R2y2as)1kn(RSS1RThe      Subselect PackageVARIABLE SELECTION IN LINEAR REGRESSION  PROBLEMS OF SELECTION BIAS:1. PARAMETER ESTIMATES MAY BE OVERBLOWN 2. CLASSICAL INFERENCE METHODS ARE NOT VALID 3. GOODNESS OF FIT MEASURES ARE TOO OPTIMISTIC  REMEDIES1. SHRINKAGE METHODS 3. RESAMPLING AND/OR CROSS-VALIDATION2. MULTIPLE-TESTS INFERENCE The      Subselect PackageVARIABLE SELECTION IN LINEAR REGRESSION OTHER APPROACHES :    BAYES FACTORS AND BAYES AVERAGING   GARROTE, LASSO, ELASTIC NET AND RELATIVES GENERAL TO SPECIFIC MODELLINGComparison Criteria: Multivariate Indices21ccr1/rr1i2i2)ccr(11  τ ri 112112iccrrrccrr1i2i2ξ( max ccr12  max Roy 1 )( max 2  max Lawley-Hotelling trace )( max 2  min Wilks  ) ( max 2  max Bartlett-Pillai trace )The      Subselect PackageLINEAR MODELS WITH MULTIPLE RESPONSES)H(TEigvalccr1i2iY =  X B  +  UH = SXX - SXY SYY-1 SYXT =  SXXr =  min(dim(X),dim(Y))The      Subselect PackageA LINEAR HYPOTHESIS FRAMEWORK:X  =  A   +  U SELECT COLUMNS OF X IN ORDER TO EXPLAIN H1PARTICULAR CASES:H0:  C  =  0 LINEAR DISCRIMINANT ANALYSIS MULTI-WAY MANOVA/MANCOVA EFFECTSA = [1g ]  = [g] 1...01............0...11C)H(TEigvalccr1i2i T =  X’ (I - P) X H = X’ (P - P) X = R(A) r = dim() - dim()  = R(A)  N (C (ATA)- AT)Comparison Criteria:The      Subselect PackageGENERALIZED LINEAR MODELSg(y) =  X B  + u = [ X1 X2] [B1T B2T]T + u 22211211BBT2IˆIˆIˆIˆ)B()B()B(LlnIˆ )0,Bˆ(Lln)Bˆ(Lln2 R12121112122T2 Bˆ)IˆIˆIˆIˆ(Bˆ* )0,B~(Lln)Bˆ(Lln2** 1 21211111 BˆIˆIˆBˆB~ AIC = # + 2(k + m(k))(Likelihood ratio statistic)# =  , * OR **(Wald statistic)The Subselect PackageSearch routines for (combinatorial) criteria optimizationannealExact Algorithm:leaps - based on Furnival and Wilson´s leaps andbounds algorithm for linear regression- viable with up to 30 - 35 original variablesHeuristics:geneticimprove- simulated annealing- genetic algorithm- restricted local improvementThe      Subselect Packagefunctions of T-1 H eigenvalues Linear RegressionComparison criteriamax R2“ccr12”, “tau2”, “xi2” or “zeta2”The      Subselect Package(r=1)  Linear Models with Multiple ResponsesGeneralized Linear Models(r=1)  Wald statistic (*)maxminT, H symmetric rank(H)  = rSearch functions arguments :- Tuning parameters for heuristics- Maximum time allowed for exact search- Variables forcibly included or excluded in theselected subsets- Number of solutions by subset dimensionality- Numerical tolerance for detecting singular ornon-symmetrical matricesThe      Subselect Package- Matrices T and H- Comparison criterion (models with multiple responses)Auxiliary functions:lmHmat - creates H and T matrices for linearregression/canonical correlation analysisldaHmat - creates H and T matrices for lineardiscriminant analysisglhHmat - creates H and T matrices for ananalysis based on a linear hypothesis specified by the user The      Subselect PackageExpanding a Subselect Analysis- Searches across different dimensionalities canbe handled by post-processing the results ofsubselectThe      Subselect Package- Selection biases can be estimated byresampling or cross-validation procedures thatinclude the basic searches within larger cycles Other issues in variable selection can be tackled by combining subselect with the wider       system:- Relative importance of individual variables canbe accessed by collecting statistics on groups of“good” subsets of different dimensionalities Final Remarks- Subselect implements effective searches forvariable subsets in a wide range of modelsThe      Subselect Package- Random searches are usually good and muchbetter than greedy (“stepwise”) algorithms - Many subsets lead to similar criterion values.Finding the exact optimum may not be so important - Exact searches are viable if the number oforiginal variables is not much larger than 30- Subselect is most useful when used incombination with other utilities of the       system  ReferencesCadima J, Cerdeira JO and  Minhoto M (2004). Computational Aspects of Algorithms for Variable Selection in the Context of Principal Components. Computational Statistics and Data Analysis 47: 225-236.Duarte Silva, A.P. (2001). Efficient Variable Screening for Multivariate Analysis. Journal of Multivariate Analysis 76, 35-62.Furnival, G.M. & Wilson, R.W. (1974). Regressions by Leaps and Bounds. Technometrics 16: 499-511.Lawless, J.F and Singhal, K. (1978). Efficient Screening of NonNormal Regression Models. Biometrics. 34: 318-327.Lawless, J.F and Singhal, K. (1987). ISMOD: An All-Subsets Regression Program for Generalized Linear Models. I. Statistical and Computational Background. Computing Methods and Programs in Biomedecine 24: 125-134

Selecting variables in multivariate linear models with subselect

https://repositorio.ucp.pt/bitstream/10400.14/5285/1/Selecting%20variables%20in%20multivariate%20linear%20models%20with%20subselect%20%28Compstat%202006%29.pdf

Selecting variables in multivariate linear models with subselect

Abstract

Similar works

Full text

Available Versions

Repositório Institucional da Universidade Católica Portuguesa