The two most popular classification tree algorithms in machine learning and statistics - C4.5 and CART - are compared in a benchmark experiment together with two other more recent constant-fit tree learners from the statistics literature (QUEST, conditional inference trees). The study assesses both misclassification error and model complexity on bootstrap replications of 18 different benchmark datasets. It is carried out in the R system for statistical computing, made possible by means of the RWeka package which interfaces R to the open-source machine learning toolbox Weka. Both algorithms are found to be competitive in terms of misclassification error - with the performance difference clearly varying across data sets. However, C4.5 tends to grow larger and thus more complex trees. (author's abstract)Series: Research Report Series / Department of Statistics and Mathematic

Hornik, Kurt

Meyer, David

Schauerhuber, Michael

Zeileis, Achim

English

Elektronische Publikationen der Wirtschaftsuniversität Wien

ePubWU Institutional RepositoryMichael Schauerhuber and Achim Zeileis and David Meyer and Kurt HornikBenchmarking Open-Source Tree Learners in R/RWekaWorking PaperOriginal Citation:Schauerhuber, Michael and Zeileis, Achim and Meyer, David and Hornik, Kurt (2007) BenchmarkingOpen-Source Tree Learners in R/RWeka. Research Report Series / Department of Statistics andMathematics, 54. Department of Statistics and Mathematics, WU Vienna University of Economicsand Business, Vienna.This version is available at: http://epub.wu.ac.at/1496/Available in ePubWU: May 2007ePubWU, the institutional repository of the WU Vienna University of Economics and Business, isprovided by the University Library and the IT-Services. The aim is to enable open access to thescholarly output of the WU.http://epub.wu.ac.at/Benchmarking Open-Source TreeLearners in R/RWekaMichael Schauerhuber, Achim Zeileis, David Meyer, Kurt HornikDepartment of Statistics and MathematicsWirtschaftsuniversität WienResearch Report SeriesReport 54May 2007http://statmath.wu-wien.ac.at/Benchmarking Open-Source Tree Learners inR/RWekaMichael Schauerhuber Achim Zeileis David Meyer Kurt HornikAbstractThe two most popular classification tree algorithms in machine learning and statistics—C4.5 and CART—are compared in a benchmark experiment together with two other morerecent constant-fit tree learners from the statistics literature (QUEST, conditional inferencetrees). The study assesses both misclassification error and model complexity on bootstrapreplications of 18 different benchmark datasets. It is carried out in the R system for statisticalcomputing, made possible by means of the RWeka package which interfaces R to the open-source machine learning toolbox Weka. Both algorithms are found to be competitive in termsof misclassification error—with the performance difference clearly varying across data sets.However, C4.5 tends to grow larger and thus more complex trees.1 IntroductionDue to their intuitive interpret-ability, tree-based learners are a popular tool in data mining forsolving classification and regression problems. Traditionally, practitioners with a machine learn-ing background use the C4.5 algorithm (Quinlan, 1992) while statisticians prefer CART (Breiman,Friedman, Olshen and Stone, 1984). One important reason for this is that free reference imple-mentations have not been easily available within an integrated computing environment. RPart,an open-source implementation of CART, has been available for some time in the S/R packagerpart (Therneau and Atkinson, 1997) while the open-source implementation J4.8 for C4.5 be-came available more recently in the Weka machine learning package (Witten and Frank, 2005)and is now accessible from within R by means of the RWeka package (Hornik, Zeileis, Hothornand Buchta, 2007). With these software tools available, the algorithms can be easily comparedand bench-marked on the same computing platform: the R system for statistical computing (RDevelopment Core Team 2006). We carry out such a benchmark comparison, including differentstrategies for hyper-parameter tuning as well as two further constant-fit tree models—QUEST(Loh and Shih, 1997) and conditional inference trees (Hothorn, Hornik and Zeileis, 2006). Thelearners are compared with respect to misclassification error and model complexity on each of18 different benchmarking data sets by means of simultaneous confidence intervals (adjusted formultiple testing). Across data sets, the performance is aggregated by consensus rankings. Theremainder of the paper is structured as follows: In Section 2, a general overview of the benchmarkdesign is provided before Section 3 presents the findings from our experiments. A discussion inSection 4 concludes the paper.2 Design of the Benchmark ExperimentThe simulation study includes a total of six tree-based methods for classification. All learnerswere trained and tested in the framework of Hothorn, Leisch, Zeileis and Hornik (2005) basedon 500 bootstrap samples for each of 18 data sets. All algorithms are trained on each bootstrapsample and evaluated on the remaining out-of-bag observations. Misclassification rates are usedas predictive performance measures, while model complexity requirements of the algorithms understudy are measured by the number of estimated parameters (number of splits plus number of1leafs). Performance and model complexity distributions are assessed for each algorithm on eachof the datasets. In our setting, this results in 108 performance distributions (6 algorithms on 18data sets), each of size 500. For comparison on each individual data set, simultaneous pairwiseconfidence intervals (Tukey all-pair comparisons confidence intervals which are set up by functionsfrom the R package multcomp (Hothorn, Bretz, and Westfall, 2007)) are used. For aggregatingthe pairwise dominance relations across data sets, median linear order consensus rankings areemployed following Hornik and Meyer (2007). A brief description of the algorithms and theircorresponding implementation is given below.CART/RPart: Classification and regression trees (CART, Breiman et al., 1984) is the classicalrecursive partitioning algorithm which is still the most widely used in the statistics commu-nity. Here, we employ the open-source reference implementation of Therneau and Atkinson(1997) provided in the R package rpart. For determining the tree size, cost-complexity prun-ing is typically adopted: either by using a 0- or 1-standard-errors rule. The former choosesthe complexity parameter associated with the smallest prediction error in cross-validation(RPart0), whereas the latter chooses the highest complexity parameter which is within 1standard error of the best solution (RPart1).C4.5/J4.8: C4.5 (Quinlan, 1993) is the predominantly used decision tree algorithm in the machinelearning community. Although source code implementing C4.5 is available in Quinlan (1993),it is not published under an open-source license. Therefore, the Java implementation of C4.5(revision 8), called J4.8, in Weka is the de-facto open-source reference implementation. Fordetermining the tree size, a heuristic confidence threshold C is typically used which is bydefault set to C = 0.25 (as recommended in Witten and Frank, 2005). To evaluate theinfluence of this parameter, we compare the default J4.8 algorithm with a tuned versionwhere C and the minimal leaf size M (default: M = 2) are chosen by cross-validation(J4.8(cv)). A full grid search for C = 0.01, 0.05, 0.1, . . . , 0.5 and M = 2, 3, . . . , 10, 15, 20 isused in the cross-validation.QUEST: Quick, unbiased and efficient statistical trees are a class of decision trees suggestedby Loh and Shih (1997) in the statistical literature. QUEST popularized the concept ofunbiased recursive partitioning, i.e., avoiding the variable selection bias of exhaustive searchalgorithms (such as CART and C4.5). A binary implementation is available from http://www.stat.wisc.edu/~loh/quest.html and interfaced in the R package LohTools whichis available from the authors upon request.CTree: Conditional inference trees (Hothorn et al., 2006) are a framework of unbiased recursivepartitioning based on permutation tests (i.e., conditional inference) and applicable to inputsand outputs measured at arbitrary scale. An open-source implementation is provided in theR package party.Similar to Meyer et al. (2003) datasets from the popular UCI Repository of Machine LearningDatabases (Newman, Hettich, Blake and Merz, 1998) were used for benchmarking. The datasetsare also provided in the R package mlbench (Leisch and Dimitriadou, 2006). An overview is givenin Table 1.3 Results of the Benchmark Experiment3.1 Results on Individual Datasets: Pairwise Confidence IntervalsHere, we exemplify—using the well-known Pima Indians diabetes and breast cancer data sets—how the tree algorithms are assessed on a single data set. Simultaneous confidence intervals arecomputed for all 15 pairwise comparisons of the 6 learners. The resulting dominance relations areused as the input for the aggregation analyses in Section 3.2.2Table 1: Artificial [∗] and non artificial benchmarking data setsData set # of obs. # of cat. inputs # of num. inputsbreast cancer 699 9 -chess 3196 36 -circle ∗ 1000 - 2credit 690 - 24heart 303 8 5hepatitis 155 13 6house votes 84 435 16 -ionosphere 351 1 32liver 345 - 6Pima Indians diabetes 768 - 8promotergene 106 57 -ringnorm ∗ 1000 - 20sonar 208 - 60spirals ∗ 1000 - 2threenorm ∗ 1000 - 20tictactoe 958 9 -titanic 2201 3 -twonorm ∗ 1000 - 20As can be seen from the performance plots for Pima Indian diabetes in Figure 1, standardJ4.8 is outperformed (in terms of misclassification as well as model complexity) by the othertree learners. All other algorithm comparisons indicate equal predictive performances, except forthe comparison of RPart0 and J4.8(cv), where the former learner performs slightly better thanthe latter. On this particular dataset tuning enhances the predictive performance of J4.8, whilethe misclassification rates of the differently tuned RPart versions are not subject to significantchanges. In terms of model complexity J4.8(cv) produces larger trees than the other (statistical)learners. Looking at the breast cancer data yields a rather different picture: Both RPart versionsare outperformed by J4.8 or its tuned alternative in terms of predictive accuracy. Similar to PimaIndians diabetes, J4.8 and J4.8(cv) tend to build significantly larger trees than RPart. On thisdataset, CTree has a slight advantage over all other algorithms except J4.8 in terms of predictiveaccuracy. For J4.8 as well as RPart, tuning does not promise to increase predictive accuracysignificantly. A closer look at the differing behavior of J4.8(cv) under cross validation for bothdata sets is provided in Figure 2 and further illustrates that the behavior of the algorithms heavilydepends on the type and structure of the data problems at hand. The multiple inference resultsgained from all datasets considered in this simulation experiment (just like the results derivedfrom the two datasets above), form the basis on which further aggregation analyses of Section 3.2are built upon.3.2 Results Across Data Sets: Consensus RankingsHaving 18× 6 = 108 performance distributions of the 6 different learners applied to 18 bootstrapdata settings at hand, aggregation methods can do a great favor to allow for summarizing andcomparing algorithmic performance. The underlying dominance relations derived from the mul-tiple testing are summarized by simple sums in Table 2 and by the corresponding median linearorder rankings in Table 3. In Table 2, rows refer to winners, while columns denote the losers. Forexample J4.8 managed to outperform QUEST on 11 datasets and 4 times vice versa, i.e., on theremaining 3 datasets, J4.8 and QUEST perform equally well.The median linear order for misclassification reported in Table 3 suggests that tuning of J4.8instead of using the heuristic approach is worth the effort. A similar conclusion can be made for3Pima Indians Diabetes: Misclassification−2.5 −1.5 −0.5 0.0 0.5CTree − QUESTCTree − RPart1QUEST − RPart1CTree − RPart0QUEST − RPart0RPart1 − RPart0CTree − J4.8(cv)QUEST − J4.8(cv)RPart1 − J4.8(cv)RPart0 − J4.8(cv)CTree − J4.8QUEST − J4.8RPart1 − J4.8RPart0 − J4.8J4.8(cv) − J4.8 ((((((((((((((()))))))))))))))lllllllllllllllMisclassification difference (in percent)Pima Indians Diabetes: Complexity−80 −60 −40 −20 0 20CTree − QUESTCTree − RPart1QUEST − RPart1CTree − RPart0QUEST − RPart0RPart1 − RPart0CTree − J4.8(cv)QUEST − J4.8(cv)RPart1 − J4.8(cv)RPart0 − J4.8(cv)CTree − J4.8QUEST − J4.8RPart1 − J4.8RPart0 − J4.8J4.8(cv) − J4.8 ((((((((((((((()))))))))))))))lllllllllllllllComplexity differenceBreast Cancer: Misclassification−1.0 −0.5 0.0 0.5 1.0CTree − QUESTCTree − RPart1QUEST − RPart1CTree − RPart0QUEST − RPart0RPart1 − RPart0CTree − J4.8(cv)QUEST − J4.8(cv)RPart1 − J4.8(cv)RPart0 − J4.8(cv)CTree − J4.8QUEST − J4.8RPart1 − J4.8RPart0 − J4.8J4.8(cv) − J4.8 ((((((((((((((()))))))))))))))lllllllllllllllMisclassification difference (in percent)Breast Cancer: Complexity−15 −10 −5 0 5 10CTree − QUESTCTree − RPart1QUEST − RPart1CTree − RPart0QUEST − RPart0RPart1 − RPart0CTree − J4.8(cv)QUEST − J4.8(cv)RPart1 − J4.8(cv)RPart0 − J4.8(cv)CTree − J4.8QUEST − J4.8RPart1 − J4.8RPart0 − J4.8J4.8(cv) − J4.8 ((((((((((((((()))))))))))))))lllllllllllllllComplexity differenceFigure 1: Simultaneous confidence intervals of pairwise performance differences (left: misclassifi-cation, right: complexity) for Pima Indians diabetes (top) and breast cancer (bottom) data.Figure 2: Distribution of J4.8(cv) parameters obtained through cross validation on Pima Indiansdiabetes and breast cancer data sets.4Table 2: Summary of predictive performance dominance relations across all 18 datasets based onmisclassification rates and model complexity.Misclassification J4.8 J4.8(cv) RPart0 RPart1 QUEST CTree∑J4.8 0 2 9 9 11 8 39J4.8(cv) 4 0 8 9 11 9 41RPart0 5 6 0 7 10 7 35RPart1 6 4 1 0 8 6 25QUEST 4 2 2 5 0 7 20CTree 7 6 7 8 9 0 37∑26 20 27 38 49 37Complexity J4.8 J4.8(cv) RPart0 RPart1 QUEST CTree∑J4.8 0 1 0 0 2 0 3J4.8(cv) 17 0 0 0 5 3 25RPart0 18 18 0 0 13 15 64RPart1 18 18 16 0 14 15 81QUEST 15 13 5 4 0 10 47CTree 18 14 3 2 8 0 45∑86 64 24 6 42 43the RPart versions. Here, the median linear order suggests that the common one standard errorrule performs worse. For both cases, the underlying dominance relation figures of Table 2 catchour attention. Regarding the first case, J4.8(cv) only dominates J4.8 in four of six data settings,in which a significant test decision for performance differences could be made. In addition theremaining 12 data settings yield equivalent performances. Therefore superiority of J4.8(cv) aboveJ4.8 is questionable. In contrast the superiority of RPart0 vs. RPart1 seems to be more reliablebut still the number of data settings producing tied results is high. A comparison of the figures ofCTree and the RPart versions confirms previous findings (Hothorn et al., 2006) that CTree andRPart often perform equally well. The question concerning the dominance relation between J4.8and RPart cannot be answered easily: Overall, the median linear order suggests that the J4.8decision tree versions are superior to the RPart tree learners in terms of predictive performance.But still, looking at the underlying relations of the best performing versions of both algorithms(J4.8(cv) and RPart0) reveals that a confident decision concerning predictive superiority cannot bemade. The number of differences in favor of J4.8(cv) is only two and no significant differences arereported on four data settings. A brief look at the complexity ranking (Table 3) and the underlyingcomplexity dominance relations (Table 2, bottom) shows that J4.8 and its tuned version producemore complex trees than the RPart algorithms. While analogous analyses of comparing J4.8versions to CTree do not indicate confident predictive performance differences, superiority of theJ4.8 versions versus QUEST in terms of predictive accuracy is evident.To aggregate the tuning results from J4.8(cv), Figure 3 depicts the median C andM parametersTable 3: Median linear order consensus rankings for algorithm performanceMisclassification Complexity1 J4.8(cv) RPart12 J4.8 RPart03 RPart0 QUEST4 CTree CTree5 RPart1 J4.8(cv)6 QUEST J4.85ll lllllllllllllll0.0 0.1 0.2 0.3 0.4024681012medium confidence threshold (C)medium minimal leaf size (M)lFigure 3: Medians of the J4.8(cv) tuning parameter distributions for C and Mchosen for each of the 18 parameter distributions. It confirms the finding from the individualbreast cancer and Pima Indians diabetes results (see Figure 2) that the parameter chosen bycross-validation can be far off the default values for C and M .4 Discussion and Further WorkIn this paper we present results of a medium scale benchmark experiment with a focus on popularopen source tree-based learners available in R. The main objective of our simulation experimentis to compare performances of popular tree algorithms suggested in the statistical and machinelearning literature. Our main result is that no clear statement of superiority between J4.8 andRPart can be made in terms of predictive accuracy: depending on the particular data set onealgorithm might perform slightly better than the other. However, our results show that J4.8tends to build more complex trees than RPart. In addition to delivering information on overallalgorithmic performance only, we also present some results on tuning behavior for J4.8 which couldgive rise to further discussion on calibration issues (e.g., revision of parameter default settings)of various tree learning methods. Since tree-based learners are one of the most prominent toolsin data mining for solving classification and regression problems, a comprehensive performancebenchmark experiment could provide interesting results for practitioners as well as researchers.Therefore our next steps will be to include additional datasets as well as additional learningmethods for classification as well as regression tasks.ReferencesBreiman, L., Friedman, J., Olshen, R. and Stone, C. (1984): Classification and RegressionTrees. Wadsworth, Belmont, CA, 1984.Hornik, K., Meyer, D. (2007): Deriving Consensus Rankings from Benchmarking ExperimentsIn: Advances in Data Analysis (Proceedings of the 30th Annual Conference of the Gesellschaftfu¨r Klassifikation e.V., Freie Universita¨t Berlin, March 8–10, 2006), Decker, R., Lenz, H.-J.(eds.), Springer-Verlag, 163–170.6Hornik, K., Zeileis, A., Hothorn, T., Buchta, C. (2007): RWeka: An R Interface to Weka. Rpackage version 0.3-2. http://CRAN.R-project.org/.Hothorn, T., Bretz, F., Westfall, P. (2007): multcomp: Simultaneous Inference for GeneralLinear Hypotheses. R package version 0.991-9.Hothorn, T., Hornik, K. and Zeileis, A. (2006): Unbiased recursive partitioning: A conditionalinference framework. Journal of Computational and Graphical Statistics, 15(3), 651–674.Hothorn, T., Leisch, F., Zeileis, A. and Hornik, K. (2005): The design and analysis of bench-mark experiments. Journal of Computational and Graphical Statistics, 14(3), 675–699.Leisch, F., Dimitriadou, E. (2006): mlbench: Machine Learning Benchmark Problems. R pack-age version 1.1-2.Loh, W., Shih, Y. (1997): Split Selection Methods for Classification Trees. Statistica Sinica, 7,815–840.Meyer, D., Leisch, F., Hornik, K. (2003): The Support Vector Machine under Test. Neurocom-puting, 55, 169–186.Newman, D., Hettich, S., Blake, C. and Merz C. (1998): UCI repository of machine learningdatabases. http://www.ics.uci.edu/~mlearn/MLRepository.html.Quinlan, J. (1993): C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers,Inc., San Mateo, CA.R Development Core Team (2006): R: A Language and Environment for Statistical Computing.R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org/.Therneau, T., Atkinson, E. (1997): An introduction to recursive partitioning using the rpartroutine. Technical Report. Section of Biostatistics, Mayo Clinic, Rochester, http://www.mayo.edu/hsr/techrpt/61.pdf.Witten, I., Frank, E. (2005): Data Mining: Practical Machine Learning Tools and Techniques.Second Edition, Morgan Kaufmann Publishers, San Francisco.7

Benchmarking Open-Source Tree Learners in R/RWeka

Abstract

Similar works

Full text

Available Versions

Elektronische Publikationen der Wirtschaftsuniversität Wien