The merits of linear decision fusion in multiple learner systems have been widely accepted, and their practical applications are rich in literature. In this paper we present a new linear decision fusion strategy named Bagging.LMS, which takes advantage of the least-mean-square (LMS) algorithm to update the fusion parameters in the Bagging ensemble systems. In the regression experiments on four synthetic and two benchmark data sets, we compared this method with the Bagging-based Simple Average and Adaptive Mixture of Experts ensemble methods. The empirical results show that the Bagging.LMS method may significantly reduce the regression errors versus the other two types of Bagging ensembles, which indicates the superiority of the suggested Bagging.LMS method

Ng, S. C.

Wang, C.

Wu, Y. F.

吴云峰

Xiamen University Institutional Repository

Bagging.LMS: A Bagging-based Linear Fusionwith Least-Mean-Square Error Update forRegressionYunfeng Wu', Cong Wang', and S.C. Ng2'School of Information Engineering, Beijing University of Posts and Telecommunications,Xi Tu Cheng Road 10, Haidian District, Beijing 100876, China2School of Science and Technology, The Open University of Hong Kong,30 Good Shepherd Street, Homantin, Kowloon, Hong KongAbstract-The merits of linear decision fusion in multiplelearner systems have been widely accepted, and their practicalapplications are rich in literature. In this paper we present anew linear decision fusion strategy named Bagging.LMS,which takes advantage of the least-mean-square (LMS)algorithm to update the fusion parameters in the Baggingensemble systems. In the regression experiments on foursynthetic and two benchmark data sets, we compared thismethod with the Bagging-based Simple Average and AdaptiveMixture of Experts ensemble methods. The empirical resultsshow that the Bagging.LMS method may significantly reducethe regression errors versus the other two types of Baggingensembles, which indicates the superiority of the suggestedBagging.LMS method.I. INTRODUCTIONAn ensemble is a multi-learner system in which a series ofcomponent learners are generated to solve the same task [1].The component learners are combined with a certain fusionstrategy to form the final prediction [2], [3]. In other words,the ensemble fuses the knowledge acquired by local learnersto make a consensus decision which is supposed to besuperior to the one attained by any individual learnerworking solely. The most popular ensemble algorithms areBoosting [4] and Bagging [5], which have already beenwidely used to improve accuracy for solving classificationand regression problems [6].Boosting works by repeatedly implementing a given weaklearning machine on different distributed training data sets,and then fusing their outputs. The distribution of the trainingdata in the current iteration depends on the performance ofthe prior learner in the previous iteration. The first versionof the Boosting algorithm described by Schapire is Boostingby filtering [7], which involves filtering the training data bydifferent versions of a weak learning algorithm. However, itrequires a large size of training data, which is not feasible inmany practical applications [8]. In order to overcome such adrawback, Freund and Schapire proposed AdaBoost [4] tofind a typical mapping function or hypothesis with a lowerror rate relative to a given probability distribution of thetraining data. For regression problems, Freund and Schapiredeveloped AdaBoost.R [9] which is effective by projectingthe regression data into some classification sets. In spite oftheir effectiveness, the Boosting algorithms still containsome pitfalls. First, they have to expand each regressiondata set into many classification sets and the number ofprojected classification examples grows intensively largeafter a few boosted iterations. Second, the loss functionchanges from iteration to iteration and even differs betweenexamples during the same iteration. In addition, theBoosting algorithms make use of the training errors to adjustthe fusion weights despite the fact that the training errors areoften highly biased [10].On the other hand, Bagging introduces the bootstrapsampling technique [11] into the procedure of constructingcomponent learners, and expects to generate enoughindependent variance among them [5]. The bias of theBagging ensemble would converge by averaging while thevariance gets much smaller than that of each componentlearner.Recently, linear decision fusion strategies [3], [12], [13],[14], or linear combination rules are frequently used inAdaBoost, Bagging, and other ensemble learning methods,such as Random Forests [15]. In general, they can becategorized into two styles: Fixed and Trained rules [16].The fixed rules, such as Simple Average (SA) [17], containthe fixed fusion parameters. On the contrary, the trainedrules, e.g. Weighted Average (WA) [18] and AdaptiveMixture of Experts (AME) [19], require a learning processto initialize and update the fusion coefficients. In this paper,we introduce a new linear fusion strategy taking advantageof the Least-Mean-Square (LMS) algorithm for Bagging,called Bagging.LMS. And the motivation is to incorporatethe merits of Bagging and the linear decision fusion strategyin order to improve accuracy in regression problems.The rest of this paper is organized as follows. Section 2presents the details of the LMS error update algorithm forthe Bagging ensembles. Section 3 describes the numericalexperiments of regression, and compares the empiricalresults among the Bagging-based SA, AME, and LMSstrategies. In the end, the advantages of the Bagging.LMSand the future work are summarized in Section 4.II. LINEAR FUSION WITH THE LMS ERROR UPDATE FORBAGGINGThe goal of regression is to get a statistical model of thefunctional relationship between one dependent variable yand the multivariate input x = [x,..., XN ]T . The regressionmodel can be expressed by a function plus a residual £,which is similar to a noise, i.e.y = r(x)±e. (1)1-4244-0549-1/06/$20.00 (2006 IEEE.In other words, the task of regression can be regarded tofind out the linear or nonlinear mapping throughout a set ofinput-response pairs S {(x, y)} (X E yNxPI Y.E SxP)The Bagging method contains a bootstrap samplingprocedure which generates a series of "bagged" sets byresampling with replacement from the original data set. The"bagged" sets sk, k =1, ,K, also of size P, are used fortraining the K component learners in the ensemble. Thesecomponent learners are linearly fused to form the finalprediction f(x), i.e.f(x) = aTo(X) I (2)where vector a = [a0o, a1, , aK ]T represents the fusioncoefficients correspondingly allocated to the matrixo(x) = [bias,o1 (x), (X)]T' ( Okl(X)E ,) whichincludes the fusion bias parameter and the outputs ofcomponent learners. Particularly, ao is fixed at +1 andallocated to the bias. According to the LMS error criterion,the instantaneous cost function in the i-th update iteration isC(x') = e(x') /2. (3)where e(x') is the instantaneous estimate error, i.e.e(x') = y' f(x') = y' _ a&o(xi) (4)Differentiating (3) with respect to the fusion coefficientsyields the gradient approximation of C(x'), i.e.V C(x' ) I 8e(x' )22 Oa8 )2 _ T2y'o(xi) + aiT (o(Xi )o(x'' ) i']2~Oa-y'o(x') + (&"o(x' ) o(x')_ (Yi _"aiTO(xi))o(x') = -e(x')o(xi) (5)Hence the fusion coefficients update rule can be writtenby following the steepest descent gradient algorithm asa' al +,u[-VC(x')] = a' + ,ue(x')o(x') I (6)where the positive parameter ,c represents the magnitude ofthe update rate for the fusion coefficients in the negativegradient direction.III. EXPERIMENTS AND RESULTSFor the regression performance evaluation of theTABLE II THE AVERAGED MSE OF BAGGING ENSEMBLES WITH THREEDIFFERENT TYPES OF FUSION STRATEGIESMean Squared Error (MSE) of differentData Sets Fusion Strategies for BaggingSA AME LMSZigzag 0.0354 0.0372 0.0275Rhythm 0.0035 0.0036 0.0022Polynomial 0.0164 0.0142 0.01282-D Mexican Hat 0.0027 0.0026 0.0008Friedman#1 0.2520 0.2545 0.2394Friedman#3 0.0132 0.0411 0.0065Bagging.LMS, four synthetic and two benchmark data setsare selected in our experiments. The details of the syntheticdata sets are tabulated in Table 1. The 2-D Mexican Hat hasbeen used by Stitson et al. [20] for investigating theclassification performance of support vector machines. Twobenchmark data sets are the Friedman #1 and Friedman #3,which have ever been used by Briedman [5] for testing theregression performance of Bagging.For comparison purpose, two frequently used fusionstrategies: Simple Average (SA) and Adaptive Mixture ofExperts (AME) are also applied in the experiments for theBagging ensembles (with the abbreviation of Bagging.SAand Bagging.AME, respectively). The component learnersare three Multilayer Perceptrons (MLPs) independentlytrained by the scaled conjugate gradient [8], Levenberg-Marquardt [21], and resilient backpropagation [22]algorithms, respectively. Each MLP network contains onehidden layer which consists of five hidden nodes. We didn'toptimize the MLP architecture because in this investigationwe only focus on the relative regression performancebetween the Bagging ensembles instead of the absoluteperformance of a MLP. We repeated total 10 independenttrails for each fusion strategy in order to obtain the statisticalsignificant results.Fig. 1 plots the two-dimensional predictions (dotted lines)of the three types of Bagging ensembles compared with thefour synthetic regression responses (solid line). It can beobserved that the Bagging.LMS has a better performance ofcurve fitting than either the Bagging.SA or Bagging.AME.Table 2 gives the quantitative results in terms of meansquared error (MSE) averaged on the 10 independent trails.It can also be noted that the Bagging.SA outperforms theBagging.AME for some data sets, in particular, Zigzag,Friedman #1, and Friedman #3, but the distinctions forTABLE I DESCRIPTIONS OF THE SYNTHETIC REGRESSION DATA SETSData Sets Function Expression Variables SizeZigzag y = sin 2 cos x2 -0.25x +8 xe U[O, 3] 1008 N(0, 0.05)Rhythm y=F(mod(x,1) -5)3 + xe U[0, 20] 100Rhythm~~~~~ L~8 jc8- N(0, 0.01)10Polynomial y= 1+ 2x + 3X2 + 4x3 + 5x4 ± N(O, ]) 100asin |x x e U[-2T,2T] 1002-DMexicanHat y= 8 -N(0,0. 1)0.50.40.3Expected Curve* Bagging.SABagging.AMEBagging.LMSAA0.14 0-0.1Expected Curve A ABagging.SABagging.AMEBagging.LMS0.5 1.0 1.5 2.0 2.5 3.0(a) ZigzagExpected CurveBagging.SABagging.AMEBagging.LMSA-,A kI, v-III /d,/..--'-fo.-0.2 --0.30 5 10 15 20(b) RhythmA .A-4A.1A,.m a0.8 ARw *A-A *E-k0 00 0.2 0.4 0.6 0.80.60.40.20\-0.2&-0.2 A-A-Expected Curve. Bagging.SA/ Bagging.AMEA Bagging.LMS-6 -4 -2 0(c) Polynomial (d) 2-D Mexican HatFig. 1. Predictions of the Bagging ensembles with different decision fusion strategies for four synthetic regression data sets.Rhythm, 2-D Mexican Hat are not much. On the other side,the Bagging.LMS achieves significantly higher predictionaccuracy, especially for Zigzag, 2-D Mexican Hat, andFriedman #3, versus the Bagging.SA (0.0079, 0.0019, and0.0067 higher, respectively) or Bagging.AME (0.0097,0.0018, and 0.0346 higher, respectively). Fig. 2 gives box-and-whisker regression plots of three Bagging ensembles interms of averaged MSE values with the interquartile range.It is clear that the Bagging.LMS consistently outperformsthe other two fusion strategies on the total six regressiondata sets.IV. CONCLUSIONThe empirical results show that the Bagging ensemblewith the LMS decision fusion strategy does consistentlyimprove the prediction accuracy versus the previous fusionstrategies for regression, which indicates that this ensemblemethod may also be promising for the design of multipleclassifier systems. The further analysis of classificationperformance of the Bagging.LMS and more experimentalcomparisons with other trained fusion rules would be thenext step of our work.ACKNOWLEDGMENTThis work was supported in part by National ScienceFoundation of China under the Grant No. 60575034, and bythe 2005 Innovation Research Funds from Graduate School,Beijing University of Posts and Telecommunications.REFERENCES[1] L.K. Hansen, P. Salamon, "Neural network ensembles,' IEEE Trans.Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993-1001, 1990.[2] J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, "On combiningclassifiers," IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 20, no. 3, pp. 226-239, Mar 1998.[3] Y.F. Wu, J. He, Y. Man, and J.I. Arribas, "Neural network fusionstrategies for identifying breast masses," in Proc. 2004 Int'l JointConf. Neural Networks (IJCNN'04), Budapest, Hungary, 2004, vol. 3,pp. 2437-2442.[4] Y. Freund and R.E. Schapire, "Experiment with a new boostingalgorithm," Proc. 13th Int'l Conf. on Machine Learning (ICML'96),Bari, Italy, 1996, pp. 148-156.[5] L. Breiman, "Bagging predictors," Machine Learning, vol. 24, no. 2,pp. 123-140, 1996.[6] D. Opitz and R. Maclin, "Popular ensemble methods: An empiricalstudy," Journal ofArtificial Intelligence Research, vol. 11, pp. 169-198, 1999.[7] R.E. Schapire, "The strength of weak learnability," Machine Learning,vol. 5, no. 2, pp. 197-227, 1990.[8] S. Haykin, Neural Networks. A Comprehensive Foundation, 2nd ed.,Englewood Cliffs, NJ: Prentice Hall PTR, 1998.[9] Y. Freund and R.E. Schapire, "A decision-theoretic generalization ofon-line learning and an application to boosting," Journal ofComputerand System Sciences, vol. 55, pp. 119-139, 1997.[10] G. Ridgeway, "The state of boosting," Computing Science andStatistics, vol. 31, pp. 172-181, 1999.[11] B. Efron and R. Tibshirani, An Introduction to the Bootstrap, NewYork, NY: Chapman and Hall, 1993.0.40.2-0.2-0.4-0.6-0.8-1-1.2016141210420.A\A6I'i1,A1.itI,[12] K. Tumer and J. Ghosh, "Analysis of decision boundaries in linearlycombined neural classifiers," Pattern Recognition, vol. 29, no. 2, pp.341-348, 1996.[13] L.I. Kuncheva, "A theoretical study on six classifer fusion strategies,"IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 2,pp. 281-286, 2002.[14] Y.F. Wu, J.M. Zhang, C. Wang, and S.C. Ng, "Linear decisionfusions in multilayer perceptrons for breast cancer diagnosis," in Proc.17th IEEE Int'l Conf Tools with Artificial Intelligence (ICTAI'05),Hong Kong, 2005, pp. 699-700.[15] L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp.5-32, 2001.[16] F. Roli, G. Fumera, and J. Kittler, "Fixed and trained combiners forfusion of unbalanced pattern classifiers," in Proc. 5th Int'l ConfInformation Fusion (IF'02), Annapolis, MD, USA, 2002, vol. 1, pp.278-284.[17] Y.F. Wu and J.I. Arribas, "Fusing output information in neuralnetworks: Ensemble performs better," in Proc. 25th IEEE EMBS0.050.0450.040.0350 0.030.0250.020.0150.013 5 -102.5Lu 21.5x 10-54.5-43.5u) 3-2.521.5SA AME LMS(a)0.5SA AME LMSAnnu. Int'l Conf (EMBC'03), Cancun, Mexico, 2003, vol. 3, pp.2265-2268.[18] G. Fumera and F. Roli, "A theoretical and experimental analysis oflinear combiners for multiple classifier systems," IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 27, no. 6, pp. 942-956, 2005.[19] R.A. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton, "Adaptivemixtures of local experts," Neural Computation, vol. 3, no. 11, pp.79-87, 1991.[20] M.O. Stitson, J.A.E. Weston, A. Gammerman, V. Vovk, and V.Vapnik, "Experiments with support vector machines," RoyalHolloway College, University of London, London, UK, TechnicalReport: CSD-TR-96-19, 1996.[21] M.T. Hagan and M. Menhaj, "Training feedforward networks withthe Marquardt algorithm," IEEE Trans. Neural Networks, vol. 5, no. 6,pp. 989-993, 1994.[22] M. Riedmiller and H. Braun, "A direct adaptive method for fasterbackpropagation learning: The RPROP algorithm," in Proc. 1993IEEE Int'l Conf: Neural Networks (ICNN'93), San Francisco, CA,USA, 1993, vol. 1, pp. 586-591.0.0180.0170.0160.0150o0140.0130.0120.0110.01SA AME LMS(b)0.30.280.26Lu 0.240.220.20.18SA AME LMS(c)0.060.050.04u 0.030.020.010SA AME LMS SA AME LMS(d) (e) (f)Fig. 2. Performance comparison on regression sets (a) Zigzag, (b) Rhythm, (c) Polynomial, (d) 2-D Mexican Hat, (e) Friedman #1, and (f) Friedman #3.

BaggingLMS: A bagging-based linear fusion with least-mean-square error update for regression

http://dspace.xmu.edu.cn/bitstream/handle/2288/70760/04142174%5b1%5d.pdf?sequence=1&isAllowed=y

BaggingLMS: A bagging-based linear fusion with least-mean-square error update for regression

Abstract

Similar works

Full text

Available Versions

Xiamen University Institutional Repository