Foley, Aoife M.

Sidorov, Denis N.

Zhukov, Aleksei V.

Concept drift has potential in smart grid analysis because the socio-economic behaviour of consumers is not governed by the laws of physics. Likewise there are also applications in wind power forecasting. In this paper we present decision tree ensemble classification method based on the Random Forest algorithm for concept drift. The weighted majority voting ensemble aggregation rule is employed based on the ideas of Accuracy Weighted Ensemble (AWE) method. Base learner weight in our case is computed for each sample evaluation using base learners accuracy and intrinsic proximity measure of Random Forest. Our algorithm exploits ensemble pruning as a forgetting strategy. We present results of empirical comparison of our method and other state-of-the-art concept-drift classifiers

Queen's University Belfast Research Portal

Random Forest Based Approach for Concept Drift HandlingZhukov, A. V., Sidorov, D. N., & Foley, A. M. (2017). Random Forest Based Approach for Concept DriftHandling. In Analysis of Images, Social Networks and Texts: 5th International Conference, AIST 2016,Yekaterinburg, Russia, April 7-9, 2016, Revised Selected Papers. (pp. 69-77). (Communications in Computerand Information Science; Vol. 661).Published in:Analysis of Images, Social Networks and Texts: 5th International Conference, AIST 2016, Yekaterinburg,Russia, April 7-9, 2016, Revised Selected PapersQueen's University Belfast - Research Portal:Link to publication record in Queen's University Belfast Research PortalPublisher rightsThis work is made available online in accordance with the publisher’s policies. Please refer to any applicable terms of use of the publisher.General rightsCopyright for the publications made accessible via the Queen's University Belfast Research Portal is retained by the author(s) and / or othercopyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associatedwith these rights.Take down policyThe Research Portal is Queen's institutional repository that provides access to Queen's research output. Every effort has been made toensure that content in the Research Portal does not infringe any person's rights, or applicable UK laws. If you discover content in theResearch Portal that you believe breaches copyright or violates any law, please contact openaccess@qub.ac.uk.Download date:06. Mar. 2017Random Forest Based Approach for ConceptDrift HandlingAleksei V. Zhukov1,2Denis N. Sidorov1,2 and Aoife M. Foley31 Energy Systems Institute SB RAS, Irkutsk, Russiadsidorov@isem.irk.ru,2 Irkutsk State University, Irkutsk, Russiazhukovalex13@gmail.com3 Queens University Belfast, Belfast, UKa.foley@qub.ac.ukAbstract. Concept drift has potential in smart grid analysis becausethe socio-economic behaviour of consumers is not governed by the lawsof physics. Likewise there are also applications in wind power forecast-ing. In this paper we present decision tree ensemble classification methodbased on the Random Forest algorithm for concept drift. The weightedmajority voting ensemble aggregation rule is employed based on the ideasof Accuracy Weighted Ensemble (AWE) method. Base learner weight inour case is computed for each sample evaluation using base learners accu-racy and intrinsic proximity measure of Random Forest. Our algorithmexploits ensemble pruning as a forgetting strategy. We present results ofempirical comparison of our method and other state-of-the-art concept-drfit classifiers.Keywords: machine learning · decision tree · concept drift · ensemblelearning · classification · random forest1 IntroductionEnsemble methods of classification (or, briefly, ensembles) employ various learn-ing algorithms to obtain better predictive accuracy comparing with individualclassifiers. Ensembles are much used in research and most of the papers is de-voted to stationary environments where the complete datasets are available forlearning classifiers and transfer functions of dynamical systems are not changingas time goes on. For real world applications (e.g. in power engineering [1], [2])learning algorithms are supposed to work in dynamic environments with datacontinuously generated in the form of a stream on not necessarily equally spacedtime intervals. Data stream processing commonly relies on single scans of thetraining data and implies restrictions on memory and time. Changes caused bydynamic environments (e.g. consumer behaviour in future smart grids) can becategorised into sudden or gradual concept drift subject to appearance of novelclasses in a stream and the rate of changing definitions of classes.One of the most generally effective ensemble classifier is a Random For-est. This algorithm employs bagging [3] and principles of the random subspacemethod [4] to build a highly decorrelated ensemble of decision trees [5]. Thereare some attempts [6] to adapt Random Forest to handling concept-drift, butthis approach is not fully discovered.The objective of this paper is to propose a novel approach to classificationto adapt to concept drifts. This novel classification method can be applied toconsumer behaviour in future smart grids to predict and classify the randombehaviour of humans when they interact with smart appliances in the home andcharging and discharging of electric vehicles. Likewise in these methods can beused to forecast wind power and indeed solar power.We propose to compute the base learner weight for samples evaluation usingbase learners accuracy and intrinsic proximity measure of random forest.It is to be noted that concept drifting is related to non-stationary dynami-cal systems modelling [7], where the completely supervised learning is employedusing a special set of input test signals. For more details concerning integraldynamical models theory and applications refer to monograph [8] and its bibli-ography.This paper is organised into five sections. In Section 2, a brief review ofensemble streaming classifiers based on random forest is given. Section 3 deliversdetailed presentation of proposed algorithm. In Section 4 experiments on bothmachine learning and tracking tasks are provided. Finally, the paper concludeswith Section 5 where the main results are discussed.2 Related WorkThe use of drifting concepts for huge datasets analysis is not unfamiliar to themachine learning and systems identification communities [9]. In this work werestrict ourselves to considering decision tree ensemble classification methodsonly.Let us briefly discuss methods most related to our proposal and employedin the experiments. For a more detailed overview of the results in this area,including online incremental ensembles, readers may refer to monograph [10]and review [11]. Bayesian logistic regression was used in [12] to handle driftingconcepts in terms of dynamical programming. Concept drifting handling is closeto methodology of on-line random forest algorithm [6] where ideas from on-line bagging, extremely randomised forests and on-line decision tree growingprocedure are employed. Accuracy Weighted Ensemble (AWE) approach wasproposed in [9]. The main idea is to train a new classifier on each incomingdataset and use it to evaluate all the existing classifiers in the ensemble. Weincorporate this idea in our approach.Analysis of ensemble methods with decision tree base learners is interestingtopic to be addressed in this paper. The recursive nature of decision trees makeson-line learning a difficult task due to the hard splitting rule, errors cannot becorrected further down the tree.3 Proximity Driven Streaming Random ForestIn paper [13] the authors clearly demonstrated that a classifier ensemble canoutperform a single classifier in the presence of concept drifts when the baseclassifiers of the ensemble are adherence weighted to the error similar to currenttesting samples. We propose another approach which exploits Random Forestproperties.To produce a novel classification algorithm capable of handling concept driftthe following questions need to be answered:– How to adapt original Random Forest for data streaming?– How to define the base classifier weighting function?– How to choose forgetting strategy?These questions are considered in the following subsections.3.1 Streaming classifier based on Random ForestMethodologically ensemble approaches allow concept-drift to be handled in thefollowing ways: base classifier adaptation, changing in training dataset (such asBootstrap [3] or RSM [4]), ensemble aggregation rule changing or changing instructure of an ensemble (pruning or growing). In this paper we propose Prox-imity Driven Streaming Random Forest (PDSRF) which exploit combinations ofthese approaches. Besides some methods are already incorporated to the originalRandom Forest. Contrary to conventional algorithms we use weighted majorityvoting as an aggregation rule of ensemble. This allows us to adapt the entireclassifier by changing the weights of the base learners. In order to obtain theclassifiers weight estimation we should store samples. For this purpose we usea sliding windows approach which is used in the periodically updated RandomForest [14]. The length of this window is fixed and can be estimated by cross-validation. For the sake of time and memory optimization Extremely randomizedtrees[15] is used as a base learner instead of original randomized trees.3.2 Base classifier weighting functionWe employ the assumption that the base classifiers make similar errors on sim-ilar samples even under concept-drift. Conventional Random Forest employ theso called proximity measure. It uses a tree structure to obtain similarity in thefollowing way: if two different sample are in the same terminal node, their prox-imity will be increased by one. At the end, proximities are normalised by dividingby the number of trees [5].Following the AWE approach proposed in [13] we use an error rate to produceweights (1) of classifiers, where E is an new block testing error for i-th classifier,ε is a small parameter.wi = 1/(E2 + ε) (1)3.3 Forgetting strategyOne of the main problems in concept-drifting learning is to select the properforgetting strategy and forgetting rate [11]. The classifier should be adaptiveenough to handle changes. In this case different strategies can be more appro-priate to different types of drift (for example, sudden and gradual drifts). In thispaper we focus on gradual changes only.In this paper we propose ensemble pruning technique to handle the concept-drift. This technique uses the classic replace-the-looser approach [11] to discardtrees with high error on new block samples.3.4 AlgorithmTo predict the sample we propose the following algorithm. First we use a storedwindow to find similar items using the specified similarity metric. Second weevaluate our current ensemble on similar examples. Then we compute weightsadherence to errors on k similar samples.On every chunk the algorithm tests all the trees to choose the poorest baselearner and replace it with new one trained on new block data. This process isiterative while the ensemble error on new block samples is higher than a specifiedthreshold.4 Experimental EvaluationIn this experiment, we evaluate our algorithm on publicly available datasetCoverType [16], and compared it to the most popular concept-drift classifiers.Our algorithm was implemented natively in C++ according to the same testingmethodology. With adherence to this methodology classication accuracy was cal-culated using the data block evaluation method, which exploits the test-then-trainapproach. The data block evaluation method reads incoming samples withoutprocessing them, until they form a data block of size d. Each new data block isfirst used to test the existing classifier, then it updates the classifier [17].Proposed proximity driven streaming random forest was tested with dif-ferent sizes of block, window, nearest neighbours number and ensemble size (Ta-ble 2). In order to make the results more interpretable we also test originalrandom forest with incorporated “replace-the-looser” forgetting.4.1 DatasetsWe follow the literature on adaptive ensemble classifiers and select the publiclyavailable benchmark datasets with concept drift. In this paper the proposedmethod was tested on the cover type dataset from Jock A. Blackard, ColoradoState University [16]. Cover type dataset contains the forest cover type for 30×30meter cells obtained from US Forest Service Region 2 Resource InformationSystem (RIS) data. It contains 581 012 instances and 54 attributes, and it hasbeen used as a benchmark in several papers on data stream classification.4.2 ResultsThe proposed approach shows results similar to the above mentioned AUE2 [18].Next comparison (ref. Fig. 1) of the original random forest with incorporated“replace-the-looser” forgetting and proposed proximity driven streaming randomforest are presented. In Table 1 the mean accuracy is shown. It must be notedthat in case of high window size optimal number of nearest neighbours tends toone (fig. 2).Fig. 1. Original random forest with incorporated “replace-the-looser” and PDSRF ac-curacy on first 100 chunks of the dataset.Table 1. Cover type dataset mean accuracy results. Some of the results obtained usingsettings from [19].Method Mean accuracyPDSRF 87.42HOT 86.48AUE2 85.20PDSRF(without weighting) 82.79AUE1 81.24Lev 81.04DWM 80.84Oza 80.40AWE 79.34Win 77.19Remark 1. For more details of our approach evaluation readers may refer tohttp://mmwind.github.io/pdsrf.Fig. 2. Mean accuracy for Random Forest, and proposed method with intrinsic prox-imity and euclidean measures for block size equal to 500 and window size 1500.Table 2. Cover type dataset mean accuracy results where k is a number of nearestneighbours.blocksizewindowsizek PDSRF withoutweightingPDSRF300 1000 5 77.65 81.15300 1000 10 77.69 81.05300 1000 20 77.69 80.52300 1500 5 77.67 81.11300 1500 10 77.89 81.21300 1500 20 77.8 80.83500 500 5 82.76 86.38500 500 10 82.73 86.16500 500 20 82.68 86.02500 1000 5 82.76 86.45500 1000 10 82.87 86.27500 1000 20 82.75 86.04500 1500 5 82.75 86.49500 1500 10 82.74 86.29500 1500 20 82.7 85.96Table 3. Mean accuracy on first 100 blocks of cover type dataset.blocksizewindowsizek number oftreesPDSRF (withoutweighting)PDSRF500 1000 1 5 75.03 79.25500 1000 1 7 74.37 79.42500 1000 1 10 74.34 78.76500 1000 1 13 74.22 79.56500 1000 1 15 74.00 79.72500 1000 1 17 74.18 80.09500 1000 1 20 74.23 79.90500 1000 2 5 74.96 78.77500 1000 2 7 74.36 79.33500 1000 2 10 74.14 79.51500 1000 2 13 74.03 79.02500 1000 2 15 74.02 79.51500 1000 2 17 74.35 79.68500 1000 2 20 74.09 79.85500 1000 3 5 74.83 77.78500 1000 3 7 74.66 79.59500 1000 3 10 74.34 79.14500 1000 3 13 74.35 79.17500 1000 3 15 74.09 79.32500 1000 3 17 74.09 79.68500 1000 3 20 74.13 80.22500 1000 5 5 74.59 78.70500 1000 5 7 74.29 78.76500 1000 5 10 74.26 78.66500 1000 5 13 73.94 79.52500 1000 5 15 74.37 80.16500 1000 5 17 74.09 79.41500 1000 5 20 74.01 80.165 Discussion and ConclusionsIn comparison to other concept-drift approaches like Online Random Forest andthe AWE, our approach needs more computational resources and thus more timefor both the training and prediction stages. But PDSRF can outperform themost of modern methods as it is demonstrated in the experiment. The proposedapproach is highly parallelizable and can be implemented using the GPGPU.As it was shown in Table 3 high accuracy can be achieved with relatively smallnumber of base classifiers. It must be noted that the proposed approach can beefficiently applied only to gradual concept drifts. PDSRF is sensible to all theparameters changes and all of these parameters must be accurately tuned. Asit shown the proposed approach significantly exceeds the original random forestwith incorporated “replace-the-looser” forgetting. Although the presented resultsshow that the accuracy is higher than AUE2, the approach has some promisingdirections is that the Random Forest can be used in unsupervised and allows towork with missing data, which is an issue with smart grid datasets and windpower forecasting where telecommunications signals and data recording is not100% robust.Acknowledgment. This work is funded by the RSF grant No. 14-19-00054.References1. Zhukov, A., Kurbatsky, V., Tomin, N., Sidorov, D., Panasetsky, D., Foley, A.:Ensemble methods of classification for power systems security assessment. arXiv,Artificial Intelligence (cs.AI) arXiv:1601.01675 (2016) 1–62. Tomin, N., Zhukov, A., Sidorov, D., Kurbatsky, V., Panasetsky, D., Spiryaev, V.:Random forest based model for preventing large-scale emergencies in power sys-tems. International Journal of Artificial Intelligence 13 (2015) 211–2283. Breiman, L.: Bagging predictors. Machine Learning 24(2) (1996) 123–1404. Ho, T.K.: The random subspace method for constructing decision forests. PatternAnalysis and Machine Intelligence, IEEE Transactions on 20(8) (Aug 1998) 832–8445. Breiman, L.: Random forests. Machine Learning 45(1) (2001) 5–326. Saffari, A., Leistner, C., Santner, J., Godec, M., Bischof, H.: On-line randomforests. In: Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12thInternational Conference on, IEEE (2009) 1393–14007. Sidorov, D.: Modelling of non-linear dynamic systems by Volterra series. In:Attractors, Signals, and Synergetics Workshop, 2000, Pabst Science Publ. USA-Germany (2002) 276–2828. Sidorov, D.: Integral dynamical models: singularities, signals and control. Singa-pore: World Scientific Publ. (2015)9. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams usingensemble classifiers. In: Proceedings of SIGKDD, August 24–27, 2003, Washington,DC, USA. (2003) 226–23510. Gama, J.: Knowledge discovery from data streams. Singapore: CRC Press Publ.(2010)11. Kuncheva, L.: Classifier ensembles for changing environment. In Roli, F., Kittler,J., Windeatt, T., eds.: Multiple Classifier Systems, 2004 5th Intl. Workshop on,Springer-Verlag (2004) 1–1512. Turkov, P., Krasotkina, O., Mottl, V.: Dynamic programming for Bayesian logisticregression learning under concept drift. In: Pattern Recognition and MachineIntelligence: 5th International Conference, PReMI 2013, Kolkata, India, December10-14, 2013. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)190–19513. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams usingensemble classifiers. In: Proceedings of the ninth ACM SIGKDD internationalconference on knowledge discovery and data mining, ACM (2003) 226–23514. Zhukov, A., Kurbatsky, V., Tomin, N., Sidorov, D., Panasetsky, D., Spiryaev, V.:Random forest based model for emergency state monitoring in power systems.In: Mathematical Method for Pattern Recognition: Book of abstract of the 17thAll-Russian Conference with Interneational Participation, Svetlogorsk, TORUSPRESS (2015) 27415. Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Machine learning63(1) (2006) 3–4216. Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)17. Brzezinski, D.: Mining data streams with concept drift. Diss. MS thesis. Dept. ofComputing Science and Management. Poznan University of Technology (2010)18. Brzezinski, D., Stefanowski, J.: Reacting to different types of concept drift: Theaccuracy updated ensemble algorithm. Neural Networks and Learning Systems,IEEE Transactions on 25(1) (Jan 2014) 81–9419. Brzezinski, D., Stefanowski, J.: Reacting to different types of concept drift: Theaccuracy updated ensemble algorithm. Neural Networks and Learning Systems,IEEE Transactions on 25(1) (2014) 81–94

Random forest based approach for concept drift handling:Analysis of Images, Social Networks and Texts: 5th International Conference, AIST 2016, Yekaterinburg, Russia, April 7-9, 2016, Revised Selected Papers

Random forest based approach for concept drift handling:Analysis of Images, Social Networks and Texts: 5th International Conference, AIST 2016, Yekaterinburg, Russia, April 7-9, 2016, Revised Selected Papers

Abstract

Similar works

Full text

Available Versions

Queen's University Belfast Research Portal