Popular Ensemble Methods: An Empirical Study

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund and Shapire, 1996; Shapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier -- especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees

Ensemble parameter estimation for graphical models

Parameter Estimation is one of the key issues involved in the discovery of graphical models from data. Current state of the art methods have demonstrated their abilities in different kind of graphical models. In this paper, we introduce ensemble learning into the process of parameter estimation, and examine ensemble parameter estimation methods for different kind of graphical models under complete data set and incomplete data set. We provide experimental results which show that ensemble method can achieve an improved result over the base parameter estimation method in terms of accuracy. In addition, the method is amenable to parallel or distributed processing, which is an important characteristic for data mining in large data sets.<br /

Mining Students’ Messages to Discover Problems Associated with Academic Learning

WhatsApp has become the preferred choice of students for sending messages in developing countries. Due to its privacy and the ability to create groups, students are able to express their “feelings” to peers without fear. To obtain immediate feedback on problems hindering effective learning, supervised learning algorithms were applied to mine the sentiments in WhatsApp group messages of University students. An ensemble classifier made up of Naïve Bayes, Support Vector Machines, and Decision Trees outperformed the individual classifiers in predicting the mood of students with an accuracy of 0.76, 0.92 recall, 0.72 precision and 0.80 F-score. These results show that we can predict the mood and emotions of students towards academic learning from their private messages. The method is therefore proposed as one of the effective ways by which educational authorities can cost effectively monitor issues hindering students’ academic learning and by extension their academic progress. Keywords: WhatsApp; Sentiments; Ensemble; Classification; Naïve Bayes; Support Vector Machines.

Ensembles as a Sequence of Classifiers

An ensemble is a classifier created by combining the predictions of multiple component classifiers. We present a new method for combining classifiers into an ensemble based on a simple estimation of each classifier&apos;s competence. The classifiers are grouped into an ordered list where each classifier has a corresponding threshold. To classify an example, the first classifier on the list is consulted and if that classifier&apos;s confidence for predicting the example is above the classifier&apos;s threshold, then that classifier&apos;s prediction is used. Otherwise, the next classifier and its threshold is consulted and so on. If none of the classifiers predicts the example above its confidence threshold then the class of the example is predicted by averaging all of the component classifier predictions. The key to this method is the selection of the confidence threshold for each classifier. We have implemented this method in a system called Sequel which has been applied to the task o..