Real-time classification of massive email data is a challenging task that presents its own particular difficulties. Since email data presents an important temporal component, several problems arise: emails arrive continuously, and the criteria used to classify those emails can change, so the learning algorithms have to be able to deal with concept drift. Our problem is more general than spam detection, which has received much more attention in the literature.
In this paper we present GNUsmail, an open-source extensible framework for email classification, which structure supports incremental and on-line learning. This framework enables the incorporation of algorithms developed by other researchers, such as those included in WEKA and MOA. We evaluate this framework, characterized by two overlapping phases (pre-processing and learning), using the ENRON dataset, and we compare the results achieved by WEKA and MOA algorithms

Baena-García, Manuel

Bifet, Albert

Carmona-Cejudo, José M.

del Campo-Ávila, José

Morales-Bueno, Rafael

English

Research Commons@Waikato

GNUsmail: Open Framework for On-line EmailClassificationJose´ M. Carmona-Cejudo and Manuel Baena-Garcı´a and Jose´ del Campo-A´vilaand Rafael Morales-Bueno 1 and Albert Bifet 2Abstract. Real-time classification of massive email data is a chal-lenging task that presents its own particular difficulties. Since emaildata presents an important temporal component, several problemsarise: emails arrive continuously, and the criteria used to classifythose emails can change, so the learning algorithms have to be ableto deal with concept drift. Our problem is more general than spamdetection, which has received much more attention in the literature.In this paper we present GNUsmail, an open-source extensibleframework for email classification, which structure supports incre-mental and on-line learning. This framework enables the incorpo-ration of algorithms developed by other researchers, such as thoseincluded in WEKA and MOA. We evaluate this framework, char-acterized by two overlapping phases (pre-processing and learning),using the ENRON dataset, and we compare the results achieved byWEKA and MOA algorithms.1 INTRODUCTIONThe use of email grows every day, as a result of new applications forboth personal and professional purposes. When emails are read andprocessed by the user, they can be easily stored if necessary. Thanksto the low cost of that storage, it is usual to file emails with a certainlevel of interest into some hierarchical category-based folders.However, such advantages lead to an increasing volume of emailsto be processed every day. The messages are usually stored into aspecific place of the structure, which must be determined manuallyby the user. This can become a very time-consuming job. Althoughmost email clients support hand-crafted rules for this task, these rulesare usually static, inflexible, and considered to be too difficult to beused by most users [8]. Therefore, it would be desirable to automati-cally induce such rules.Machine learning and data mining methodologies show how to in-duce models from a data set. Traditionally, batch methods have beenapplied to email classification, but recently on-line methods for deal-ing with new problems, such as concept drift or large volume of mes-sages, are emerging. Email classification is a subfield of more generaltext classification research area, and its aim is to assign labels from apredefined set to each email. From a very simplified point of view, thefollowing activities have to be carried out in order to construct a ma-chine learning-based email classifier: corpus preprocessing, modelconstruction and validation.Little effort has been devoted to applying stream mining tools [7]to email classification. However, they are necessary when a lot of1 Universidad de Ma´laga, Spain, email: {jmcarmona, mbaena, jcampo,morales}@lcc.uma.es2 University of Waikato, New Zealand, email: abifet@cs.waikato.ac.nzemails arrive continuously and must be processed, only once if pos-sible (incremental learning), and also because folders are constantlycreated, moved or deleted, and the nature of emails within a foldermay change over time (concept drift). To automatically produce clas-sifying models taking into account all of the previous information, weneed an on-line approach.That said, the main goal of our contribution is the implementationof an open framework for on-line email classification. The proposedsystem allows the incorporation of stream-based algorithms, such asthose included in the WEKA [11] and MOA [4] libraries, offering aflexible and extensible preprocessing module. The rest of this paperis organized as follows: in Section 2 our proposed preprocessing andlearning framework is introduced, in Section 3 we explain the ex-perimental evaluation, and in Section 4 we give our conclusions andmention some possibilities for future work.2 GNUSMAIL DESIGN PHILOSOPHYIn this work we propose an extended version of GNUsmail [6], whichincorporates incremental an on-line algorithms (including the man-agement of concept drift) into the original architecture. GNUsmailis an open-source framework for adaptive email classification, withan extensible preprocessing module, based on the concept of ‘fil-ters’ (each of these filters extract one or more attributes from rawemail messages), and an equally extensible learning module (new al-gorithms, methods and libraries can be easily integrated).Since arbitrary machine learning algorithms can be used insideour framework, GNUsmail is particularly useful for research pur-poses, as the framework does not limit itself to a single method oralgorithm. The experimental evaluation of this work is an example,showing how new paradigms for classifiers are evaluated. Neverthe-less, GNUsmail can also be used as a ready-to-use application, sincean interface with the popular Mozilla Thunderbird email client hasbeen implemented (see [6]). Our source code is licensed under theGPL license, and it is available at http://code.google.com/p/gnusmail/.GNUsmail contains an email reader module, a text preprocessingmodule and a configurable learning module that can be configured touse specific algorithms and parameters. The preprocessing moduledeals with the extraction of attributes and their corresponding val-ues from each email. When an incremental or on-line approach isfollowed, the email messages are read while the model is built. How-ever, current models have an important limitation that affects thisprocess: the learning attributes to be used to induce the model haveto be fixed before beginning the induction of the algorithm since it isnot possible to start to use a new attribute in the middle of the life-ECAI 2010H. Coelho et al. (Eds.)IOS Press, 2010© 2010 The authors and IOS Press. All rights reserved.doi:10.3233/978-1-60750-606-5-11411141time of a learning model. Our main learning attributes are the mostrepresentative words in the email corpus, and thus some emails haveto be read in advance, in order to decide which words are to be usedas attributes. Hence, the first step of the training phase results in alist of attributes to be used by the model. After we have fixed theseattributes, the incremental or on-line algorithm can start to analyzethe incoming messages one by one, updating the model.WEKA methods are used mainly with small datasets in environ-ments without time and memory restrictions. MOA [5], on the otherhand, is a data mining framework that adjusts to more demandingproblems. In the streaming setting, data arrives at high speed andthere are very strict time and space constraints.We have used some ensemble methods from the MOA collec-tion, such as OzaBag and OzaBoost, which outperform boostingin the streaming setting. For concept drift detection we have usedDDM [10] and EDDM [1]. These methods detect change in the ac-curacy error of the classifier by comparing window statistics.3 EXPERIMENTAL EVALUATIONWe have evaluated our framework for both incremental and on-linelearning schemes, using the well-known ENRON dataset [12]. Wehave used a prequential metric [9], that monitors the evolution of thesystem’s performance over time.For the evaluation of incremental methods in WEKA, we haveused an updateable Naı¨ve Bayes, IB-k and NN-ge models. The per-formance obtained by NN-ge is comparable with the best results ob-tained by [2] and [3], which use SVM-based approaches.We have also evaluated the performance of several on-line algo-rithms with concept drift detection, using algorithms from the MOAframework. See Table 1 and Figure 1.Table 1. Final folder-wise MOA prequential accuracies with bagging ofNN-ge with DDM detector for some authorsUser Correct / Total Percentagefarmer-d 2743/3590 76.41%kaminsky-v 1798/2699 66.62%lokay-m 1953/2479 78.78%sanders-r 887/1207 73.49%williams-w3 2653/2778 95.5%0204060801000 500 1000 1500 2000 2500 3000 3500 4000MajorityClassHoeffdingTreeNBAdaptiveOzaBag and DDM over NNgeNNgeNBFigure 1. Evolution of the prequential error for farmer-d4 CONCLUSIONIn this paper we have presented GNUsmail, an open-source extensi-ble framework for email classification which incorporates new meth-ods for on-line learning with concept drift. GNUsmail flexible archi-tecture allows to incorporate new filters for attribute generation, aswell as alternative learning algorithms.In our experiments we have compared MOA and Weka ma-chine learning frameworks. We have found out that MOA outper-forms the learning methods in Weka. By automatically adapting themodel, concept-drift detection avoids a decline in performance whenconcept-drift occurs in the flow of messages.With respect to on-line classifiers for email classification, somedeficiencies have been detected. For example, it would be necessaryto develop algorithms able to learn with only a few hundreds exam-ples. Most importantly, up-to-date on-line methods need to know apriori all the attributes, values and classes. This limitation makes itnecessary to process the data twice: first to extract features, and thento learn. Future methods should support on-line feature extraction, aswell as learning.Finally, considering feature extraction, it would be useful to in-clude some new techniques in future work, such as named entityrecognition, management of temporal features, or formality features,like spelling or the use of formal terms.ACKNOWLEDGEMENTSThis work has been partially supported by the SESAAME project(code TIN2008-06582-C03-03) of the MEC, Spain.REFERENCES[1] Manuel Baena-Garcı´a, Jose´ del Campo-A´vila, Rau´l Fidalgo, AlbertBifet, Ricard Gavalda`, and Rafael Morales-Bueno, ‘Early drift detec-tion method’, in Fourth International Workshop on Knowledge Discov-ery from Data Streams, (2006).[2] R. Bekkerman, A. Mccallum, and G. Huang, ‘Automatic categoriza-tion of email into folders: Benchmark experiments on Enron and SRICorpora’, Technical report, Center for Intelligent Information Retrieval,(2004).[3] P. Bermejo, J. A. Ga´mez, J. M. Puerta, and R. Uribe-Paredes, ‘Im-proving KNN-based e-mail classification into folders generating class-balanced datasets’, in Proceedings of the 12th International Confer-ence on Information Processing and Management of Uncertainty inKnowledge-Based Sytems (IPMU-2008), pp. 529–536, (2008).[4] A. Bifet, G. Holmes, B. Pfahringer, R. Kirkby, and R. Gavalda`, ‘Newensemble methods for evolving data streams’, in Proceedings of the15th ACM SIGKDD International Conference on Knowledge Discoveryand Data Mining (KDD-2009), pp. 139–148, (2009).[5] Albert Bifet and Richard Kirkby. http://sourceforge.net/projects/moa-datastream/.[6] J.M. Carmona-Cejudo, M. Baena-Garcı´a, and R. R. Morales-Bueno,‘Amc-gnusmail: an extensible machine learning based framework foremail classification’, in Proceedings of TTIA ’09. IASK, (2009).[7] Stream Data Management, eds., N. Chaudhry, K. Shaw, and M. Ab-delguerfi, Advances in Database Systems, Springer, 2005.[8] E. Crawford, J. Kay, and E. McCreath, ‘IEMS - the intelligent emailsorter’, in Proceedings of the 19th International Conference on Ma-chine Learning (ICML -2002), pp. 83–90, (2002).[9] A. P. Dawid, ‘Present position and potential developments: Some per-sonal views: Statistical theory: The prequential approach’, Journal ofthe Royal Statistical Society. Series A, 147(2), 278–292, (1984).[10] J. Gama, P. Medas, G. Castillo, and P. Rodrigues, ‘Learning with driftdetection’, in SBIA Brazilian Symposium on Artificial Intelligence, pp.286–295, (2004).[11] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H.Witten, ‘The WEKA data mining software: An update’, in SIGKDDExplorations, volume 11-1, pp. 10–18, (2009).[12] B. Klimt and Y. Yang, ‘The enron corpus: A new dataset for email clas-sification research’, in Proceedings of the 15th European Conferenceon Machine Learning (ECML-2004), (2004).J.M. Carmona-Cejudo et al. / GNUsmail: Open Framework for On-Line Email Classification1142

a, ‘New ensemble methods for evolving data streams’,

Amc-gnusmail: an extensible machine learning based framework for email classiﬁcation’,

Automatic categorization of email into folders: Benchmark experiments on Enron and SRI Corpora’,Technicalreport,CenterforIntelligentInformationRetrieval,

IEMS - the intelligent email sorter’,

Improving KNN-based e-mail classiﬁcation into folders generating classbalanced datasets’,

Jos´ e del Campo-´ Avila, Ra´ ul Fidalgo, Albert Bifet, Ricard Gavald` a, and Rafael Morales-Bueno, ‘Early drift detection method’,

Learning with drift detection’,

Present position and potential developments: Some personal views: Statistical theory: The prequential approach’,

The enron corpus: A new dataset for email classiﬁcation research’,

The WEKA data mining software: An update’,

GNUsmail: Open framework for on-line email classification

Name not available

GNUsmail: Open framework for on-line email classification

Abstract

Similar works

Full text

Available Versions

Research Commons@Waikato

Name not available