Eldesoukyy, Mohamed

Magdy, Walid

In this paper we present our multilingual tweet classification tool. ClassStrength provides a set of classification models in different languages that classify tweets into 14 generalpurpose categories, including: sports, politics, entertainment, comedy, etc. Our classifier uses a distant-supervision approach for creating training data in any available language on Twitter. The classifier uses a soft-classification scheme, where it generates a likelihood score for a tweet to match each of the 14 categories. The initial version of our tool covers five languages, namely: English, Arabic, French, German, and Russian. More languages are to be covered in next releases. The classification model created for each language is generated from hundreds of thousands of training tweets. Our evaluation to the classifier shows superior accuracy compared to standard manual methods. Our reported accuracy is 84% based on crowd preferences over a balanced test set of English tweets covering all 14 classes

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      ClassStrength: A Multilingual Tool for Tweets ClassificationCitation for published version:Magdy, W & Eldesoukyy, M 2017, ClassStrength: A Multilingual Tool for Tweets Classification. in ASONAM2017 Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social NetworksAnalysis and Mining 2017. ACM, pp. 593-596, 2017 IEEE/ACM International Conference on Advances inSocial Networks Analysis and Mining 2017, Sydney, Australia, 31-3 August. DOI:10.1145/3110025.3110162Digital Object Identifier (DOI):10.1145/3110025.3110162Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:ASONAM 2017 Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social NetworksAnalysis and Mining 2017General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Jun. 2018ClassStrength:A Multilingual Tool for Tweets ClassificationWalid Magdy∗ and Mohamed Eldesouky†∗School of Informatics, The University of Edinburgh, UK†Qatar Computing Research Institute, HBKU, Doha, Qatarwmagdy@inf.ed.ac.uk, mohamohamed@hbku.edu.qaAbstract—In this paper we present our multilingual tweetclassification tool. ClassStrength provides a set of classificationmodels in different languages that classify tweets into 14 general-purpose categories, including: sports, politics, entertainment,comedy, etc. Our classifier uses a distant-supervision approachfor creating training data in any available language on Twitter.The classifier uses a soft-classification scheme, where it generatesa likelihood score for a tweet to match each of the 14 categories.The initial version of our tool covers five languages, namely:English, Arabic, French, German, and Russian. More languagesare to be covered in next releases. The classification model createdfor each language is generated from hundreds of thousands oftraining tweets. Our evaluation to the classifier shows superioraccuracy compared to standard manual methods. Our reportedaccuracy is 84% based on crowd preferences over a balancedtest set of English tweets covering all 14 classes.I. INTRODUCTIONInterest in classifying social posts has increased with thewide spread of social websites, such as Facebook and Twitter.Classifying short social posts such as tweets has been inves-tigated for different applications, including sentiment analy-sis [12], news detection [5], [10], and general-purpose catego-rization [4], [11]. Classifying social posts into general-purposecategories has various applications, such as categorized search,social media analysis, user profiling, and recommendationsystems. In addition, general categories could still be usedfor other classification schemes through transfer learning [9].Most of previous work on tweets classification focused onpreparing a set of manually labeled tweets to train a classifierfor the targeted classification task [1], [3], [5], [8], [10].Data annotation is an expensive task in terms of both timeand money, especially when a large number of classes isused, since a sufficient number of examples are required toeffectively model each class.One of the main characteristics of social media is its highconnectivity, even across different social networks. In thiswork we utilize the network links among different socialplatforms for automatically driving labels for the purpose ofbuilding classification models. The process is called distance-supervision [7], [6]. This method does not require the prepa-ration of any manual annotation. It uses crowd-sourced labelsfrom one social domain, namely YouTube, to train a classifierfor tweet classification. The advantage of this method stemsfrom the ubiquitous availability of free training instancesfor automatic classification. Also, standardized categories areadopted and used in YouTube.The approach collects a large set of tweets, in a givenlanguage, linking to YouTube videos. Each YouTube videois assigned to one category out of 18 predefined general-purpose categories. The category is assigned by users whenuploading a video to YouTube. The approach transfers videocategories as labels to the tweets linking them. This createsa large set of automatically labeled tweets regardless of thelanguage. The collected set of labeled tweets is then usedto train a classifier after a set of preprocessing and pruningsteps to reduce noise and normalize text. Previous work [7],[6] has shown that the proposed approach for collecting largenumber of automatically-labeled tweets to train a classifierachieves significantly better results compared to using a smallset of manually labeled tweets. Analysis in [6] shows thata set of 50,000 automatically-labeled tweets used to train aclassifier would outperform a set of 1,600 carefully manuallylabeled tweets. In spite of this large difference in the size oftraining data of both methods, automatically collecting 50,000tweets linking to YouTube is significantly more efficient thanmanually labeling a set of 1000 tweets in both cost and time.In this paper, we offer our tweets classification toolClassStrength1, which applies distant-supervision algorithm tocollect labels automatically for training tweet classifiers in5 different languages. For each of the languages, hundredsof thousands of training data were collected and used fortraining a classification model. These models are then usedby our classifier. ClassStrength is composed of mainly threecomponents:1) Feature extraction tool, which is responsible of extract-ing features from tweets that would be used by ourclassifier. This component contains the feature extractionscript, in addition to, the feature vector files, where thereis a vector file for each language.1http://alt.qcri.org/class strength/Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from permissions@acm.org.ASONAM ’17, July 31 -August 03, 2017, Sydney, Australiac© 2017 Association for Computing Machinery.ACM ISBN 978-1-4503-4993-2/17/07?/$15.00http://dx.doi.org/10.1145/3110025.31101622) Classification models, are the models used by the classi-fier to calculate the likelihood probabilities of classifyinga tweet into each of the possible general-purpose classes.There is a model file for each language.3) Language detection module, which is responsible fordetecting the language of a tweet before forwarding itto the correct classification model. This component isoptional, since user can specify the language manually.In the rest of the paper, we describe the components of ourtool in details, including the collection process of the trainingdata and the classifier performance.II. TRAINING DATAWe collected a set containing millions of tweets in 5different languages that are linked to YouTube videos. Weused the Twitter API2 with the string “youtube lang:xx” toquery the stream of tweets in a given language (“xx”) withlinks to YouTube videos3. We collected in one month a setof 6M to 20M tweets in each language, that have a link toYouTube videos covering 18 categories.The large amount of data collected contained some noiseand duplicates. In order to clean up the data for the classifiers’training phase, we applied the following pre-processed steps:• Shallow text normalization, simple cleaning and nor-malization is applied to the tweet’s text, including case-folding (for Arabic, we use normalization describedin [2]), Hyperlinks removal (mostly are YouTube links),tokenization, and punctuation removal.• Removing duplicates, retweets and tweets with exactcontent are filtered out. This helps to avoid over-fitting fortweets that receive large number of retweets. In addition,with this step, automated tweets with fixed text phraseswould be mostly filtered out, e.g. “I liked a video onYouTube: ...”.• Filtering and Merging Categories; One of the largestcategories in YouTube is People & Blogs. However,this category is the most noisy category on YouTube,since it is the default category that gets assigned to avideo in case the user did not assign any. Therefore,we decided to remove tweets carrying this label fromour set. Furthermore, we noticed that categories Film &Animation, Movies, Trailers, and Shows are the leastpopular ones in all the languages. Thus we merged themall into one category since they are all related. This endup our training data to contain 14 categories only.• Balancing Data, as shown in [6], collected labels fromYouTube are highly unbalanced. For example, Music isusually the most popular category linked on Twitter. Inthis step, training data are balanced by reducing the sizeof training samples from each category to the size of thesmallest one. This insures training a less-biased classifier.The previous steps reduces the amount of training samplessignificantly. Nonetheless, the amount of data we end up with2http://twitter4j.org/en/index.html3This also captures tweets with shortened links to YouTube.Language Size of balanced training dataEnglish 913KArabic 482KFrench 544KGerman 349KRussian 405KTABLE I: Number of tweets used to train the classificationmodel of each languageis still large, and more stream of tweets could still be used toadd more training data. Table I shows the size of training dataset used for training the classification model of each language.III. CLASSIFICATION ALGORITHMA. ClassifierDue to the humongous size of our training data, we simplyapplied a bag-of-words (BOW) approach, where each featurerepresents a term and the feature value is binary, denotingpresence or absence of the term in the tweet.SVM mulitclass classifier 4 is used to train our classificationmodel for each language separately using a linear kernel. SVMwas shown to be the most effective and efficient among otherclassification methods in [6].B. PerformanceAn objective evaluation was conducted in [6] showingthat this classification approach using distant-supervision issuperior over standard approach that uses limited number ofmanually labeled data, where the first approach was reportedto achieve an accuracy of 61% compared to only 53% to thelatter.The evaluation in [6] assumes a hard classification task,where each tweet can only be assigned to only one la-bel. In fact, this is not the case in many situations. Thus,ClassStrength is designed to assign a weight for each classaccording to its suitability to the tweet’s text. To measure theperformance of our tool, we applied an additional evaluationto the classifier. Since it assumes a soft-classification task,where multiple categories can apply to the same tweet, weused the same test set in [6] that contains 1677 English tweetsand applied ClassStrength to it. We then presented the toppredicted class for each tweet to workers on Crowdflower5and asked how they evaluate the classification of the tweetsbased on the predicted category. The workers were allowed tochoose among three different choices: 1) Perfect: the categorymatches the tweet’s content perfectly; 2) Acceptable: thepredicted category is OK, but not perfect; or 3) Bad: thepredicted category has no relation to the tweet. We asked threeannotators to label each tweet, and majority voting was used.The agreement among annotators was 65%.Figure 1 shows the distribution of answers of annotators. Asshown, 56% of the tweets were seen to be perfectly classified,4https://www.cs.cornell.edu/people/tj/svm light/svm multiclass.html5https://crowdflower.comBad 16%Acceptable 28%Perfect 56%0% 10% 20% 30% 40% 50% 60%BadAcceptablePerfectHow do you rate the classification of the tweet?Fig. 1: Classes list of the two classification schemas existingin ClassStrengthwhile 28% were seen not perfect, but still acceptable. Only16% of the tweets were seen to carry inaccurate category.This result shows that 84% of the time ClassStrength leads toa reasonable prediction for tweets category out of 14 differentcategories. This results are expected to generalize to otherlanguages as well.IV. CLASSIFICATION TOOLClassStrength tool is available for free use and download atthe following link: http://alt.qcri.org/class strength/. The toolhas three operational modes:Online Mode: This is the mode used mainly for Demopurposes. On the tool webpage, a text box is provided forusers to type in some text in any of the five languages, and agraph of the strength of each category of the 14 categories isdisplayed showing the prediction of classification of the typedtext. The user can manually select the language of the text,or keep the default selection which automatically detects thelanguage and apply the classification to it.Figure 2 shows an example tweet in the text box and thedisplayed strength of each of the categories as estimated byClassStrength. As shown, the tweet is about President Obamameeting an NBA basketball team at the White House. Thelanguage detector identified the language of the tweet, andthen passed the text to the English classification model. Asshown, the classifier detected that the main category of thetweet is News & Politics, while the Sports category alsoreceived some positive score. All the remaining categoriesreceived negative scores, showing that they do not apply tothe tweet.This mode allows users to type freely in the text box andget an illustration to the strength of each category of the inputtext. The best matching category to the input text receives anormalized score of 100%, while the remaining ones receivefractions of this, or negative scores.Batch Mode: This mode allows users to submit a text filecontaining a list of tweets to be classified on ClassStrengthserver, and the output would be ready for download directly. Ifthe user knows the language of the submitted tweets, it is betterto select the language manually, and then the classificationmodel of this language would be applied directly to the listFig. 2: Example of classification strength of an English tweetusing ClassStrength online modeof tweets. Otherwise, the language detector module woulddetect the language of the tweets in the file and classifies themaccordingly.The input file should contain a tweet in each line with thefollowing formant: Tweet ID <tab> Tweet textThe output of classification would contain the tweet ID,followed by a list of 14 numbers representing the strength ofeach class. The strength of each class is a number from -100to 100, where negative values mean that these categories arenot applicable to this tweet.Offline Mode: The offline mode allows users to downloada full version of the tool to be used locally on their machine.Some system requirements are essential to get it working asdescribed in the ReadMe file provided with the tool package.V. CONCLUSION AND FUTURE VERSIONSIn this paper we offer our tweet classification toolClassStrength, which is a tool for classifying social text in gen-eral and tweets in specific into 14 general-purpose categories.The tool currently works with 5 different languages, namelyEnglish, Arabic, French, German, and Russian. The tool usesa state-of-the-art method for collecting training data to trainthe classifiers. The performance of the tool was reported tobe highly satisfactory to users. An online mode of the toolwebpage is available for users to test the performance of thetool.For future versions of ClassStrength, we look forward tointegrating additional languages, hopefully covering most ofthe languages on Twitter. In addition, we will keep updatingour classification models by using fresh training data every-while to keep our classification as accurate as possible. An-other version of our classifier is planned to be developed toautomatically adapt the classification periodically, since someof the categories drifts over time, especially for topics such aspolitics and movies.REFERENCES[1] H. Becker, M. Naaman, and L. Gravano. Beyond trending topics: Real-world event identification on twitter. 2011.[2] K. Darwish and W. Magdy. Arabic information retrieval. NowPublishers, 2014.[3] D. Irani, S. Webb, C. Pu, and K. Li. Study of trend-stuffing on twitterthrough text classification. In Collaboration, Electronic messaging, Anti-Abuse and Spam Conference (CEAS), 2010.[4] S. Kinsella, A. Passant, and J. G. Breslin. Topic classification insocial media using metadata from hyperlinked objects. In Advancesin Information Retrieval, pages 201–206. Springer, 2011.[5] A. Kothari, W. Magdy, K. Darwish, A. Mourad, and A. Taei. Detectingcomments on news articles in microblogs. In AAAI press, 2013.[6] W. Magdy, H. Sajjad, T. El-Ganainy, and F. Sebastiani. Bridging socialmedia via distant supervision. Social Network Analysis and Mining,5(1):1–12, 2015.[7] W. Magdy, H. Sajjad, T. El-Ganainy, and F. Sebastiani. Distantsupervision for tweet classification using youtube labels. In NinthInternational AAAI Conference on Web and Social Media, 2015.[8] D. Quercia, H. Askham, and J. Crowcroft. Tweetlda: supervised topicclassification and link prediction in twitter. In Proceedings of the 4thAnnual ACM Web Science Conference, pages 247–250. ACM, 2012.[9] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng. Self-taughtlearning: transfer learning from unlabeled data. In Proceedings of the24th international conference on Machine learning, pages 759–766.ACM, 2007.[10] J. Sankaranarayanan, H. Samet, B. E. Teitler, M. D. Lieberman, andJ. Sperling. Twitterstand: news in tweets. In Proceedings of the17th acm sigspatial international conference on advances in geographicinformation systems, pages 42–51. ACM, 2009.[11] B. Sriram, D. Fuhry, E. Demir, H. Ferhatosmanoglu, and M. Demirbas.Short text classification in twitter to improve information filtering.In Proceedings of the 33rd international ACM SIGIR conference onResearch and development in information retrieval, pages 841–842.ACM, 2010.[12] M. Thelwall, K. Buckley, and G. Paltoglou. Sentiment strength detectionfor the social web. Journal of the American Society for InformationScience and Technology, 63(1):163–173, 2012.

ClassStrength: A Multilingual Tool for Tweets Classification

https://www.pure.ed.ac.uk/ws/files/39236407/classstrength_multilingual_tool_1.pdf

ClassStrength: A Multilingual Tool for Tweets Classification

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer