Frenda, Simona

Somnath, Banerjee

English

Institutional Research Information System University of Turin

Deep analysis in aggressive Mexican tweetsSimona Frenda1,2 and Somnath Banerjee31 University of Turin, Italy2 Universitat Polite`cnica de Vale`ncia, Spainsfrenda@unito.it3 Jadavpur University, Indiasb.cse.ju@gmail.comAbstract. The importance of the detection of aggressiveness in socialmedia is due to real effects of violence provoked by negative behavior on-line. Indeed, this kind of legal cases are increasing in the last years. Forthis reason, the necessity of controlling user-generated contents has be-come one of the priorities for many Internet companies, although currentmethodologies are far from solving this problem. Therefore, in this workwe propose an innovative approach that combines deep learning frame-work with linguistic features specific for this issue. This approach hasbeen evaluated and compared with other ones in the framework of theMEX-A3T shared task at IberEval on aggressiveness analysis in SpanishMexican tweets. In spite of our novel approach, we obtained low results.Keywords: Aggressiveness Detection · Deep Learning · Linguistic Anal-ysis.1 IntroductionThe opinions expressed online by users are usually uncontrolled and this lack ofcontrol facilitates and supports negative online behaviors such as cyberbullying,racism, sexism and any form of hate. In the last few years, governments, socialmedia platforms, Internet companies and communities of citizens are spendinga growing amount of efforts to monitor and contrast such forms of online ag-gressive behaviors and attitudes, with the main aim of limiting it. An exampleof governmental dedication about this subject is the campaign No Hate SpeechMovement of the Council of Europe for human rights online. On the academicside, the research interest about this issue is increasing and the approach is natu-rally interdisciplinary. Especially in the natural language processing (NLP) field,the attention is supported by international and national workshops or campaignsof evaluation like the competition proposed in the framework of IberEval 2018 bythe organizers of MEX-A3T4 [1] on the aggressiveness analysis in Twitter. Thistrack proposes to detect the aggressiveness on Mexican Spanish tweets providingtexts containing offensive messages that disparage or humiliate specific target.In this paper we present our participation in this task proposing a new approach4 https://mexa3t.wixsite.com/home/aggressive-detection-track2 Frenda et al.that combines deep learning with linguistic features.The remainder of the paper is organized as follows. In Section 2 we describesynthetically the previous approaches used until today. In Section 3 we presentour proposal followed by the results obtained in the competition (Section 4).Finally, in Section 5 we draw some conclusions.2 Related WorkCurrently, commercial and simple methods to deal with the automatic detectionof negative online behaviors rely on the use of blacklists , essentially composedwith slurs and swear words. However, filtering the messages in this way does notprovide a sufficient remedy because it falls short when user-generated contentis more subtle. Therefore, the research challenges in this field are oriented atinvestigating deeply all dimensions of language and also the communication onthe Web, to envision deeper and more sophisticated solutions exploiting surfacefeatures ([6], [12]), syntactic features [4], semantic and conceptual features, po-larity information [11], word-embedding techniques ([13], [16]), world knowledgeinformation from ontologies [8], or proposing profile-based approach [9]. Someauthors focus also on the extraction of meta-information from social platformsabout users (like gender) and on their social activity (like history or geolocaliza-tion of posts) as predictive features [7]. In addition, some scholars take advantageof the connection between sentiment analysis and hate speech, benefiting fromsentiment lexicons [18] or using a multi-step approach that combines sentimentor subjectivity classifiers with systems of hate speech detection ([8], [10]). Thisrelation is due to the fact that hate speech expressions mostly exhibit a negativepolarity, although the polarity intensity depends above all on cultural factors.Indeed, the aggressiveness involves different aspects of the user/author of themessage that are difficult to define. So, taking into account the literature, weanalyzed linguistically the data and we tried to understand what are the char-acteristics of aggressive tweets in the context of the Mexican culture and alsothe emotions that arouse this behavior.3 MethodologyThe common approach to detect aggressiveness online is formulating a predictiontask, and in particular MEX-A3T organizers proposed a classification task withthe aim to distinguish aggressive tweet from the non-aggressive [1]. Consideringthe complexity of this task, we needed to analyze the provided data in order tocontemplate the different factors involved: linguistic characteristics proper of atweet (like shortness or informal language), emotive traits of the aggressivenessand cultural aspects considering the fact that the provided data are geolocalizedin Mexico. Therefore, we propose in this paper an innovative approach thatincorporates linguistic features into deep learning architecture.In the next subsections we describe the set of features provided along with thetraining data and deep learning architecture operations.Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)109Deep analysis in aggressive Mexican tweets 33.1 Linguistic FeaturesThe linguistic features employed aim to cover all the above aspects about theaggressiveness in the context of a tweet. Textual features As textual features wetake into account the polarity (positive/negative/neutral) of emoticons5, usedespecially for giving contextual information to readers.Style and writing density We consider also stylistic traits of authors, such as:the use of specific abbreviations used in Mexican tweets (hdp, alv), the numberof characters per sentence and word, the use of some elements of punctuation(question, exclamation marks and sequences of dots) and the uppercase charac-ters, inspecting if the user writes all in uppercase or just some letters.Bag of words In order to understand the importance of some words respectto others, we extract trigrams of words weighted with tf-idf.Lists of aggressive words Considering the fact that the aggressive text aims tooffend, attack, humiliate and hurt an individual or collective target, we createdtwo lists containing specifically derogatory adjectives and vulgar expressions likeprofanities and insults (chinga a tu madre, vete a la verga).Syntactic patterns Another factor involved in aggressive texts is the target,implicit or explicit, to whom the insults or profanities are addressed. There-fore, we examined the syntactic combinations of target explicited with mention(@usuario) or proper name with derogatory adjectives and vulgar expressions.Affective features Finally, as said above, we take into account the emotionsconcerning aggressiveness and we observed that anger and disgust are the princi-pal emotions that provoke this kind of behavior. For this feature, we used SpanishEmotion Lexicon (SEL) provided by [17] and [15], considering the words with ahigher Probability Factor of Affective use in Spanish language. In addition, weincreased it with slang words usually used in social networks [14], taking intoconsideration also the cases of synonymy.In order to allow our architecture to process these features, we preprocessedthe data deleting symbols and urls that can hinder the process of extraction offeatures and pos-tagging the texts using FreeLing [5].3.2 Deep Learning FrameworkIn this section, we describe the deep learning (DL) framework for detectingthe aggressive tweets. The proposed model is inspired by the deep architectureproposed in [3]. They combined the feature engineering with DL and increasedthe classification accuracy for the code-mixed question classification task [2].To understand the effectiveness of combining feature engineering with the DLframework, we have experimented with two setups: one with feature engineeringand another without it. Therefore, we have proposed two models: Model-1 isbased on the DL framework with feature engineering and Model-2 is based on DL5 The annotated list of emoticons used for this work is provided by the Unicode Con-sortium: http://www.unicode.org/Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)1104 Frenda et al.framework without feature engineering. The deep learning framework is basedon Convolutional Neural Network (CNN).Embedding layer: Instead of using any pre-trained word embedding scheme,we have built a vocabulary table which is learned from the training data. Theembedding layer works as a lookup table which maps vocabulary word indicesinto low-dimensional vector representations. As the aggressive tweets are of vari-able length, we used the zero-padding (i.e., the missing part replaced by zeros)to maintain the input vector to a fixed size L.Features: For Model-1, we integrated the features in the embeddings. Wederived a feature set as described in Section 3.1. We combined these features withDL in Model-1. However, we did not combine the features with DL frameworkin Model-2.Convolutional layer: Let ti ∈ Rk be the k-dimensional vector correspondingto the i-th word in the tweet. A tweet is represented as t1:n = t1 ⊕ t2 ⊕ ...⊕ tn,where, the tweet contains the words t1, t2, . . . , tn and ⊕ is the concatenationoperator.Also, let tf1:m = tf1 ⊕ tf2 ⊕ ...⊕ tfm be the feature set for the tweet t1:n. Aftercombining the feature set tf1:m with the vector representation of the tweet t1:n,the resulting vector is l1:m+n = tf1:m⊕t1:n. Therefore, l1:m+n = l1⊕l2⊕...⊕lm+n,where either li ∈ tf1:m or li ∈ t1:n.Let li:i+j refer to the concatenation of li, li+1, . . . , li+j . In the convolution op-eration, the filter w ∈ Rhk is applied to a window of h words to producenew features such as feature si is generated from a window of words li:i+h1by si = f(w.li:i+h−1 + b), where, b ∈ R is a bias term and f is a non-linearfunction. A feature map s = [s1, s2, . . . , snh+1] (where, s ∈ Rn−h+1) is pro-duced by applying the aforesaid filter to each possible window of h words (i.e.,{l1:h, l2:h+1, . . . , lnh+1:n}) in the tweet. The max-pooling operation is applied tothe feature map s to obtain the maximum value s′ = max{s} for the particularfilter. The objective of the max pooling is to capture the most important featurewith the highest value for each feature map. However, the proposed architectureuses multiple filters with varying window sizes to obtain multiple features. Then,these features are passed to the next layer, i.e., a fully-connected layer.Fully-connected layer: The fully-connected layer is also known as the denselayer. The max-pooling operation selects the best features from each convolu-tional kernel. Thus, all the resulting features which are selected from the max-pooling are combined in the fully-connected layer. The output of fully connectedlayer is passed to the output layer.Output layer: The final layer (i.e., the output layer) is made of 2 neurons asthe given tweets are of 2 target classes (i.e., aggressive and non aggressive). Theoutput layer uses ‘softmax’ as the nonlinear activation function.4 ResultsIn the framework of the evaluation campaign, we have submitted two runs: theDL based on CNN with feature engineering (DLF+FE) and a simple DL frame-Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)111Deep analysis in aggressive Mexican tweets 5work based on CNN (DLF). In order to evaluate the performance of the systemsin the competition, the organizers use the F-measure of aggressiveness class. InTable 1, we report the scores obtained along with our position in the ranking forthe aggressive tweets prediction. In spite of the novelty of our approach, the re-sults are low and the feature engineering does not outperform the deep learningbased model.Table 1. ResultsOverall prediction Aggressiveness predictionAccuracy F-score Precision Recall Precision Recall F-score RankDLF 0.6702 0.5585 0.5585 0.5585 0.3363 0.3367 0.3365 9DLF+FE 0.5865 0.5094 0.5149 0.5183 0.2676 0.3827 0.3150 105 Discussion and ConclusionsIn this work, we investigate the automatic detection of aggressive texts by in-corporating linguistic features into deep learning architecture. Considering thelow results, we carry out error analysis that reveals that our systems mainlyfail to classify tweets with orthographic errors and sarcastic or ironic utterances,such as: “USUARIO #LOS40MeetAndGreet 9. Por q es una mama´ luchona quecuida a su bendicio´n”; “Quiero hablar con el que invento´ el hecho de “levantarsetemprano” Que xxxxx estaba pensando”. Therefore, taking into account theseobservations, we will investigate the use of humorous devices to express nega-tive opinions. Moreover, as future work, in order to make deeper analysis aboutthe impact of feature engineering on deep learning approach, we would like topropose this approach in similar research issues.AcknowledgementThe work of Simona Frenda was partially funded by the Spanish MINECO underthe research project SomEMBED (TIN2015-71147-C2-1-P).References1. A´lvarez-Carmona, Miguel A´ and Guzma´n-Falco´n, Estefan´ıa and Montes-y-Go´mez,Manuel and Escalante, Hugo Jair and Villasen˜or-Pineda, Luis and Reyes-Meza,Vero´nica and Rico-Sulayes, Antonio: Overview of MEX-A3T at IberEval 2018: Au-thorship and aggressiveness analysis in Mexican Spanish tweets. Notebook Papers of3rd SEPLN Workshop on Evaluation of Human Language Technologies for IberianLanguages (IBEREVAL), Seville, Spain, September (2018)2. Banerjee, Somnath, Sudip Kumar Naskar, Paolo Rosso, and Sivaji Bandyopad-hyay: The First Cross-Script Code-Mixed Question Answering Corpus. In Multi-LingMine@ ECIR, pp. 56-65. (2016)3. Banerjee, Somnath, Sudip Naskar, Paolo Rosso, and Sivaji Bandyopadhyay: Codemixed cross script factoid question classification-A deep learning approach. Journalof Intelligent & Fuzzy Systems 34, no. 5: 2959-2969. (2018)Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)1126 Frenda et al.4. Burnap, Pete, and Matthew L. Williams: Us and them: identifying cyber hate onTwitter across multiple protected characteristics. EPJ Data Science 5, no. 1: 11.(2016)5. Carreras, Xavier, Isaac Chao, Lluis Padr, and Muntsa Padr: FreeLing: An Open-Source Suite of Language Analyzers. In LREC, pp. 239-242. (2004)6. Chen, Ying, Yilu Zhou, Sencun Zhu, and Heng Xu: Detecting offensive language insocial media to protect adolescent online safety. In Privacy, Security, Risk and Trust(PASSAT), 2012 International Conference on and 2012 International Confernece onSocial Computing (SocialCom), pp. 71-80. IEEE. (2012)7. Dadvar, Maral, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong: Im-proving cyberbullying detection with user context. In European Conference on In-formation Retrieval, pp. 693-696. Springer, Berlin, Heidelberg. (2013)8. Dinakar, Karthik, Birago Jones, Catherine Havasi, Henry Lieberman, and RosalindPicard: Common sense reasoning for detection, prevention, and mitigation of cyber-bullying. ACM Transactions on Interactive Intelligent Systems (TiiS) 2, no. 3: 18.(2012)9. Escalante, Hugo Jair, Esa Villatoro-Tello, Sara E. Garza, A. Pastor Lpez-Monroy,Manuel Montes-y-Gmez, and Luis Villaseor-Pineda: Early detection of deceptionand aggressiveness using profile-based representations. Expert Systems with Appli-cations 89: 99-111. (2017)10. Gitari, Njagi Dennis, Zhang Zuping, Hanyurwimfura Damien, and Jun Long: Alexicon-based approach for hate speech detection. International Journal of Multi-media and Ubiquitous Engineering 10, no. 4: 215-230. (2015)11. Justo, Raquel, Thomas Corcoran, Stephanie M. Lukin, Marilyn Walker, and M.Ins Torres: Extracting relevant knowledge for the detection of sarcasm and nastinessin the social web. Knowledge-Based Systems 69: 124-133. (2014)12. Mehdad, Yashar, and Joel Tetreault: Do Characters Abuse More Than Words?. InProceedings of the 17th Annual Meeting of the Special Interest Group on Discourseand Dialogue, pp. 299-303. (2016)13. Nobata, Chikashi, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang:Abusive language detection in online user content. In Proceedings of the 25th in-ternational conference on world wide web, pp. 145-153. International World WideWeb Conferences Steering Committee. (2016)14. Posadas-Durn, Juan-Pablo, Ilia Markov, Helena Gmez-Adorno, Grigori Sidorov,Ildar Batyrshin, Alexander Gelbukh, and Obdulia Pichardo-Lagunas: Syntactic n-grams as features for the author profiling task. Working Notes Papers of the CLEF.(2015)15. Daz Rangel, Ismael, Grigori Sidorov, and Sergio Surez Guerra: Creacin y evaluacinde un diccionario marcado con emociones y ponderado para el espaol. Onomazein1, no. 29 (2014).16. Samghabadi, Niloofar Safi, Suraj Maharjan, Alan Sprague, Raquel Diaz-Sprague,and Thamar Solorio: Detecting Nastiness in Social Media. In Proceedings of theFirst Workshop on Abusive Language Online, pp. 63-72. (2017)17. Sidorov Grigori, Miranda-Jimnez Sabino, Viveros-Jimnez Francisco, GelbukhAlexander, Castro-Snchez No, Velsquez Francisco, Daz-Rangel Ismael, Surez-GuerraSergio, Trevio Alejandro and Gordon Juan: Empirical study of opinion mining inSpanish tweets. LNAI 7629, pp. 1-14 (2012)18. Van Hee, Cynthia, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, GuyDe Pauw, Walter Daelemans, and Vronique Hoste: Detection and fine-grained clas-sification of cyberbullying events. In International Conference Recent Advances inNatural Language Processing (RANLP), pp. 672-680. (2015)Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)113

Deep analysis in aggressive Mexican tweets

https://iris.unito.it/bitstream/2318/1676282/1/MEX-A3T_paper3.pdf

Deep analysis in aggressive Mexican tweets

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin