International audienceIn this paper, we present the method we developed when participating to the e-Risk pilot task. We use machine learning in order to solve the problem of early detection of depressive users in social media relying on various features that we detail in this paper. We submitted 4 models which differences are also detailed in this paper. Best results were obtained when using a combination of lexical and statistical features

Abdou Malam, Idriss

Arziki, Mohamed

Benamara, Farah

El Kaidi, Assafa

Es-Saghir, Bouchra

He, Zhaolong

Housni, Mouad

Moriceau, Véronique

Mothe, Josiane

Nezar Bellazrak, Mohammed

Ramiandrisoa, Faneva

In this paper, we present the method we developed when participating to the e-Risk pilot task. We use machine learning in order to solve the problem of early detection of depressive users in social media relying on various features that we detail in this paper. We submitted 4 models which differences are also detailed in this paper. Best results were obtained when using a combination of lexical and statistical features

Open Archive Toulouse Archive Ouverte

Any correspondence concerning this service should be sent  to the repository administrator: tech-oatao@listes-diff.inp-toulouse.fr This is an author’s version published in: http://oatao.univ-toulouse.fr/n° de post 19082 To cite this version: Abdou Malam, Idriss and Arziki, Mohamed and Nezar Bellazrak, Mohammed and Benamara, Farah and El Kaidi, Assafa and Es-Saghir, Bouchra and He, Zhaolong and Housni, Mouad and Moriceau, Véronique and Mothe, Josiane and Ramiandrisoa, Faneva IRIT at e-Risk. (2017) In: International Conference of the CLEF Association, CLEF 2017 Labs Working Notes (CLEF 2017), 11 September 2017 - 14 September 2017 (Dublin, Ireland). Official URL: http://ceur-ws.org/Vol-1866/paper_135.pdf Open  Archive  Toulouse  Archive  Ouverte OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible IRIT at e-RiskIdriss Abdou Malam 1 Mohamed Arziki 1 Mohammed Nezar Bellazrak 1Farah Benamara 2 Assafa El Kaidi 1 Bouchra Es-Saghir 1 Zhaolong He2Mouad Housni 1 Ve´ronique Moriceau 3 Josiane Mothe 2 Faneva Ramiandrisoa2(1) IRIT, UMR5505, CNRS & ENSEEIHT, France,(2) IRIT, UMR5505, CNRS & Univ. Toulouse, France,(3) LIMSI, CNRS, Univ. Paris-Sud, Universit Paris-Saclay, FranceAbstract. In this paper, we present the method we developed whenparticipating to the e-Risk pilot task. We use machine learning in orderto solve the problem of early detection of depressive users in social mediarelying on various features that we detail in this paper. We submitted 4models which differences are also detailed in this paper. Best results wereobtained when using a combination of lexical and statistical features.1 IntroductionThe WHO (World Health Organization) reports that “the number of peoplesuffering from depression and/or anxiety increased by almost 50% from 416million to 615 million” from 1990 to 20131. Depression and Bipolar SupportAlliance also estimates that “major depressive disorder affects approximately14.8 million American adults” and “annual toll on U.S. businesses amounts toabout $70 billion in medical expenditures, lost productivity and other costs”(http://www.dbsalliance.org).Depression detection is crucial and many studies are devoted to this challenge[7]. While there are clinical factors that can help for early detection of patientsat risk for depression [10], in this paper we present our approach to help earlydepression detection from social media analysis, as part of our participation toCLEF e-risk 2017 pilot task [6].Recent related work focus on people communication and social media postanalysis to detect depression. Rude’s study shows that depressed people tend touse the personal pronoun (“I”) more intensively than others [9]. Other featureshave also been noticed. For example, De Choudhury et al. noticed that thedepressive people show less activity during the day and more activity duringthe night [3]. Schwartz at al. reported that depressive people tend to use swearwords and talk more about the past [11].These previous studies show that some cues and features extracted fromsocial media posts can be related to depression. In this paper, we report ourinvestigations on using various features in order to answer the e-risk challenge1 http://www.who.int/mediacentre/news/releases/2016/depression-anxiety-treatment/fr/as described in [6]. The e-risk pilot task aims to detect a depressive person assoon as possible by analysing her or his posts in Reddit2 that are provided as asimulated data flow.In our participation runs, the features we used to characterize posts are of twotypes: lexicon-based (extracted using NLTK toolkit3) and numerical features.These features are used in a machine learning method using Weka.The remaining of this paper is organized as follows: Section 2 provides anoverview of the model we used. Section 3 details the different features we im-plemented to train different models. In Section 4 we detail the 4 runs we havesubmitted and the underlying models and present the results. In Section 5 wediscuss the results and depict future work.2 Model overviewFigure 1 presents an overview of the model we use.Fig. 1. Overview of the model used by the e-risk IRIT system.2 https://www.reddit.com/3 NLTK is a platform for building Python programs for natural language pro-cessing that interfaces easily with text processing and machine learning libraries(www.nltk.org)The model is composed of three modules. In the first one, we pre-processthe XML files that contain the users’ posts. The second module aims at extract-ing the features. Notice that while some features capture information from anytextual parts, others focus either on the Title part (corresponding to the initialpost) or on the Text part which corresponds to comments on the initial post.The feature extraction module is extensible: while we developed some features,new features can easily be added. Then, in the formatting module, we select asubset of the features to be used in the model.3 Features and modelsWe developed different types of features. Some have linguistic foundation whileothers are more statistically-based. We distinguish lexicon-based features fromother numerical features.For lexicon-based features, we rely either on previous observations on depres-sive subjects’ behaviour [3, 11] or on hypothesis that we wanted to evaluate.Table 1 presents the features that rely on a lexicon. Each feature is calculatedas follows: (a) we extract the considered lexicon words from each user’s post, (b)for a given user, each lexicon word is weighted by the normalized word frequency(division of the frequency by the total number of words in the user’s posts), (c)we then create one feature by averaging the obtained weights over the lexiconwords.Features are calculated for each user as follows: we first calculate the featurevalue for each of his or her post or comment, then we average the value over hisor her posts in the chunk ; when several chunks are used, we average the featurevalues obtained for each chunk for the considered user.We also used some other numerical features that are described in Table 2.The details of the features are described in [8].We submitted 4 runs corresponding to 4 models. The features that wereused for each model are listed in Table 3. While we used model GPLA to startwith, the other models were introduced later on. The second column of Table 3indicates the chunk number when each model was introduced. The 4 runs cor-responding to our 4 models were performed with the Random Forest learningalgorithm under the Weka platform using the default parameters.In order to decide whether to issue a decision for a subject or wait for morechunks, we used the prediction confidence rate that Weka generates for eachprediction. We set a threshold (estimated using samples of depressive subjects)and we only issued decisions that had a prediction confidence that exceeds theselected threshold. The evolution of the threshold for each model through theruns and according to the chunks can be tracked using Table 4, a threshold of0.5 basically means that all predictions are considered.Num Name Hypothesis or tool/resource used1 Self-Reference High frequency of self-reference words.2 Over generalization Depressive users use words like: ”everyone”,”everywhere”, ”everything” a lot.3 Sentiment Use of Vader analyser [4] for assigning apolarity score to users’ posts:- Negative < -0.05 and Positive > 0.05- Neutral otherwise4 Emotion High frequency of emotionaly negative wordsUsed WordNet-Affect [12], to assign a labelto each word: Negative, Positive or Ambiguouswe then calculated the frequency of each category5 Depression symptoms From De Choudhury et al. [3]& related drugs and Wikipedia list4.6 Past words High frequency of past words.7 Specific verbs High frequency of ”were” and ”was”, ”like””have”, ”being”8 Targeted ”I” Depressive people tend to target themselvesmore in subjective context expecially usingadjectives9 Negative words High frequency of negative wordsUsed SentiWordNet [1] to detect negative wordsin texts10 Part-Of-Speech Higher usage of verbs and adverbs and lowerfrequency usage of nouns11 Relevant 3-grams Higher frequency of 3-grams describedby Gualtiero B. et al. [2] and suggested ones12 Relevant 5-grams Higher frequency of 5-grams describedby Gualtiero B. et al. [2] and suggested ones13 Relevant 1-grams Higher frequency of 1-grams describedby Gualtiero B. et al. [2] and suggested onesTable 1. Details of the features based on lexicons.4 ResultsThe evaluation takes into account not only the correctness of the output of thesystem (i.e. whether or not the user is depressed) but also the delay taken toemit its decision. To this aim, the ERDE (Early Risk Detection Error) metricproposed in [5] is used. This measure rewards early alerts and the delay taken bythe system to make its decision is measured by counting the number of distincttextual items seen before giving the answer.Our best results when considering ERDE measures are obtained using modelGPLC which does not use POS results nor the most frequent n-grams. Includingthem in the model slightly improves F1 measure mainly because of higher recallNum Name Hypothesis or tool/resource used14 Variation of the For depressive people, the variation ofnumber of posts the number of posts is generally small.15 Average number Depressive users have a much lowerof posts number of posts.16 Average number The two groups of users have differentof words per post means.17 Minimum number Depressive users have a lowerof posts value in general.18 Variation of the For depressive people, the variation ofnumber of comments the number of comments is generallysmall.19 Average number Depressive users have a much lowerof comments number of comments.20 Average number The two groups of users have differentof words per comment variances.Table 2. Details of the other numerical features.Name Details Used for the first time in chunkGPLA Features: 1, 7-9, 14-19 1GLPB Features: 1, 6 (verbs only), 7-9 , 14-19 2GLPC Features: 1-9, 14-19 3GLPD All features 10Table 3. Features used in each run.Model / Chunk 1 2 3 4 - 8 9 10GPLA 0.9 0.9 0.9 0.9 0.6 0.5GPLB - 0.8 0.8 0.8 0.6 0.5GPLC - - 0.8 0.8 0.6 0.5GPLD - - - - - 0.5Table 4. Evolution of the decision threshold for the 4 models according to the consid-ered chunk.(0.60 against 0.50) (see Table 5). Our run GPLB had the 2nd best Recall (0.83)across participants and GPLA the 5th.5 Conclusion and Future WorkIn the runs we submitted we consider 19 features. However, some additionalfeatures are worth studying. In future work, we aim at considering temporalfeatures such as the date of the posts, part of the day, etc. Moreover, we wouldlike to modify the way features are calculated : in the case of lexicon-basedName ERDE5 ERDE50 F1 P RGPLA 17.33% 15.83% 0.35 0.22 0.75GPLB 19.14% 17.15% 0.30 0.18 0.83GPLC 14.06% 12.14% 0.46 0.42 0.50GPLD 14.52% 12.78% 0.47 0.39 0.60Table 5. Results for our 4 runs.features, each lexicon item would be a distinct feature. By this way, we wouldobtained a richer representation of each user and potentially a better detection.References1. S. Baccianella, A. Esuli, and F. Sebastiani. Sentiwordnet 3.0: An enhanced lexicalresource for sentiment analysis and opinion mining. Istituto di Scienza e Tecnolo-gie dell Informazione Consiglio Nazionale delle Ricerche Via Giuseppe Moruzzi 1,56124 Pisa, Italy, 2010.2. G. B. Colombo, P. Burnap, A. Hodorog, and J. Scourfield. Analysing the connec-tivity and communication of suicidal users on twitter. Computer Communications,2015.3. M. De Choudhury, M. Gamon, S. Counts, and E. Horvitz. Predicting depressionvia social media. In ICWSM, page 2, 2013.4. C. Hutto and E. Gilbert. Vader: A parsimonious rule-based model for sentimentanalysis of social media text. Eighth International Conference on Weblogs andSocial Media (ICWSM-14). Ann Arbor, MI, June 2014, 2014.5. D. E. Losada and F. Crestani. A test collection for research on depression andlanguage use. In Conference Labs of the Evaluation Forum, page 12. Springer,2016.6. D. E. Losada, F. Crestani, and J. Parapar. eRISK 2017: CLEF Lab on Early RiskPrediction on the Internet: Experimental foundations. In Proceedings Conferenceand Labs of the Evaluation Forum CLEF 2017, Dublin, Ireland, 2017.7. L.-S. A. Low, N. C. Maddage, M. Lech, L. B. Sheeber, and N. B. Allen. Detec-tion of clinical depression in adolescents speech during family interactions. IEEETransactions on Biomedical Engineering, 58(3):574–586, 2011.8. I. A. Malam, M. Arziki, M. N. Bellazrak, A. El Kaidi, B. Es-Saghir, and M. Housni.Automatic detection of depression in social networks. Technical report, Universitde Toulouse, France, 07 2017.9. S. Rude, E.-M. Gortner, and J. Pennebaker. Language use of depressed anddepression-vulnerable college students. Cognition & Emotion, 18(8):1121–1133,2004.10. U. Sagen, A. Finset, T. Moum, T. Mørland, T. G. Vik, T. Nagy, and T. Dammen.Early detection of patients at risk for anxiety, depression and apathy after stroke.General hospital psychiatry, 32(1):80–85, 2010.11. H. A. Schwartz, J. Eichstaedt, M. L. Kern, G. Park, M. Sap, D. Stillwell, M. Kosin-ski, and L. Ungar. Towards assessing changes in degree of depression through face-book. In Proceedings of the Workshop on Computational Linguistics and ClinicalPsychology: From Linguistic Signal to Clinical Reality, pages 118–125, 2014.12. C. Strapparava and A. Valitutti. Wordnet-affect: an affective extension of word-net. Proceedings of the 4th International Conference on Language Resources andEvaluation (LREC 2004), Lisbon, May 2004, pp. 1083-1086, 2004.

IRIT at e-Risk

Scientific Publications of the University of Toulouse II Le Mirail

http://oatao.univ-toulouse.fr/19082/1/abdoumalam_19082.pdf

IRIT at e-Risk

Abstract

Similar works

Full text

Available Versions

Open Archive Toulouse Archive Ouverte

Scientific Publications of the University of Toulouse II Le Mirail