Cignarella, ALESSANDRA TERESA

Spanish

Institutional Research Information System University of Turin

Dependency Syntax in the Automatic Detectionof Irony and StanceSintaxis de dependencias en la detección automáticade irońıa y posicionamientoAlessandra Teresa Cignarella1,21PRHLT Research Center, Universitat Politècnica de València, Spain2Dipartimento di Informatica, Università degli Studi di Torino, Italyalessandrateresa.cignarella@unito.itAbstract: PhD thesis in Computer Science written by Alessandra Teresa Cigna-rella under the supervision of Dr. Cristina Bosco (University of Turin) and Prof. Dr.Paolo Rosso (Universitat Politècnica de València). This thesis was developed undera cotutelle between the PRHLT Research Center of the Universitat Politècnica deValència, Spain and the Computer Science Department of the University of Turin,Italy. The thesis defense was held in Torino (online), Italy on October 26th, 2021.The doctoral committee was composed by: Prof. Dr. Joakim Nivre (Departmentof Linguistics and Philology, Uppsala University, Sweden), Dr. Saif M. Mohammad(National Research Council Canada, Ottawa, Canada), and Prof. Dr. VeroniqueHoste (Department of Translation, Interpreting and Communication, Ghent, Bel-gium). An international mention was achieved after having spent 18 months at theUniversità degli Studi di Torino and 18 months at the Universitat Politècnica deValència.Keywords: Universal Dependencies, Irony Detection, Stance Detection, NLP, Mul-tilingual, User-Generated Content, Twitter, BERT.Resumen: Tesis doctoral en Informática realizada por Alessandra Teresa Cignarel-la y dirigida por la Dra. Cristina Bosco (University of Turin) y el Prof. Dr. PaoloRosso (Universitat Politècnica de València) en el marco de un convenio de cotutelaentre el PRHLT Research Center de la Universitat Politècnica de València, Españay el Departamento de Informatica de la Universidad de Turin, Italia. La defensade la tesis fue en Turin (en linea), Italia el 26 de octubre de 2021 ante un tribunalcompuesto por: Prof. Dr. Joakim Nivre (Department of Linguistics and Philology,Uppsala University, Sweden), Dr. Saif M. Mohammad (National Research CouncilCanada, Ottawa, Canada), and Profa. Dra. Veronique Hoste (Department of Trans-lation, Interpreting and Communication, Ghent, Belgium). Se obtuvo la mencióninternacional tras una estancia de 18 meses en la Università degli Studi di Torino y18 meses en la Universitat Politècnica de València.Palabras clave: Dependencias universales, detección de irońıa, detección de posi-cionamiento, PNL, multilingüe, contenido generado por el usuario, Twitter, BERT.1 IntroductionThe present dissertation is part of the broadpanorama of studies of Natural LanguageProcessing (NLP). In particular, it is a workof Computational Linguistics (CL) designedto study in depth the contribution of syntaxin the field of sentiment analysis and, therefo-re, to study texts extracted from social mediaor, more generally, online content.Furthermore, given the recent interest ofthe scientific community in the Universal De-pendencies1 (UD) project (De Marneffe etal., 2021), which proposes an annotation for-mat aimed at creating a “universal” repre-sentation of the phenomena of morphologyand syntax in a manifold of languages, in thiswork we made use of this format, thinkingof a study in a multilingual perspective (Ita-1https://universaldependencies.org/.Procesamiento del Lenguaje Natural, Revista nº 68, marzo de 2022, pp. 145-148 recibido 25-11-2021 revisado 25-01-2022 aceptado 30-01-2022ISSN 1135-5948 DOI 10.26342/2022-68-11 145 ©2022 Sociedad Española para el Procesamiento del Lenguaje Naturallian, English, French and Spanish). Althou-gh the UD format was originally conceivedto be applied to texts that are more “stan-dard” from the point of view of morphosyn-tactic norms and punctuation, in more recentyears the same scheme has begun to be ap-plied also to user-generated content (UGC),i.e. texts extracted from social media, blogs,forums and microblogging platforms, such asReddit, Twitter or Wikipedia pages. Inevi-tably, the application of this annotation fra-mework to such a peculiar textual genre, inwhich the texts are accompanied by multi-media elements such as links, photos and vi-deos, emojis and non-standardized punctua-tion, has opened up several problems in theUniversal Dependencies community, many ofwhich are still the subject of open and heateddebate today.In this work we provide an exhaustive pre-sentation of the morphosyntactic annotationformat of UD, in particular underlining themost relevant issues regarding their applica-tion to UGC. Two sub-areas of NLP will bepresented, and used as case studies, in orderto test the research hypotheses: the first casestudy will be in the field of Irony Detection(Van Hee, Lefever, and Hoste, 2018) and thesecond in the area of Stance Detection (Mo-hammad et al., 2016). In both cases, histo-rical notes are provided that can serve as acontext for the reader, the problems faced areintroduced and the activities proposed in thecomputational linguistics community are de-scribed. Furthermore, particular attention ispaid to the resources currently available aswell as to those developed specifically for thestudy of the aforementioned phenomena. Fi-nally,through the discussion of a set of expe-riments performed within or outside evalua-tion campaigns, we describe how syntax cancontribute to the resolution of such tasks.1.1 Motivation and ObjectivesMy main purpose is to explore the impactof morpho-syntactic information in sentimentanalysis related tasks.Firstly, for both irony and stance, I focu-sed on the importance of the formulation of aclear problem statement, and the subsequentcomputational modeling of it. Secondly, I hi-ghlighted my experience in the creation of an-notated corpora for those problems, my con-tribution to the organization of shared tasksand the important lessons learned, in termsof research understanding. Later on, I pro-posed my first approaches to solve both tasksfrom a shallow perspective, starting to explo-re the most feasible way to represent morpho-syntactic information, to extract it, and ex-ploit it for classification purposes. I endedthis process by relying on the UD Depen-dencies annotation format. Finally, after ha-ving encountered a satisfactory combinationof features, I exploited the best sets of them– some of which are encoded in UD format –and I performed a handful of experiments ina variety of settings.In the whole thesis a multilingual scenariois kept in mind, exploring four different lan-guage settings: English, Spanish, French, andItalian, for both irony detection and stance.Furthermore, due to the availability of ben-chmark datasets, regarding stance detection,I also experiment on a fifth language, i.e.,Catalan.The research questions that I aimedanswering to are as follows:RQ-1: Could features derived from morpho-logy and syntax help to address the task ofirony detection?RQ-2: To what extent does using resourcessuch as treebanks for training NLP modelsimprove the performance in irony detection?RQ-3: Could features derived from morpho-logy and syntax help to address the task ofstance detection?RQ-4: To what extent does using resourcessuch as treebanks for training NLP modelsimprove the performance in stance detection?2 Thesis OverviewThis thesis consists in a reorganized collec-tion of the most relevant investigations ex-tracted from some research projects in whichI was involved during my Ph.D. studies.A brief overview of the contents of the the-sis is presented below, summarizing all thework done and resuming the results obtainedin the framework of this three-year-long re-search path. In Chapter 3, I also show someunpublished results regarding stance detec-tion with dependency syntax and neural net-works. Lastly, I draw some conclusions anddiscuss future work in the final chapter.Alessandra Teresa Cignarella146Chapter 1 – Introduction: In this chap-ter I introduced the reader to the main topicsthat will be discussed in the thesis, startingwith a broad description of Natural Langua-ge Processing and automatic text classifica-tion, followed by an introduction on Univer-sal Dependencies, morphology and syntax. Ialso proposed a brief discussion on the issuesthat can arise while applying the UD formatto social media data, mainly referring to thefollowing work: Sanguinetti et al. (2022).Chapter 2 – Irony Detection: In the se-cond chapter, which deals with the topic ofirony detection, I described several works re-garding such topic. In particular, in Section2.1.1, I mainly referred to Cignarella et al.(2018) in order to describe described the or-ganization of the IronITA 2018 shared ta-sk. In Section 2.1.2, I described the creationof the multilayered corpus TWITTIRÒ-UD,annotated both of irony and morphology anddependency syntax.In Section 2.3, I finally described someexperiments performed in which I leveragedmorpho-syntactic information for irony de-tection, mainly referring to what was done inthe participation of the IroSvA 2019 sharedtask (Cignarella and Bosco, 2019; Cignarellaet al., 2020).Chapter 3 – Stance Detection: In thethird chapter I dealt with the task of stancedetection. In Section 3.1.1, I described theorganization of the SardiStance shared taskat EVALITA 2020. In Section 3.2.1, I pre-sented the work done in Lai, Cignarella, andHernandez Fariás (2017), for describing theparticipation in the StanceCat 2017 sharedtask at IberEval 2017. In Section 3.3, I de-scribed my participation in RumorEval 2019where I first applied a syntax-based approachto the task of stance detection.In Section 3.3.2, I presented a completelynew research, specifically done for the PhDthesis, where I introduced a BERT-based ap-proach leveraging morphosyntactic informa-tion for the automatic detection of stance indifferent languages.Chapter 4 – The Interaction of Ironyand Stance: In the fourth chapter I pro-posed a new part of my research, in which Iexplored the interaction between irony andstance, through the analysis of the Sardi-Stance dataset, which has been annotatedaccordingly to both phenomena.Chapter 5 – Conclusions and FutureWork – In the last chapter, I finally summa-rized all the important lessons learned and Iproposed new research directions for futurework.3 ConclusionsThis thesis collocates within the growingtrend of studies devoted to make Artifi-cial Intelligence results more explainable,going beyond the achievement of highestscores in performing tasks, but rather ma-king their motivations understandable andcomprehensible for experts in the domain.The novel contribution of this work main-ly consists in the exploitation of featuresthat are based on morphology and dependen-cy syntax, which were used in order to crea-te vectorial representations of social mediatexts in various languages and for two diffe-rent tasks. Such features have then been pai-red with a manifold of machine learning clas-sifiers, with some neural networks and alsowith the language model BERT.Results suggest that fine-graineddependency-based syntactic informationis more informative for the detection ofirony, and less informative for what concernsstance detection. Nonetheless, dependencysyntax might still prove useful in the taskof stance detection if, firstly, irony detectionis considered as a pre-processing step. Ialso believe that the approach based ondependency syntax that I proposed couldhelp in understanding and explaining a sucha complex phenomenon like irony.In fact, the several studies presented he-re allowed to investigate whether syntacticstructures, independently from the targetlanguage, may provide information useful tounderstand whether a message is ironic ornot.Although it has been duly noted that syn-tax does not seem to be particularly infor-mative regarding directly the task of stancedetection (the second case study, presented inChapter 3). On the other hand, also suppor-ted by some previous linguistic studies, syn-tax seems to play an important role in thedetection of irony. Therefore, a new specu-lation that comes to mind, is that it couldbe more useful to perform a “cascade task”.Meaning that, firstly it might be useful topredict irony, with the help of morphosyntac-tic cues (step 1), and only then (as step 2),Dependency Syntax in the Automatic Detection of Irony and Stance147proceeding in the detection of stance. In ge-neral, my assumption, is that predicting iro-ny could be the first step in numerous othertasks, even shallow sentiment analysis, or theidentification of fake news.This outcome is something that shouldnot be ignored, but obviously carrying outsupervised studies in this sense would alsomean dedicating a great effort and consumingmuch time in the creation of annotated data-sets (that ought to be annotated on variouslayers, for different dimensions and phenome-na). In fact, to further study this line of in-vestigation, in Chapter 4, I proposed a shal-low analysis of the Italian dataset regardingthe Sardines Movement, which is only a smalland limited beginning, but it is also certainlyopening a new research perspective.My work has certainly many limitations.Firstly, I needed to deal with the scarcity ofdata annotated in some adequate way andwith the reduced size of the few datasets thatare indeed available or those I helped to deve-lop. Furthermore, this kind of investigation,mostly based on morphosyntactic cues thatare applied to NLP tasks, is a rather newone. In fact, there are very few studies goingtowards this direction up to these days.By having looked at some of the resultsobtained in the wide variety of experimentsperformed in this thesis, it is fundamental tostress that we did not solely want to appre-ciate the outcomes in terms of numerical per-formances, but rather being more focused inthe more profound linguistic reasons behindthem. And the same is valid also for why so-metimes results are poorer and why featuresdo not make improvements on a certain task.I am positive that if we manage to un-derstand what is the linguistic knowledge acertain approach, or a group of features, le-verages when it produces good (or poor) re-sults, among many possible approaches, itcould allow us to make more mature choi-ces for following work. Indeed, the future ofNLP research needs to go towards approa-ches that better integrate different types ofknowledge (such as syntactic knowledge, foronce) and that manage to be more versati-le for certain types of data and in differentapplication contexts.AcknowledgmentsThis work has been funded by the scholarship“Be Positive!” (under the 2019 “Google.orgImpact Challenge on Safety” call) and alsosupported by the European project “STE-RHEOTYPES - STudying European RacialHoaxes and sterEOTYPES” funded by theCompagnia di San Paolo and VolksWagenStiftung under the “Challenges for Europe”call for Project (CUP: B99C20000640007).ReferencesCignarella, A. T., V. Basile, M. Sanguinet-ti, C. Bosco, F. Benamara, and P. Rosso.2020. Multilingual Irony Detection withDependency Syntax and Neural Models.In Proceedings of the 28th InternationalConference on Computational Linguistics(COLING 2020). ACL.Cignarella, A. T. and C. Bosco. 2019.ATC at IroSvA 2019: Shallow SyntacticDependency-based Features for Irony De-tection in Spanish Variants. In Procee-dings of IberLEF 2019. CEUR-WS.org.Cignarella, A. T., S. Frenda, V. Basile, C. Bo-sco, V. Patti, and P. Rosso. 2018. Over-view of the EVALITA 2018 Task on Iro-ny Detection in Italian Tweets (IronI-TA). In Proceedings of EVALITA 2018.CEUR-WS.org.De Marneffe, M.-C., C. D. Manning, J. Ni-vre, and D. Zeman. 2021. Universal De-pendencies. Computational Linguistics,47(2):255–308.Lai, M., A. T. Cignarella, and D. I. Her-nandez Fariás. 2017. iTACOS at Ibe-rEval2017: Detecting Stance in Catalanand Spanish Tweets. In Proceedings ofIberEval 2017. CEUR-WS.org.Mohammad, S., S. Kiritchenko, P. Sobhani,X. Zhu, and C. Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets.In Proceedings of SemEval 2016. ACL.Sanguinetti, M., C. Bosco, L. Cassidy,Ö. Çetinoğlu, A. T. Cignarella, T. Lynn,I. Rehbein, J. Ruppenhofer, D. Seddah,and A. Zeldes. 2022. Treebanking User-generated Content: A UD based Overviewof Guidelines, Corpora and Unified Re-commendations. Language Resources andEvaluation, pages 1–52.Van Hee, C., E. Lefever, and V. Hoste. 2018.SemEval-2018 Task 3: Irony Detectionin English Tweets. In In Proceedings ofSemEval 2018. ACL.Alessandra Teresa Cignarella148

Dependency Syntax in the Automatic Detection of Irony and Stance | [Sintaxis de dependencias en la detección automática de ironía y posicionamiento]

https://iris.unito.it/bitstream/2318/1873182/1/6413-5871-1-PB.pdf

Dependency Syntax in the Automatic Detection of Irony and Stance | [Sintaxis de dependencias en la detección automática de ironía y posicionamiento]

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin