Este artículo presenta la Tarea sobre Análisis de Sentimientos basado en Aspectos en Portugués (ABSAPT), realizada en el IberLEF 2022. Les pedimos a los participantes que desarrollaran sistemas capaces de identificar aspectos (AE) y extraer la polaridad (ASC) en textos escritos en portugués. Doce equipos se inscribieron en la tarea, entre los cuales cinco presentaron predicciones e informes técnicos. El sistema con mejor rendimiento logró un valor de precisión (Acc) de 0,67 para la subtarea de AE (Equipo Deep Learning Brasil) y un valor de precisión equilibrada (Bacc) de 0,82 para la subtarea de ASC (Equipo Deep Learning Brasil).This paper presents the task on Aspect-Based Sentiment Analysis in Portuguese (ABSAPT), held within Iberian Languages Evaluation Forum (IberLEF 2022). We asked the participants to develop systems capable of extracting aspects (AE) and classifying sentiment of aspects (ASC) in texts. We created one corpora containing reviews about hotels. Twelve teams registered to the task, among which five submitted predictions and technical reports. The best performing system achieved an Accuracy (Acc) value of 0.67 in AE sub-task (Team Deep Learning Brasil) and a Balanced Accuracy (Bacc) value of 0.82 in ASC sub-task (Team Deep Learning Brasil).This work was financed in part by the following Brazilian research agencies: CAPES and CNPq

Araujo, Ricardo M.

Corrêa, Ulisses B.

da Silva, Felix L.V.

de Freitas, Larissa A.

dos Santos, Leonardo P.

Mensenburg, Heliks M.

Rodrigues, Rodrigo F.

Xavier, Guilherme da S.

English

Repositorio Institucional de la Universidad de Alicante

ABSAPT 2022 at IberLEF: Overview of the Task onAspect-Based Sentiment Analysis in PortugueseResumen de la Tarea de Análisis de Sentimientos Basado enAspectos en Portugués (ABSAPT) en IberLEF 2022Felix L. V. da Silva1, Guilherme da S. Xavier1, Heliks M. Mensenburg1,Rodrigo F. Rodrigues1, Leonardo P. dos Santos3, Ricardo M. Araújo1 2,Ulisses B. Corrêa1 2, Larissa A. de Freitas1 21 Center for Technological Development (CDTec),Federal University of Pelotas (UFPel), Pelotas, RS, Brazil2 Artificial Intelligence Innovation Hub (H2IA),Federal University of Pelotas, Pelotas, RS, Brazil3 University of São Paulo, São Paulo, SP, Brazil{flvdsilva, gdsxavier, hmmersenburg, rfrodrigues, ricardo, ub.correa, larissa}@inf.ufpel.edu.br,leonardosantsper@gmail.comResumen: Este art́ıculo presenta la Tarea sobre Análisis de Sentimientos basadoen Aspectos en Portugués (ABSAPT), realizada en el IberLEF 2022. Les pedimosa los participantes que desarrollaran sistemas capaces de identificar aspectos (AE)y extraer la polaridad (ASC) en textos escritos en portugués. Doce equipos seinscribieron en la tarea, entre los cuales cinco presentaron predicciones e informestécnicos. El sistema con mejor rendimiento logró un valor de precisión (Acc) de0,67 para la subtarea de AE (Equipo Deep Learning Brasil) y un valor de precisiónequilibrada (Bacc) de 0,82 para la subtarea de ASC (Equipo Deep Learning Brasil).Palabras clave: Análisis de Sentimiento basado en Aspectos, Portugués, Reseñasde Hoteles.Abstract: This paper presents the task on Aspect-Based Sentiment Analysis inPortuguese (ABSAPT), held within Iberian Languages Evaluation Forum (IberLEF2022). We asked the participants to develop systems capable of extracting aspects(AE) and classifying sentiment of aspects (ASC) in texts. We created one cor-pora containing reviews about hotels. Twelve teams registered to the task, amongwhich five submitted predictions and technical reports. The best performing sys-tem achieved an Accuracy (Acc) value of 0.67 in AE sub-task (Team Deep LearningBrasil) and a Balanced Accuracy (Bacc) value of 0.82 in ASC sub-task (Team DeepLearning Brasil).Keywords: Aspect-Based Sentiment Analysis, Portuguese, Hotel Reviews.1 IntroductionSentiment Analysis (SA) is the field of Nat-ural Language Processing (NLP) that au-tomatically analyzes people’s sentiments oropinions towards some entity. These senti-ments can be valuable sources of informationabout the consumer’s feelings about a par-ticular product or idea, which can help indecisions by companies or governments (Liu,2015).SA can be done on different levels, focus-ing mainly on three possible granularity lev-els: document level, sentence level, and as-pect level (de Freitas, 2015). At the aspectlevel, it is possible to analyze different opin-ions held towards different aspects of someentity or different entities in the same docu-ment or sentence.The ABSAPT 2022 task aims to challengedifferent teams to propose techniques capa-Procesamiento del Lenguaje Natural, Revista nº 69, septiembre de 2022, pp. 199-205 recibido 05-07-2022 revisado 22-07-2022 aceptado 25-07-2022ISSN 1135-5948. DOI 10.26342/2022-69-17 © 2022 Sociedad Española para el Procesamiento del Lenguaje Naturalble of extracting aspects and classifying sen-timent of aspects in the hotel reviews. Thepresent paper presents an overview of thetask. First, we briefly present some theoreti-cal reflections Aspect-Based Sentiment Anal-ysis (ABSA) (Section 2) and describe theproposal of our task (Section 3). Section 4presents the corpora description and the an-notation process. In Section 5, we describethe evaluation measures. Participant systemsand the results are discussed in Section 6. Fi-nally, the final remarks are done in Section 7.2 Aspect-Based SentimentAnalysisOn this granularity level, all opinions ex-pressed towards any aspect of the entity areanalyzed individually. This level allows a bet-ter understanding of the opinions and entitiesin the text. To accomplish the analysis onthis level, the task used to be broken intotwo sub-tasks: Aspect Extraction (AE) andAspect Sentiment Classification (ASC).2.1 Aspect ExtractionThis sub-task determines which aspects of agiven entity are considered in a text.For example, in the sentence “Hotel comboa localização” [“Hotel with good local-ization”], the goal of AE would be to iden-tify the aspect ‘localização’.2.2 Aspect SentimentClassificationThis sub-task consists of the classification ofthe polarity for each aspect that has beenidentified in the text.For example, in the sentence “Hotel comboa localização” [“Hotel with good local-ization”], the goal of ASC would be to clas-sify the aspect ‘localização’ as positive.3 Task DescriptionPeople’s opinions are a great source of infor-mation for other people and organizations,public or private. Typically works focusedon Portuguese perform document level SA. Itis hard to find ABSA approaches or datasetsavailable for Portuguese.We propose to create an ABSA for Tri-pAdvisor reviews written in Portuguese. Twosub-tasks will be available: AE and ASC. Thefirst sub-task comprehends the identificationof aspects in the reviews, and the second sub-task proposes to extract the sentiment orien-tation (polarity) of the review about a singleaspect mentioned in it.The availability of corpora written in Por-tuguese is scarce, which limits the amount ofresearch done for this language.This task will contribute to the progressof Portuguese NLP, as there is a demand fordeveloping new methods and tools.Previous ABSA competitions, such as Se-mEval 2014 (Pontiki et al., 2014), SemEval2015 (Pontiki et al., 2015), SemEval 2016(Pontiki et al., 2016) and EVALITA (Matteiet al., 2020) inspired us to develop a specifictask for Portuguese.4 Corpora Description andAnnotation ProcessThis section describes the corpora proposedfor evaluation and the annotation process(annotation guidelines and inter-annotatoragreement).4.1 Corpora DescriptionThe corpora contain travellersŕeviews aboutaccommodation services companies, writtenin Portuguese. In ABSAPT 2022, we usedcorpora developed previously by Freitas (deFreitas, 2015) and Corrêa (Corrêa, 2021).Freitasćorpus is publicly available, so it willbe used only in the training dataset (3111samples from 847 reviews). Corrêa’s cor-pus is private and will be split into train-ing and test dataset (257 samples to AE and686 samples to ASC). The full dataset willbe available after the event on the websitehttp://absapt2022.tk/.Both datasets were annotated followingthe same annotation guidelines (de Freitas,2015). The concepts on the Accommoda-tion Services Domain Ontology, HOntology(Chaves, de Freitas, and Vieira, 2012) wereaspects annotated. HOntology contains 282concepts categorized into 16 top-level con-cepts. The concept hierarchy has a maximumdepth of 5.In Figure 1, one can see an overview of thetraining dataset. The full training datasetcontains 77 aspects. Due to space limitation,we present the 40 most frequent aspects andthe polarities distribution. In Figure 2, wepresent an overview of the test dataset.4.2 Annotation ProcessThe manual annotation of Freitas’ corpus(training dataset) was conducted by two an-Felix L. V. da Silva, Guilherme da S. Xavier, Heliks M. Mensenburg, Rodrigo F. Rodrigues, Leonardo P. dos Santos, Ricardo M. Araújo, Ulisses B. Corrêa, Larissa A. de Freitas200Figure 1: Training dataset: polarity frequency for aspects with at least 10 samples.notators, both native speakers of Portuguese,one linguist, and one computer scientist. Andthe manual annotation of Corrêa’s corpus(training and test dataset) was conducted bytwelve students and professors of computerscience and engineering annotators. Bothused the tool developed in the context of (deFreitas, 2015).The agreement between annotators (Fre-itas’ corpus) was measured with KappaStatistics (Landis and Koch, 1977). The an-notators agreement about ABSA from do-ABSAPT 2022 at IberLEF: Overview of the Task on Aspect-Based Sentiment Analysis in Portuguese201Figure 2: Test dataset: polarity frequency for aspects with at least 10 samples.main ontology using Kappa was 0.58 for ex-plicit aspects, which is considered a moderateagreement. We believe that the annotationhas an acceptable Kappa value. It is also im-portant to note that only in a few cases theannotators disagreed between negative andpositive polarities, the majority of disagree-ments was about positive and neutral polar-ities, or negative and neutral polarities.The agreement between annotators(Corrêa’s corpus) was measured with FleissKappa (Fleiss, 1971) suitable for more thantwo annotators. The majority annotatorgroup share k > 0.4 (moderate agreement).5 Evaluation MeasuresThe training set was released on April 08th,and participants had sixteen days to traintheir systems. The test set was released onApril 24th, and each participant had twelvedays to submit one run.Participating teams will receive trainingand test datasets. The latter was sent with-out the label of the samples.We evaluated the predictions sent by theparticipants using several metrics: Acc (Eq.1), Precision (Eq. 2), Recall (Eq. 3), F1(Eq. 4), and Bacc (Eq. 6). Bacc to rankcompetitors. The submissions will be rankedaccording to Acc in AE sub-task and Bacc inASC sub-task.Acc =True Positives + True NegativesTotal Number of Instances(1)Precision =True PositivesTrue Positives + True Negatives(2)Recall =True PositivesTrue Positives + False Negatives(3)F1 = 2× Precision ·RecallPrecision + Recall(4)Specificity =True NegativesTrue Negatives + False Positives(5)Bacc =(Recall + Specificity)2(6)6 Participants Systems andDiscussion of the ResultsTwelve teams registered for the task, amongwhich five submitted predictions and techni-cal reports. Participants are from universitiesFelix L. V. da Silva, Guilherme da S. Xavier, Heliks M. Mensenburg, Rodrigo F. Rodrigues, Leonardo P. dos Santos, Ricardo M. Araújo, Ulisses B. Corrêa, Larissa A. de Freitas202and institutes in Brazil (UFG, UFPI, IFPI,UFSCAR, USP, UFPR and UESC).Participants used rules and lexicon (TeamUFSCAR), traditional machine learning ap-proaches as CRF (Team NILC and TeamUFPR), and deep learning methods as Trans-formers (Team Deep Learning Brasil, TeamPiLN, and Team UFPR).Tables 1 and 2 present participants’ re-sults for each sub-task submitted run. Theresults are ranked according to the Acc inAE and Bacc in ASC.Accuracy Team0.67 TeamDeepLearningBrasil0.65 TeamPiLN0.59 TeamUFSCAR0.22 TeamNILC0.17 TeamUFPRTable 1: Participants results ranked in termsof Acc in AE sub-task.For each system, best run is highlightedin bold. Team Deep Learning Brasil, usedtransformers to achieve an Acc of 0.67 in AEand a Bacc of 0.82 in ASC.Below we summarize the proposed ap-proach of each team:• Team Deep Leaning Brasil: The au-thors proposed different methodologiesfor both sub-tasks of ABSA. The AEused a single sentence tagging approach,and the ASC tested with two differ-ent strategies, one as a Sentence PairClassification and the other as a Con-ditional Text Generation. In addition,also were used other ABSA datasets suchas Evalita, MAMs, Semeval 2014, 2015,and 2016 competition, and BidirectionalEncoder Representations from Trans-formers (BERT) pre-trained models onthe Portuguese language and multilin-gual. The proposed approach reachedthe best-performing systems, achievingnew state-of-the-art results on both sub-tasks (Gomes et al., 2022).• Team PiLN: The authors proposedsimple and well-known approaches tothe sub-tasks of ABSA. The AE useda string-match strategy using a mul-tilingual ontology for the accommoda-tion sector named HOntology. The ASCused a BERTimbau (BERT pre-trainedmodel on the Portuguese language), ap-proaching the reviews as a Sentence PairClassification. The proposed approachreached the second best-performing sys-tem, achieving the following results: Accof 0.65 in AE, Bacc of 0.78, F1 of 0.77,Precision of 0.76, Recall of 0.78 in ASC(Neto et al., 2022).• Team UFSCAR: The AE sub-task in-cludes preprocessing, tokenization, fea-ture extraction, a lexicon, and rule-based aspect identification. The ASCsub-task has two main steps: meaningfulsurroundings extraction and sentimentextraction using GoEmotions (Dem-szky et al., 2020), followed by polar-ity extraction. The proposed approachreached the third best-performing sys-tem, achieving the following results: Accof 0.59 in AE, Bacc of 0.62, F1 of 0.61,Precision of 0.65, Recall of 0.62 in ASC(Assi et al., 2022).• Team NILC: The authors participatedonly in AE sub-task. Its approach isbased on the Conditional Random Fields(CRF) machine learning algorithm com-bined with a post-processing step. Af-ter applying the method, the authorsperformed an error analysis of detectedand non-detected aspects (Machado andPardo, 2022).• Team UFPR: In the AE used CRF,the dataset was adapted with tokeniza-tion and the POS Tagging technique,and a pre-trained model for Portuguesewas used for the POS Tagging process.If the POS Tagging process has a bet-ter adaptation for Portuguese, there willbe a gain in results for the CRF perfor-mance. In the ASC used BERTimbau(Heinrich and Marchi, 2022). The pro-posed approach reached the worst run,achieving the Acc of 0.17 in AE.7 Final RemarksMotivated by the necessity of improvementsin the ABSA task focused on Portuguese, weproposed a task within the IberLEF 2022.This paper overviews the first task on ABSAin Portuguese to identify aspects and extractthe polarity in hotel reviews.The datasets (training and test) have beenmanually annotated. The inter-annotatorABSAPT 2022 at IberLEF: Overview of the Task on Aspect-Based Sentiment Analysis in Portuguese203Bacc F1 Precision Recall Team0.82 0.81 0.81 0.82 TeamDeepLearningBrasil0.78 0.77 0.76 0.78 TeamPiLN0.62 0.61 0.65 0.62 TeamUFSCAR0.62 0.61 0.65 0.62 TeamUFPRTable 2: Participants results ranked in terms of Bacc in ASC sub-task.agreement for training and test dataset isconsidered moderate.The deep learning methods based onTransformers performed better than ap-proaches based on rules and lexicons. TheDeep Learning Brasil Team achieved a Baccof 0.67 for the AE and Acc of 0.82 for theASC, while the UFSCAR Team achieved aBacc of 0.59 for the AE and Acc of 0.62 forthe ASC.AcknowledgmentsThank you to all annotators for their essen-tial work. We gratefully acknowledge thesupport of NVIDIA Corporation with the do-nation of the Titan X Pascal GPU used forthis research. This work was financed in partby the following Brazilian research agencies:CAPES and CNPq.ReferencesAssi, F. M., G. B. Candido, L. N.dos Santos Silva, D. F. Silva, andH. de Medeiros Caseli. 2022. Ufs-car’s team at absapt 2022: Using syn-tax, semantics and context for solvingthe tasks. In Proceedings of the IberianLanguages Evaluation Fórum (IberLEF2022), co-located with the 38th Conferenceof the Spanish Society for Natural Lan-guage Processing (SEPLN 2022), Online.CEUR.org.Chaves, M. S., L. A. de Freitas, and R. Vieira.2012. Hontology: A multilingual on-tology for the accommodation sector inthe tourism industry. In J. Filipe andJ. L. G. Dietz, editors, KEOD, pages 149–154. SciTePress.Corrêa, U. B. 2021. Análise de sentimentobaseada em aspectos usando aprendizadoprofundo: uma proposta aplicada à ĺınguaportuguesa. Ph.D. thesis, UniversidadeFederal de Pelotas, Pelotas.de Freitas, L. A. 2015. Feature-level sen-timent analysis applied to Brazilian Por-tuguese reviews. Ph.D. thesis, Pontif́ıciaUniversidade Católica do Rio Grande doSul, Porto Alegre.Demszky, D., D. Movshovitz-Attias, J. Ko,A. Cowen, G. Nemade, and S. Ravi. 2020.GoEmotions: A dataset of fine-grainedemotions. In Proceedings of the 58th An-nual Meeting of the Association for Com-putational Linguistics, pages 4040–4054,Online, July. Association for Computa-tional Linguistics.Fleiss, J. L. 1971. Measuring nominal scaleagreement among many raters. Psycho-logical Bulletin, 76(5):378–382.Gomes, J. R. S., R. C. Rodrigues, E. A. S.Garcia, A. F. B. Junior, D. F. C. Silva,and D. F. Maia. 2022. Deep learningbrasil at absapt 2022: Portuguese trans-former ensemble approaches. In Proceed-ings of the Iberian Languages EvaluationFórum (IberLEF 2022), co-located withthe 38th Conference of the Spanish Societyfor Natural Language Processing (SEPLN2022), Online. CEUR.org.Heinrich, T. and F. Marchi. 2022. Tea-mufpr at absapt 2022: Aspect extrac-tion with crf and bert. In Proceedings ofthe Iberian Languages Evaluation Fórum(IberLEF 2022), co-located with the 38thConference of the Spanish Society for Nat-ural Language Processing (SEPLN 2022),Online. CEUR.org.Landis, J. R. and G. G. Koch. 1977. Themeasurement of observer agreement forcategorical data. Biometrics, 33(1).Liu, B. 2015. Sentiment Analysis: Min-ing Opinions, Sentiments, and Emotions.Cambridge University Press.Machado, M. T. and T. A. S. Pardo. 2022.Nilc at absapt 2022: Aspect extraction forportuguese. In Proceedings of the IberianLanguages Evaluation Fórum (IberLEF2022), co-located with the 38th ConferenceFelix L. V. da Silva, Guilherme da S. Xavier, Heliks M. Mensenburg, Rodrigo F. Rodrigues, Leonardo P. dos Santos, Ricardo M. Araújo, Ulisses B. Corrêa, Larissa A. de Freitas204of the Spanish Society for Natural Lan-guage Processing (SEPLN 2022), Online.CEUR.org.Mattei, L. D., G. D. Martino, A. Iovine,A. Miaschi, M. Polignano, and G. Ram-belli. 2020. Ate absita@ evalita2020:Overview of the aspect term extractionand aspect-based sentiment analysis task.Proceedings of the 7th Evaluation Cam-paign of Natural Language Processing andSpeech tools for Italian (EVALITA 2020),Online. CEUR.org.Neto, F. A. R., R. F. de Sousa, R. L.de S. Santos, R. T. Anchiêta, and R. S.Moura. 2022. Piln at absapt 2022: Lex-ical and bert strategies for aspect-basedsentiment analysis in portuguese. In Pro-ceedings of the Iberian Languages Evalu-ation Fórum (IberLEF 2022), co-locatedwith the 38th Conference of the SpanishSociety for Natural Language Processing(SEPLN 2022), Online. CEUR.org.Pontiki, M., D. Galanis, H. Papageor-giou, I. Androutsopoulos, S. Manandhar,M. AL-Smadi, M. Al-Ayyoub, Y. Zhao,B. Qin, O. D. Clercq, V. Hoste, M. Apid-ianaki, X. Tannier, N. Loukachevitch,E. Kotelnikov, N. Bel, S. M. Jiménez-Zafra, and G. Eryiğit. 2016. SemEval-2016 task 5: Aspect based sentiment anal-ysis. In Proceedings of the 10th Interna-tional Workshop on Semantic Evaluation(SemEval-2016), pages 19–30, San Diego,California, June. Association for Compu-tational Linguistics.Pontiki, M., D. Galanis, H. Papageorgiou,S. Manandhar, and I. Androutsopoulos.2015. SemEval-2015 task 12: Aspectbased sentiment analysis. In Proceedingsof the 9th International Workshop on Se-mantic Evaluation (SemEval 2015), pages486–495, Denver, Colorado, June. Associ-ation for Computational Linguistics.Pontiki, M., D. Galanis, J. Pavlopoulos,H. Papageorgiou, I. Androutsopoulos, andS. Manandhar. 2014. SemEval-2014task 4: Aspect based sentiment analy-sis. In Proceedings of the 8th Interna-tional Workshop on Semantic Evaluation(SemEval 2014), pages 27–35, Dublin, Ire-land, August. Association for Computa-tional Linguistics.ABSAPT 2022 at IberLEF: Overview of the Task on Aspect-Based Sentiment Analysis in Portuguese205206

Resumen de la Tarea de Análisis de Sentimientos Basado en Aspectos en Portugués (ABSAPT) en IberLEF 2022

http://rua.ua.es/dspace/bitstream/10045/127429/1/PLN_69_17.pdf

Resumen de la Tarea de Análisis de Sentimientos Basado en Aspectos en Portugués (ABSAPT) en IberLEF 2022

Abstract

Similar works

Full text

Available Versions

Repositorio Institucional de la Universidad de Alicante