This paper describes our participation in SemEval-2015 Task 12, and the opinion mining
system sentiue. The general idea is that systems must determine the polarity of the
sentiment expressed about a certain aspect of a target entity. For slot 1, entity and attribute
category detection, our system applies a supervised machine learning classifier, for each label,
followed by a selection based on the probability of the entity/attribute pair, on that domain.
The target expression detection, for slot 2, is achieved by using a catalog of known
targets for each entity type, complemented with named entity recognition. In the opinion
sentiment slot, we used a 3 class polarity classifier, having BoW, lemmas, bigrams after
verbs, presence of polarized terms, and punctuation based features. Working in unconstrained
mode, our results for slot 1 were assessed with precision between 57% and 63%, and recall varying between 42% and 47%. In sentiment polarity, sentiue’s result accuracy was approximately 79%, reaching the best score in 2 of the 3 domains

Saias, José

Repositório Científico da Universidade de Évora

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 767–771,Denver, Colorado, June 4-5, 2015. c©2015 Association for Computational LinguisticsSentiue: Target and Aspect based Sentiment Analysisin SemEval-2015 Task 12Jose´ SaiasDI - ECT - Universidade de E´voraRua Roma˜o Ramalho, 597000-671 E´vora, Portugaljsaias@uevora.ptAbstractThis paper describes our participation inSemEval-2015 Task 12, and the opinion min-ing system sentiue. The general idea isthat systems must determine the polarity of thesentiment expressed about a certain aspect ofa target entity. For slot 1, entity and attributecategory detection, our system applies a super-vised machine learning classifier, for each la-bel, followed by a selection based on the prob-ability of the entity/attribute pair, on that do-main. The target expression detection, for slot2, is achieved by using a catalog of knowntargets for each entity type, complementedwith named entity recognition. In the opin-ion sentiment slot, we used a 3 class polarityclassifier, having BoW, lemmas, bigrams afterverbs, presence of polarized terms, and punc-tuation based features. Working in uncon-strained mode, our results for slot 1 were as-sessed with precision between 57% and 63%,and recall varying between 42% and 47%.In sentiment polarity, sentiue’s result ac-curacy was approximately 79%, reaching thebest score in 2 of the 3 domains.1 IntroductionSocial networks and other online platforms are animportant communication mechanism in currentlifestyle. These platforms aggregate user-generatedcontent, such as opinions that people write andpublish freely on the Web, and are now valuedfor market research and trend analysis. Naturallanguage processing (NLP) helps to automaticallyextract information from these written opinions.This paper describes a participation in SemEval-2015 Task 121, Aspect Based Sentiment Analysis(Pontiki et al., 2015), with the sentiue system,from Universidade de E´vora. In previous editionsof SemEval, we participated in Sentiment Analysis(SA) tasks, but in terms of overall polarity, overTwitter messages (Rosenthal et al., 2014), not beingaspect oriented. The general idea for this challenge,is that, for a text, the system must determine thepolarity of the sentiment expressed about a certainaspect of a particular target entity. Our sentiuesystem is an evolution from our previous work(Saias and Fernandes, 2013; Saias, 2014), for targetoriented SA.Task 12 was run in two phases. In phase A systemsare tested for aspect detection with one slot to aspectcategory, and a second slot for the opinion targetexpression on the text. Test data includes reviewtexts for two domains: restaurants and laptops. Inphase B, aspect category is provided, and systemsmust assign a polarity (positive, negative, or neutral)for each opinion. In this phase, systems receivedalso texts from a third domain, hotels, for which nosentiment training data was given.We used a supervised machine learning classifiercombined with a probability based selection pro-cess, for entity and attribute category detection, onslot 1. Target expression detection was performedwith an entity catalog, filled with known targetsfor each entity type, and named entity recognition(NER). For the sentiment polarity slot, we used a asupervised machine learning classifier, having bag-of-words (BoW), lemmas, bigrams after verbs, and1http://alt.qcri.org/semeval2015/task12/767punctuation based features, along with sentimentlexicon based features. The detailed procedure isexplained in section 3.2 Related WorkMany SA related publications, originating both inindustry and in academia, have appeared, and it isnotorious the growing interest by companies. Pop-ular scientific forums and events include activitiesand workshops on this area, such as RepLab (Amigo´et al., 2014) at CLEF2, for online reputation, orABSA and Twitter SA tasks in SemEval.In last year’s edition of this SemEval task (Pontiki etal., 2014), there were 26 systems participating in thepolarity subtask. The two systems with better polar-ity classification accuracy were from NRC-Canadaand DCU teams. NRC-Canada system (Kiritchenkoet al., 2014) was trained with the data provided inthe task, and complemented with lexicons generatedfrom other corpora of customer reviews, to helpfeature extraction in machine learning. StanfordCoreNLP was used to tokenize, POS tagging, anddependency parse trees. They address polarityclassification with a linear SVM classifier, withfeatures for: the target, and its surrounding words;POS based features; dependency tree based features;unigrams and bigrams; lexicon based features. TheDCU system (Wagner et al., 2014) also uses SVMfor aspect and for polarity classification, combiningbag-of-n-gram features with rule-based features.N-grams (with size from 1 to 5) in a window aroundthe aspect term, are used as features, as well asfeatures derived from a sentiment lexicon. Therule-based approach to predict the polarity of anaspect term, generated features considering allwords score and their distance to the aspect term.3 MethodOur participation involved the adaptation of our pre-vious real-time system, for text overall sentimentclassification, into a target oriented SA system. Thenext subsections explain how the system works, foreach part of Task 12 challenge.2http://clef2014.clef-initiative.eu/3.1 Aspect Entity and AttributeThe first annotation task focuses on aspect category.This category is an entity and attribute pair, eachchosen from an inventory with possible values, ineach domain, for entity types and attributes. Sincethe possible category types are known and limited,we decided to use a classifier for each entity type(e.g. food, laptop) and for each attribute label (e.g.price, quality). Our approach comprises two stages.The first processes each review sentence assigningto it zero, one, or more entity types and attribute la-bels. The second stage chooses and combines iden-tified entities and attributes, forming the aspect an-notation. Analyzing the training data, we found thatin the same sentence, there may be opinions on var-ious types of entity (e.g. CPU, battery) or attributes.Thus, we have chosen to train a classifier for eachentity type, and a classifier for each attribute la-bel. We set a supervised machine learning text clas-sifier, using MALLET (McCallum, 2002), a Java-based tool for NLP, with machine learning applica-tions to text. For the purpose of this stage, it wasnecessary to prepare the training data for each bi-nary classifier, that would determine whether a sen-tence contains an opinion on its tag (entity type or at-tribute label). The train process was the same for alltags, entity type or attribute label, of each domain.We created a dataset where each instance is a sen-tence text, and its class is tag, if the sentence had atleast one opinion with that tag, or no tag otherwise.Text preprocessing includes tokenization, POS tag-ging and lemmatization, all performed with StanfordCoreNLP (Toutanova et al., 2003; Manning et al.,2014) tool. The classifier algorithm was MaximumEntropy3, and the classifier model features were textwords and lemmas.Second stage starts with each sentence annotatedwith a set and tags, some for entity type and somefor attribute label. When a sentence has no annota-tions, the system assumes that there is no opinion. Incase of 1 tag on entity type and 1 tag in the attributelabel, then it is the trivial case where the junctionof the two results in the aspect annotation. For sen-tences with 1 tag on entity type and 0 tags for theattribute, our system searches for the most frequentaspect annotation, within the sentence domain, that3MALLET class: cc.mallet.classify.MaxEnt768includes that entity type. The equivalent is appliedin the case of 0 tags to entity and 1 tag for the at-tribute label. If both sides have one or more tags,the system applies a cycle, where each loop iterationforms the more frequent pair (entity,attribute) in thatdomain, and removes these two tags from the sen-tence tag set. This is repeated until the first, entityor attribute side, exhausts the tags provided by theprevious stage classifier. And if some tags are left,on the opposite side of the pair, the system applies,for each, the same process already explained for case0-1 or 1-0.3.2 Opinion Target ExpressionAt this point, sentences are already marked as hav-ing (or not) opinions on certain aspect category.For each opinion on restaurants domain, the sys-tem needed to identify the entity mention on thesentence text, referred to as the opinion target ex-pression (OTE). We collected the opinion targets foreach entity type, from the training data, forming acatalog. If any of the targets already known (e.g.restaurant name, or meal) appears in the sentencetext, next to a verb or adjective, it is chosen as theOTE. If this does not lead to any OTE candidate, oursystem applies named entity recognition, looking forreferences to organization and location entities, us-ing Stanford NER tool (Finkel et al., 2005; Manninget al., 2014). Having found one OTE, through thecatalog or by NER, its text and position are markedin slot 2. If no mention is found, OTE slot is filledwith the NULL value.3.3 Sentiment PolarityPhase B was held in a subsequent period, and theinput given to the systems is a little different, hav-ing the correct annotations on the aspect category,in restaurants, laptops and hotels domains. For eachopinion, the participating systems must assign a sen-timent polarity (positive, negative or neutral), con-sidering the opinion aspect.For training, there were 1654 opinions on restau-rants domain, and 1974 more opinions about lap-tops, all annotated for polarity. No sentiment train-ing data was given for hotels domain. Consideringthe available data, and the objective of this phase, weused a supervised machine learning classifier to pre-dict each opinion polarity. Instead of multiple classi-fiers, such as implemented for slot 1, we prepared asingle classifier, thought, as before, for text but tunedwith a different model, so that it can choose betweenpositive, negative or neutral polarity.Sentences without opinion are not considered in thetraining, because here the polarity is associated withopinions. Further, a single sentence may have sev-eral opinions about different aspects, and each mayhave a different and independent polarity. To trainthe classifier, for each opinion we created a polaritydata instance, containing the sentence text, its do-main, its aspect entity and attribute, OTE (if avail-able, in restaurants), and the opinion polarity to belearned. As before, MALLET was used with a Max-imum Entropy classifier. The sentence text prepro-cessing was the same we did for aspect categoryclassification. The features to represent each in-stance were:• BoW with a feature for each token text;• lemmas for verbs and adjectives;• bigram after verb (lemmatized);• presence of negation terms;• bigram after negation term;• presence of exclamation/question mark;• presence of polarized terms (positive or nega-tive), according to each sentiment lexicon;• whether there are polarized terms before excla-mation mark and question mark;• bigram before, and after, any polarized term;• polarity inversion, by negation detection beforesome polarized term;• presence of polarized terms in the last 5 tokens;• a feature for the domain, and two features forthe entity type and the attribute label.To see whether a term is polarized, each token textis verified in each sentiment lexicon. These polaritysupport resources are AFINN lexicon (Nielsen,2011), Bing Liu’s opinion lexicon (Liu et al., 2005)and MPQA subjectivity clues (Wiebe et al., 2005).769Domain Precision Recall F-measurerestaurants 0,633 0,472 0,541laptops 0,577 0,441 0,500Table 1: sentiue’s evaluation on aspect category.Domain Precision Recall F-measurerestaurants 0,488 0,336 0,398Table 2: sentiue’s evaluation on target detection.After some experimentation, we decided to use asingle full train, joining the instances of restaurantsand laptops as a whole training set. The resultingmodel was used to classify the opinion polarity forthe three domains.Because we used sentiment lexicons, our systemoperates in unconstrained mode. These additionalresources served as support for features extraction.No supplementary training texts were used. In ourdevelopment testing, we obtained an 80% accuracyfor polarity. After this, the result is written in XMLformat for submission.4 ResultsThe phase A test data had 685 sentences on restau-rants domain and 761 on laptops domain. With themethod described above, the sentiue system ex-tracted 596 opinion categories for restaurants do-main and 751 other for laptops domain.Table 1 shows the evaluation for slot 1. Among the15 submissions evaluated in the first domain, thebest system F-score value was 0,627, while our re-sult F-score was 0,541. For laptops aspect category,sentiue’s scores were lower, but improving in thecomparison with other systems, achieving the sec-ond best F-measure, out of 9 evaluated submissions.The evaluation of our result in opinion target expres-sion in given in Table 2. This was a poor result,when compared with the 0,524 average F-measureof the 21 submissions for this slot.In phase B systems had to fill the slot 3, with senti-ment polarity to 845 opinions on restaurants domain,949 opinions on laptops domain, and 339 opinionson hotels domain.On Table 3 we find our system’s result accuracy, inthe two trained domains plus the untrained hotels do-Domain Accuracyrestaurants 0,787laptops 0,793hotels 0,788Table 3: sentiue accuracy on sentiment polarity.Domain,Polarity Precision Recall F-measurerestaurants, positive 0,767 0,914 0,834restaurants, negative 0,825 0,708 0,762restaurants, neutral 0,714 0,111 0,192laptops, positive 0,831 0,891 0,860laptops, negative 0,766 0,787 0,777laptops, neutral 0,387 0,152 0,218hotels, positive 0,887 0,840 0,863hotels, negative 0,608 0,738 0,667hotels, neutral 0,143 0,083 0,105Table 4: sentiue’s SA evaluation per domain.main. In this slot we got the most satisfactory re-sult, with the best accuracy in restaurants and lap-tops, and an above average score, in the hotels do-main. The detailed evaluation is shown in Table 4,with values for precision, recall and f-measure, perdomain and polarity class.5 ConclusionsBy participating in this SemEval edition, we soughtto develop our previous work, in order to achieveSA results focused on the opinion targets.Our results were poor for OTE detection, but wethink it will be easy to correct the implementationproblems for that part. As example, while checkingif a sentence contained a known target, from thecatalog, the system did not require whole words tobe matched, and this led to some misidentificationof word substrings as target.Our result was more satisfactory for slot 1, with aF-measure slightly above average between the 15evaluated submissions for restaurants domain, and4.5% better than submissions average for laptopsdomain. The distribution of opinions for each aspectcategory is not uniform. For example, for attributelabel classification, we already know that QUALITYand GENERAL have much more instances than otherlabels. This analysis inspired our approach in thesecond stage, explained in section 3.1. To improvethis part, we think to introduce a cascade classifier.770After the classification obtained in the current firststage, other machine learning classifier will decidehow to pair entity+attribute, based on the wordingof the sentence. Another future work idea is to usemore corpora for training the aspect classifiers, asother systems (Kiritchenko et al., 2014) have tried.In phase B sentiue achieved good results. This,perhaps, is justified by our previous experience inoverall SA. Many of the polarity classifier featuresare inherited from our former system. SemEvalchallenge is always a motivation to test our systemand an opportunity to learn from other participants.AcknowledgmentsWe would like to thank to LabInterop project, forproviding the infrastructure for the developed sys-tem. LabInterop is funded by Programa Opera-cional Regional do Alentejo (INALENTEJO).ReferencesEnrique Amigo´, Jorge Carrillo-de-Albornoz, IrinaChugur, Adolfo Corujo, Julio Gonzalo, Edgar Meij,Maarten de Rijke, Damiano Spina. 2014. Overviewof RepLab 2014: Author Profiling and Reputation Di-mensions for Online Reputation Management. In In-formation Access Evaluation. Multilinguality, Multi-modality, and Interaction. Lecture Notes in ComputerScience, Volume 8685, 2014, pp 307-322.Jenny Rose Finkel, Trond Grenager, and ChristopherManning. 2005. Incorporating Non-local Informationinto Information Extraction Systems by Gibbs Sam-pling. Proceedings of the 43nd Annual Meeting ofthe Association for Computational Linguistics (ACL2005), pp. 363-370.Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, andSaif M. Mohammad. 2014. NRC-Canada-2014: De-tecting Aspects and Sentiment in Customer Reviews.Proceedings of the 8th International Workshop onSemantic Evaluation (SemEval 2014). p. 437–442.Dublin, Ireland, August 2014.Bing Liu, Minqing Hu and Junsheng Cheng. 2005. Opin-ion Observer: Analyzing and Comparing Opinionson the Web. In Proceedings of the 14th InternationalWorld Wide Web conference (WWW-2005). Chiba,Japan.Manning, Christopher D., Surdeanu, Mihai, Bauer, John,Finkel, Jenny, Bethard, Steven J., and McClosky,David. 2014. The Stanford CoreNLP Natural Lan-guage Processing Toolkit. In Proceedings of 52ndAnnual Meeting of the Association for ComputationalLinguistics: System Demonstrations, pp. 55-60.Andrew Kachites McCallum. 2002. MALLET: A Ma-chine Learning for Language Toolkit.Finn A˚rup Nielsen. 2011. A New ANEW: Evaluation ofa Word List for Sentiment Analysis in Microblogs.In Proceedings, 1st Workshop on Making Sense ofMicroposts (#MSM2011): Big things come in smallpackages. pp: 93-98. Greece.Maria Pontiki, D. Galanis, J. Pavlopoulos, H. Papageor-giou, I. Androutsopoulos, S. Manandhar. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. Pro-ceedings of the 8th SemEval, 2014. Dublin, Ireland.Maria Pontiki, Dimitrios Galanis, Haris Papageogiou,Suresh Manandhar, and Ion Androutsopoulos. 2015.SemEval-2015 Task 12: Aspect Based SentimentAnalysis. In Proceedings of the 9th InternationalWorkshop on Semantic Evaluation (SemEval 2015),Denver, Colorado.Sara Rosenthal, Alan Ritter, Veselin Stoyanov, andPreslav Nakov. 2014. SemEval-2014 Task 9: Senti-ment Analysis in Twitter. In Proceedings of the EighthInternational Workshop on Semantic Evaluation (Se-mEval’14). August 23-24, 2014, Dublin, Ireland.Jose´ Saias and Hila´rio Fernandes. 2013. Senti.ue-en:An approach for informally written short texts inSemEval-2013 Sentiment Analysis task. In SecondJoint Conference on Lexical and Computational Se-mantics (*SEM), Volume 2: Proceedings of the Sev-enth International Workshop on Semantic Evaluation(SemEval 2013), pages 508–512, Atlanta, Georgia,USA.Jose´ Saias. Senti.ue: Tweet overall sentiment classifica-tion approach for SemEval-2014 task 9. In Proceed-ings of the 8th International Workshop on SemanticEvaluation (SemEval 2014), pages 546–550, Dublin,Ireland, August 2014. ISBN 978-1-941643-24-2.Kristina Toutanova, Dan Klein, Christopher Manning,and Yoram Singer. 2003. Feature-Rich Part-of-SpeechTagging with a Cyclic Dependency Network. In Pro-ceedings of HLT-NAACL 2003, pp. 252-259.Joachim Wagner, Piyush Arora, Santiago Cortes, UtsabBarman, Dasha Bogdanova, Jennifer Foster and LamiaTounsi. 2014. DCU: Aspect-based Polarity Classifica-tion for SemEval Task 4. Proceedings of the 8th Inter-national Workshop on Semantic Evaluation (SemEval2014). p. 223–229. Dublin, Ireland, August 2014.Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.Annotating expressions of opinions and emotionsin language. Language Resources and Evaluation,39:165–210.771

Sentiue: Target and Aspect based Sentiment Analysis in SemEval-2015 Task 12

Abstract

Similar works

Full text

Available Versions

Repositório Científico da Universidade de Évora

Crossref