This paper reports the description and perfor- mance of our system, FBK-HLT, participating in the SemEval 2015, Task #3 "Answer Se- lection in Community Question Answering" for English, for both subtasks. We submit two runs with different classifiers in combining typ- ical features (lexical similarity, string similar- ity, word n-grams, etc.) with machine transla- tion evaluation metrics and with some ad hoc features (e.g user overlapping, spam filtering). We outperform the baseline system and achieve interesting results on both subtasks

Ngoc Phuoc An Vo

Octavian Popescu

Simone Magnolini

Ngoc Phuoc An, Vo

Magnolini, Simone

Popescu, Octavian

Archivio della ricerca - Fondazione Bruno Kessler

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 231–235,Denver, Colorado, June 4-5, 2015. c©2015 Association for Computational LinguisticsFBK-HLT: An Application of Semantic Textual Similarity for AnswerSelection in Community Question AnsweringNgoc Phuoc An VoUniversity of Trento,Fondazione Bruno KesslerTrento, Italyngoc@fbk.euSimone MagnoliniUniversity of Brescia,Fondazione Bruno KesslerTrento, Italymagnolini@fbk.euOctavian PopescuIBM Research, T.J. WatsonYorktown, USo.popescu@us.ibm.comAbstractThis paper reports the description and perfor-mance of our system, FBK-HLT, participatingin the SemEval 2015, Task #3 "Answer Se-lection in Community Question Answering"for English, for both subtasks. We submit tworuns with different classifiers in combining typ-ical features (lexical similarity, string similar-ity, word n-grams, etc.) with machine transla-tion evaluation metrics and with some ad hocfeatures (e.g user overlapping, spam filtering).We outperform the baseline system and achieveinteresting results on both subtasks.1 IntroductionAnswer selection is an important task inside the widertask of question answering that represents at the mo-ment a topic of great interest for research and busi-ness as well. Analyzing social data like answersgiven inside a forum is a way to maximize the valueof this type of knowledge source that is usually af-fected by a very noisy information due to out of topicspam, double posting, cross posting or other issues.Recognizing useful posts from bad ones, and auto-matically detecting the main polarity of answers toa given question is a way to treat an amount of datathat otherwise might be difficult to handle.A promising way to provide insight into these ques-tions was brought forward as Shared Task #3 in theSemEval-2015 campaign for "Answer Selection inCommunity Question Answering" (Màrquez et al.,2015) for English and Arabic languages. In the Sub-task A, each system is given a set of questions inwhich each one contains some data like posting date,author’s Id, a set of comments, at least one, but usu-ally more; then the participating the system has toclassify comments as good, bad or potential accord-ing to their relevance with the question. In Subtask B,a subset of these questions are predefined as yes/noquestions, system has to classify them into yes, no orunsure classes based on the individual good answers.We participate in this shared task (only in English)with a system composing several different featuresusing a multiclass classifier. We are interested infinding out whether similarity, machine translationevaluation metrics and task specific techniques couldincrease the accuracy of our system. In this paper,we outline our method and present the results forthe answer selection task; the paper is organized asfollows: Section 2 presents the System Description,Section 3 describes the Experiment Settings, Section4 reports the Evaluations, Section 5 is the Error Anal-ysis and finally, Section 6 presents the Conclusionsand Future Work.2 System DescriptionIn order to build our system, we extract and adoptseveral different linguistic features from a SemanticTextual Similarity (STS) system (Vo et al., 2015) andthen consolidate them by a multiclass classifier. Dif-ferent features can be used independently or togetherwith others to measure the semantic similarity andrecognize the paraphrase of a given sentence pair aswell as to evaluate the significance of each featureto the accuracy of system’s predictions. Hence, thesystem is expandable and scalable for adopting moreuseful features aiming for improving the accuracy.2312.1 Data PreprocessingAs data preprocessing is a crucial step for preparinguseful information to be learned by the system, wefocus the beginning of our work trying to simplifydata without losing information. Our system is basedon semantic similarity, so it needs pairs of sentencesto compare; we pair-up every question with all of itscomments, one by one, e.g. a question with five com-ments becomes five pairs of sentences composed bythe question and five different comments. Questionsand comments are composed by subject and body,so for questions, we merge the subject and body to-gether if the subject does not occur inside the body;and for comments, we also check if the comment’ssubject is not identical to question’s subject with theprefix RE:. As the forum data also contains lot ofinformal writing, we normalize them by applying asimple filter that substitutes common abbreviation:"u - you", "r - are", "ur - your", "Iam - I am", "any1 -anyone".2.2 Syntactic GeneralizationGiven a pair of parse trees, the Syntactic Generaliza-tion (SG) (Galitsky, 2013) finds a set of maximal com-mon subtrees. The toolkit "relevance-based-on-parse-trees" is an open-source project, which evaluates textrelevance by using syntactic, parse-tree-based simi-larity measure.1 It measures the similarity betweentwo sentences by finding a set of maximal commonsubtree for a pair of parse trees, using representa-tion of constituency parse trees via chunking. Eachtype of phrases (NP, VP, PRP etc.) will be alignedand subject to generalization. It uses the OpenNLPsystem to derive constituent trees for generalization(chunker and parser).2 As it is an unsupervised ap-proach, we apply the tool directly to the preprocessedtexts to compute the similarity of syntactic structureof sentence pairs.2.3 Machine Learning Evaluation Metric -METEORWe also use evaluation metrics for machine transla-tion as suggested in (Madnani et al., 2012) for para-phrase recognition on Microsoft Research paraphrasecorpus (MSRP) (Dolan et al., 2004). In machine1https://code.google.com/p/relevance-based-on-parse-trees/2https://opennlp.apache.orgtranslation, the evaluation metric scores the hypothe-ses by aligning them to one or more reference trans-lations. We take into consideration to use all theeight metrics proposed, but we find that adding someof them without a careful process of training on thedataset may decrease the performance of the system.We use the latest version of METEOR (Denkowskiand Lavie, 2014) that finds alignments between sen-tences based on exact, stem, synonym and paraphrasematches between words and phrases. We used thesystem as distributed on its website, using only the"norm" option that tokenizes and normalizes punctu-ation and lowercase as suggested by documentation.3We compute the word alignment scores between ques-tions and comments.2.4 Weighted Matrix Factorization (WMF)WMF (Guo and Diab, 2012) is a dimension reductionmodel to extract nuanced and robust latent vectorsfor short texts/sentences. To overcome the sparsityproblem in short texts/sentences (e.g. 10 words onaverage), the missing words, a feature that Latent Se-mantic Analysis (LSA) and Latent Dirichlet Alloca-tion (LDA) typically overlook, is explicitly modeled.We use the pipeline to compute the similarity scoresfor question-comment pairs.42.5 User OverlappingWe extract a simple binary feature focused on com-ment’s author. We suppose that question’s author isnot usually as same as comment’s author, so if a ques-tion has one or more comments associated with thesame question’s author, these comments are probablydescriptions or explanations about the question. Welabel 1 for comments made by the same question’sauthor and 0 otherwise.2.6 Spam Filtering - JFilterRecognizing good comments from bad comment isa task somehow similar to spam filtering, to capturethis feature, we use a Java implementation, Jfilter(Francesco Saverio Profiti, 2007), based on a fuzzyversion of the Rocchio algorithm (Rocchio, 1971).This system uses a classifier that needs training, soto avoid overfitting, from the training and develop-ment datasets, we randomly choose a subset of good3http://www.cs.cmu.edu/%7Ealavie/METEOR/index.html4http://www.cs.columbia.edu/%7Eweiwei/code.html232Accuracy F1 (G) F1 (B) F1 (D) F1 (P) F1 (NE) F1 (O) F1 WMBaseline 53.19 0.694 0 0 0 0 0 0.3691-against-all 60.06 0.731 0.189 0.545 0 0 0 0.523Random Correction Code 59.02 0.722 0.319 0.540 0 0 0 0.539Exhausted Correction Code 60.00 0.731 0.18 0.545 0 0 0 0.521Table 1: Result obtained using different classification algorithms for Subtask A (G good; B bad; D dialog; P potential;NE not-English; O other; WM Weighted Mean) on Development dataset.Accuracy F1 (Yes) F1 (No) F1 (Unsure) F1 (Not-Applicable) F1 WMStandard Features 44.4444 0.316 0 0.077 0.593 0.355Standard Features+ Subtask A output45.4444 0.327 0 0.08 0.589 0.358Standard Features+ Subtask Agold-standard labels70.7071 0.667 0 0.069 1 0.635Table 2: Subtask B system performances on Development dataset.comments to use as non-spam dataset; in contrast, weselect a subset of bad and potential to use as spamdataset to train JFilter. This configuration was usedto train our system during development; for the finalrun with test dataset, we train JFilter with both devel-opment and training datasets. JFilter gives a binaryjudgment (HAM or SPAM) which is used as a featurefor our system in Subtask A.3 Experiment SettingsWe use the machine learning toolkit WEKA (Hall etal., 2009) to obtain robust and efficient implemen-tation of different classifiers, as well as to reducedevelop time of the system. For Subtask A, we buildone model using all the features described in Section2. Table 1 reports some experiments in which weselect a good classifier to optimize both the Accuracyand F1-score of the system. During the development,we select the default implementation "1-against-all"classification algorithm (with logistic regression) forboth subtasks.For Subtask B, we make some modifications to thesystem due to some important differences betweentwo subtasks. As the question classification dependson the quality of its comments, we substitute thespam filtering feature by the comments’ labels fromSubtask A system’s output. In order to examine thishypothesis, we firstly use the gold-standard labels ofcomments from Subtask A as a feature for the ques-tion classification in Subtask B. The high Accuracyand F1-score from this setting proves our hypothesiscorrect. To avoid the overfitting, we again use onlythe label predictions from Subtask A as a feature forour Subtask B system. Table 2 shows that a preciseoutput from Subtask A can significantly benefit theperformance of Subtask B system.As Subtask B does not focus on comment labeling,but question labeling, to achieve this purpose afterclassifying all comments as yes, no, unsure or NotApplicable, we simply aggregate comments of everyquestion with a majority vote. We label a question asyes if the majority of its comments are classified asyes, the same for no; if there no major judgment ofeither yes or no, the question is classified as unsure.Subtask A Subtask BTeam Mac F1 Acc Mac F1 AccJAIST 57.19 72.52VectorSlu 63.7 72.0FBK-HLT 47.32 69.13 27.8 40.0Table 3: Evaluation Results on Subtasks A and B.233Team Accuracy F1 (G) F1 (B) F1 (D) F1 (P) Macro F1JAIST (3-classes) 72.67 79.11 78.29 0 14.48 57.29HLT-FBK (3-classes) 69.13 75.80 66.15 0 0 47.32JAIST (4-classes) 59.62 76.52 40.38 57.21 18.41 48.13HLT-FBK (4-classes) 62.40 75.80 43.42 51.23 0 42.61Table 4: Subtask A - Comparison with best system for 3-classes and 4-classes evaluation (G good; B bad; D dialog; Ppotential; Macro F1).Team Accuracy F1 (Yes) F1 (No) F1 (Unsure) Macro F1VectorSlu 72.0 83.87 57.14 50.0 63.67FBK-HLT 40.0 50.0 0.0 33.33 27.78Table 5: Subtask B - Comparison with best system.4 EvaluationsWe submit only one run for both subtasks (Englishlanguage) using the "1-against-all" classification al-gorithms. In Subtask A, we achieve good results,especially, we are ranked 4th out of 12 teams in Ac-curacy. In Subtask B, as we only apply the simpleapproach "majority vote", the result is reasonable asexpected. Table 3 shows our performance in bothsubtasks in regard to the best systems, both in MacroF1 and Accuracy measures.5 Error AnalysisIn this section, we conduct an analysis of our sys-tem’s performance on test dataset. In Subtask A,our analysis consists of some comparison betweenour system and the best system, JAIST. Accordingto results in Table 4, for the evaluation on 3-classes(good, bad, and potential), our system is dramati-cally penalized by low performance on detecting badcomments, besides, it is not able to classify the po-tential ones. This particular class of comments isvery small in training dataset. There are 50.45% forgood comments, 41.09% for bad and only 8.25% forpotential. During the development, as we decide tooptimize the Accuracy and F1 weighted on the num-ber of comments, this decision misleads our systemto ignore this small class. Hence, in order to improvethe system performance, we may need to search fora specific feature for potential comments like whatwe did with user overlapping for dialog ones. For theevaluation on 4-classes (good, bad, dialog and po-tential), our system performance rises significantly,our system shows a good capability to distinguishbetween dialog and other comments.In Subtask B, the performance comparison in Ta-ble 5 shows that our system achieves reasonable per-formance on the Yes and Unsure classes, but has nocapability to capture the No class. Moreover, mostof the instances of No class have been misclassifiedas Unsure class. This shows an unclear separationbetween these two classes which confuses the system.Thus, to fix this issue, we need to find more specificfeatures which may help to distinguish the No classand others.6 Conclusions and Future WorkIn this paper, we describe our system participatingin the SemEval 2015, Task #3 "Answer Selectionin Community Question Answering" in English, forboth subtasks. We present a supervised system whichconsiders multiple linguistic features such as lexical,string and some task-specific features. Our perfor-mance is much above the baseline and shows someinteresting properties in specific scenarios. We alsoshow some error analysis in which we investigatethe limit and drawback of our system on specificcomment and question classes.For future work, we expect to study to exploit moreuseful features, especially, task-related features, toimprove the classification performance on potentiallabeled comments and No labeled questions, whichwill lead to a significant improvement of the overallperformance.234ReferencesMichael Denkowski and Alon Lavie. 2014. Meteor uni-versal: Language specific translation evaluation for anytarget language. In Proceedings of the EACL 2014Workshop on Statistical Machine Translation.Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Un-supervised construction of large paraphrase corpora:Exploiting massively parallel news sources. In Pro-ceedings of the 20th international conference on Com-putational Linguistics, page 350.Claudio Biancalana Francesco Saverio Profiti. 2007. Jfil-ter: un filtro antispam intelligente in java. Mokabyte,(124). in Italian.Boris Galitsky. 2013. Machine learning of syntactic parsetrees for search and classification of text. Engineer-ing Applications of Artificial Intelligence, 26(3):1072–1091.Weiwei Guo and Mona Diab. 2012. Modeling sentencesin the latent space. In Proceedings of the 50th AnnualMeeting of the Association for Computational Linguis-tics: Long Papers-Volume 1, pages 864–872.Mark Hall, Eibe Frank, Geoffrey Holmes, BernhardPfahringer, Peter Reutemann, and Ian H Witten. 2009.The weka data mining software: an update. ACMSIGKDD explorations newsletter, 11(1):10–18.Nitin Madnani, Joel Tetreault, and Martin Chodorow.2012. Re-examining machine translation metrics forparaphrase identification. In Proceedings of the 2012Conference of the North American Chapter of the As-sociation for Computational Linguistics: Human Lan-guage Technologies, pages 182–190.Lluís Màrquez, James Glass, Walid Magdy, AlessandroMoschitti, Preslav Nakov, and Bilal Randeree. 2015.Semeval-2015 task 3: Answer selection in communityquestion answering. In Proceedings of the 9th Inter-national Workshop on Semantic Evaluation (SemEval2015).Joseph John Rocchio. 1971. Relevance feedback in infor-mation retrieval.Ngoc Phuoc An Vo, Simone Magnolini, and OctavianPopescu. 2015. FBK-HLT: A new framework forsemantic textual similarity. In Proceedings of the9th International Workshop on Semantic Evaluation(SemEval-2015), Denver, US.235

FBK-HLT: An Application of Semantic Textual Similarity for Answer Selection in Community Question Answering

https://cris.fbk.eu/bitstream/11582/302183/1/FBK-HLT-%20An%20Application%20of%20Semantic%20Textual%20Similarity%20for%20Answer%20Selection%20in%20Community%20Question%20Answering.pdf

FBK-HLT: An Application of Semantic Textual Similarity for Answer Selection in Community Question Answering

Abstract

Similar works

Full text

Available Versions

Archivio della ricerca - Fondazione Bruno Kessler