Bosco, Cristina

Mazzei, Alessandro

English

Institutional Research Information System University of Turin

This is an author version of the contribution published on:Cristina BOSCO, Alessandro MAZZEIThe Evalita 2011 Parsing Task: the Dependency TrackEditor: CELCT a r.l.2012inWorking Notes of EVALITA 20111 - 6Evalita 2011Roma24-25 gennaio 2012The definitive version is available at:http://www.evalita.it/2011/working_notesThe Evalita 2011 Parsing Task: the DependencyTrackCristina Bosco and Alessandro MazzeiDipartimento di Informatica, Universita` di TorinoCorso Svizzera 185, 101049 Torino, Italy{bosco,mazzei}@di.unito.itAbstract. The aim of Evalita Parsing Task is at defining and extendingItalian state of the art parsing by encouraging the application of exist-ing models and approaches. As in the Evalita’07 and ’09, the Task isorganized around two tracks, i.e. Dependency Parsing and ConstituencyParsing. In this paper, we describe only the Dependency Parsing trackby presenting the data sets for development and testing, and reportingthe test results.Keywords: Dependency Parsing, Evaluation, Italian1 MotivationThe Evalita Parsing evaluation campaign aims at defining and extending Italianstate of the art parsing with reference to existing resources, by encouraging theapplication of existing models to this language, which is morphologically richand currently less-resourced. As in the editions held in 2007 [9, 7] and 2009[3, 4], the focus is mainly on the application to Italian of various approaches,i.e. rule–based and statistical, and paradigms, i.e. constituency and dependency.Therefore, the task is articulated in two tracks, i.e. dependency and constituency,which share the same development and test data, distributed both in dependencyand constituency format. In this paper we will analyze the dependency track ofthe competition, while the constituency track is described in [2].The paper is organized as follows. First, we describe the task, then, we show thedevelopment and test data and the measures applied in the evaluation procedure.We conclude with the presentation of participation results and a brief discussion.2 Definition of the TaskAs described in [11, 17], the parsing task is the activity of assigning a syntacticstructure to a given set of PoS tagged sentences. A large set of syntacticallyfully annotated sentences, i.e. the development set is given to the participantsin order to train and tune their parsers. The evaluation is based on a manuallysyntactically annotated smaller set of sentences, called gold standard test set.For each track in which the Evalita Parsing task is articulated, a test set and a de-velopment set is given by organizers. Moreover, in order to allow for a meaningfuland direct comparison between the two tracks, all these datasets for dependencyhave been built by including exactly the same sentences as for constituency.Nevertheless, even if the organizers encouraged the participation to both tracks,only one participant submitted runs for both tracks.3 DatasetsThe data proposed for the training and development are from the Turin Univer-sity Treebank (TUT), the treebank for Italian developed by the Natural Lan-guage Processing group of the Department of Computer Science of the Universityof Turin1. This resource, newly released in 2011, after automatic and manual re-visions oriented to improving consistency and size of the treebank, is currentlyas large as other existing Italian resources, i.e. VIT and ISST–TANL. Moreover,in order to allow a variety of training and comparisons across various theoreticallinguistic frameworks, TUT makes available several annotation formats [8], e.g.native TUT, TUT–Penn, and CCG–TUT, which is an application to Italian ofthe Combinatory Categorial Grammar [5].The native scheme of TUT applies the major principles of dependency gram-mar [12] using a rich set of grammatical relations2 and is featured by the dis-tinctive inclusion of null elements to deal with non-projective structures, longdistance dependencies, equi phenomena, pro drop and elliptical structures, whichare quite common in a flexible word order language like Italian. On the one hand,this allows in the most of cases for the representation and the recovery of argu-ment structures associated with verbs and nouns, and it permits the processingof long distance dependencies in a similar way to the Penn format. On the otherhand, by using null elements crossing edges and non–projective dependency treescan be avoided.Nevertheless, in order to make possible the application of standard evaluationmeasures within Evalita contests, the native format of TUT has been auto-matically converted in a format more proximate to the standard CoNLL. Theresulting format differs from native TUT for the following features: it splits theannotation in the ten standard columns (filling eight of them) as in CoNLL,rather than organize them in round and square brackets; it exploits only part ofthe rich set of grammatical relations (72 in CoNLL versus 323 in TUT native);it does not include pointed indexes3. Since CoNLL does not allow null elements,1 For the free download of the resource, see http://www.di.unito.it/∼tutreeb.2 See [10], [6].3 In TUT native format the representation of amalgamated words uses pointed in-dexes, e.g. a definite prepositions ’del’ occurring as 33th word of a sentence is splitin two lines, ’33 del (PREP ....’ and ’33.1 del (ART ....’ respectively representingthe Preposition and the Article. In CoNLL format, where pointed indexes are notallowed, these two lines became ’33 del (PREP ....’ and ’34 del (ART ....’.they are deleted in this format, but the projectivity constraint is maintained atthe cost of a loss of information with respect to native TUT in some cases.3.1 Development SetThe development set includes 3,452 Italian sentences (i.e. 102,150 tokens in TUTnative, and 93,987 in CoNLL4) and represents five different text genres organizedin:– NEWS and VEDCH, from newspapers (700 + 400 sentences, 18,044 tokens)– CODCIV, from the Italian Civil Law Code (1,100 sentences, 28,048 tokens)– EUDIR, from the JRC-Acquis Corpus5 (201 sentences, 7,455 tokens)– Wikipedia, from Wikipedia (459 sentences, 14,746 tokens)– COSTITA, the full text of the Costituzione Italiana (682 sentences, 13,178tokens)In 2009 the development set was smaller, and consisted in 2,400 sentences (i.e.72,149 annotated tokens in TUT native format, and 66,055 in CoNLL format),organized in only three subcorpora, i.e. Italian newspapers, Civil Law Code andJRC-Acquis. All these sentences are included also in the Evalita 2011 develop-ment set.3.2 Test SetThe test set is composed by 300 sentences (i.e. 7,836 tokens) around balancedas in the development set: 150 sentences from Civil Law Code (3,874 tokens),75 sentences from newspapers (2,035 tokens) and 75 sentences from Wikipedia(1,927 tokens). In Evalita 2009 the test set included 240 sentences (5,287 tokens).4 Evaluation MeasuresThe standard methodology for the evaluation of dependency parsers is to applythem to a test set and compare their output to the gold standard test set, i.e.the test set annotated according to the treebank used for the development of theparsers.Among the most widely used evaluation metrics, we have selected for the evalua-tion of the official results those used in the CoNLL parsing shared task, i.e. LAS(Labeled Attachment Score) that is the percentage of tokens with correct headand dependency type. Moreover, in accord with literature, we report the UAS(Unlabeled Attachment Score) measure too, i.e. the percentage of tokens withcorrect head [11, 17]. Note that the use of a single accuracy metric is possiblein dependency parsing thanks to the single-head property of dependency trees,which implies that the amount n of nodes/words always corresponds to n − 1dependency relations. This property allows the unification of measures of preci-sion and recall and makes parsing resemble a tagging task, where every word isto be tagged with its correct head and dependency type [13].4 In the following we will refer only to number of tokens in CoNLL format.5 http://langtech.jrc.it/JRC-Acquis.html5 Participation ResultsThe participants6 to the dependency parsing track were four. Among them onlyone did not participate at the previous editions of the contest, where the partic-ipants were six.Two participant systems, i.e. UniTo Lesmo DPAR and Parsit Grella DPAR,do not follow the statical approach. UniTo Lesmo DPAR system is a rule–basedwide coverage parser developed in parallel with TUT, which has been applied tovarious domains. The Parsit Grella DPAR uses instead a hybrid approach thatmixes rules and constraints. The other two participating systems belong insteadto the class of statistical parsers: FBKirst Lavelli DPAR is an application toItalian of different parsing algorithms implemented in MaltParser [16] and ofan ensemble model made available by Mihai Surdeanu; UniPi Attardi DPAR isinstead DeSR, a Shift/Reduce deterministic transition–based parser [1] whichparticipated also to CoNLL contests.According to the main evaluation measure, i.e. LAS, the best results havebeen achieved by Parsit Grella DPAR followed by UniPi Attardi DPAR (seeTable 1. Dependency parsing: evaluation on the test set (300 sentences).LAS UAS Participant91.23 96.16 Parsit Grella DPAR89.88 93.73 UniPi Attardi DPAR88.62 92.85 FBKirst Lavelli DPAR85.34 91.47 UniTo Lesmo DPARtable 1) with a difference statistically significant according to the p-value7. Theaverage scores of the participants are 88.76 for LAS and 93.55 for UAS.In table 2, we see instead how the performance varies according to text genres.If evaluated on the civil law texts the difference among the three best scoredsystems is not statistically significative, while it is significative on Wikipediaand more valuable on newspaper. In the latter text genre, all the scores achievedby Parsit Grella DPAR are significantly higher than those of the others, and thismotivates the success of this parser in the contest.6 The name of each system that participated to the contest is composed according tothe following pattern: institution author XPAR, where X is D for dependency andC for constituency.7 The difference between two results is taken to be significant ifp < 0.05 (see http://depparse.uvt.nl/depparse-wiki/AllScores andhttp://nextens.uvt.nl/∼conll/software.html#eval).Table 2. Dependency parsing, evaluation on subcorpora: civil law (150 sentences),newspaper (75 sentences), wikipedia (75 sentences).civillaw newspaper wikipedia ParticipantLAS UAS LAS UAS LAS UAS92.85 96.18 86.34 91.19 86.91 90.88 UniPi Attardi DPAR92.21 97.01 90.75 95.54 89.51 94.51 Parsit Grella DPAR91.56 95.12 83.84 89.72 87.09 91.05 FBKirst Lavelli DPAR89.06 94.43 80.69 87.70 81.87 88.80 UniTo Lesmo DPAR6 DiscussionThe results positively compare with other experiences, both for Italian, i.e.Evalita’07 and ’09, which were based on the same (revised) treebank, and forother languages, e.g. English (LAS 89, 61%) and Japanese (LAS 91, 65%) [17].The best scores passed from 86.94 for LAS and 90.90 for UAS in 2007 (byUniTo Lesmo DPAR), to 88.73–88.69 for LAS(by UniTo Lesmo DPAR andUniPi Attardi DPAR) and 92.72 for UAS (by UniPi Attardi DPAR) in 2009,to 91.23 for LAS and 96.16 for UAS by Parsit Grella DPAR in 2011. The aver-age LAS is passed from 72.48 in 2007, to 82.88 in 2009), to 88.76 in 2011, whilethe average UAS from 83.09, to 87.96, to 93.55. As in previous editions, the bestperformances are referred to the Civil Law texts.With respect to the approaches, the observed improvement of results for the twostatistical systems can be probably motivated by the availability of larger setsof data for training from 2007 to 2011. For instance, by contrast with Evalita’09results, the top rule–based parser in Evalita’09 and ’07 (UniTo Lesmo DPAR)scores significantly worst than the two stochastic parsers (UniPi Attardi DPAR)and FBKirst Lavelli DPAR. But the best performing system is again a non sta-tistical system (i.e. Parsit Grella DPAR). Nevertheless, also the results of thisedition confirm that non–statistical systems can achieve good scores only if de-veloped in parallel with the reference resource, like UniTo Lesmo DPAR in thepast contests and Parsit Grella DPAR; while rule–based approaches not enoughtuned on the resource obtained negative results, see e.g. [19] or [18].We conclude by observing that in a wider perspective of evaluation of the contri-bution of parsing to the overall quality of applicative NLP systems, other kindsof information should be taken into account, in particular those coming fromnull elements and semantic features currently annotated only in a few resources.References1. Attardi, G., DellOrletta, F., Simi, M., Turian J.: Accurate Dependency Parsingwith a Stacked Multilayer Perceptron. In: Proceedings of Evalita’09 at AI*IA,Reggio Emilia (2009)2. Bosco, C., Mazzei, A.: The Evalita 2011 Parsing Task: the constituency track. InWorking Notes of EVALITA 2011, 24th-25th January 2012, Rome (2012)3. Bosco, C., Montemagni, S., Mazzei, A., Lombardo, V., Dell’Orletta, F., Lenci,A.: Evalita’09 Parsing Task: comparing dependency parsers and treebanks. In:Proceedings of Evalita’09 at AI*IA, Reggio Emilia (2009)4. Bosco, C., Mazzei, A., Lombardo V.: Evalita’09 Parsing Task: constituency parsersand the Penn format for Italian. In: Proceedings of Evalita’09 at AI*IA, ReggioEmilia (2009)5. Bos, J., Bosco, C., Mazzei, A.:Converting a Dependency-Based Treebank to a Cat-egorial Grammar Treebank for Italian. In: Proceedings of the 8th Workshop onTreebanks and Linguistic Theories, Milano (2009)6. Bosco C.: A grammatical relation system for treebank annotation. UnpublishedPhD thesis discussed at the University of Turin (2004)7. Bosco, C., Mazzei, A., Lombardo, V., Attardi, G., Corazza, A., Lavelli, A., Lesmo,L., Satta, G., Simi, M.: Comparing Italian parsers on a common treebank: theEvalita experience. In: Proceedings of LREC’08, Marrakesh (2008)8. Bosco C.: Multiple-step treebank conversion: from dependency to Penn format.In: Proceedings of the Workshop on Linguistic Annotation at the ACL’07, Prague(2007)9. Bosco, C., Mazzei, A., Lombardo, V.: Evalita Parsing Task: an analysis of the firstparsing system contest for Italian. Intelligenza Artificiale 12 (2007)10. Bosco C., Lombardo V., Vassallo D., Lesmo L.: Building a Treebank for Italian:a Data-driven Annotation Schema. In: Proceedings of LREC’00, Athens, Greece(2000)11. Buchholz S., Marsi E.: CoNLL-X Shared Task on Multilingual Dependency Pars-ing. In: Proceedings of the CoNLL-X (2006)12. Hudson R.: Word grammar. Basil Blackwel, Oxford and New York (1984)13. Ku¨bler S., McDonald R., Nivre J.: Dependency parsing. Morgan and ClaypoolPublishers (2009)14. Levy R., Manning C.: Is it harder to parse Chinese, or the Chinese treebank? In:Proceedings of ACL’03 (2003)15. Magnini, B., Cappelli, A., Tamburini, F., Bosco, C., Mazzei, A., Lombardo, V.,Bertagna, F., Calzolari, N., Toral, A., Bartalesi Lenzi, V., Sprugnoli, R., Sper-anza, M.: Evaluation of Natural Language Tools for Italian: EVALITA 2007. In:Proceedings of LREC’08, Marrakesh (2008)16. Nivre J., Hall J., Nilsson J., Chanev A., Eryigit G., Ku¨bler S., Marinov S., Marsi E.:MaltParser: a language-independent system for data-driven dependency parsing.In: Natural Language Engineering 13(2) (2007)17. Nivre J., Hall J., Ku¨bler S., McDonald R., Nilsson J., Riedel S., Yuret D.: TheCoNLL 2007 Shared Task on Dependency Parsing. In: Proceedings of the EMNLP-CoNLL (2007)18. Testa, M., Bolioli, A., Dini, L., Mazzini, G.: Evaluation of a Semantically OrientedDependency Grammar for Italian at EVALITA 2009. In: Proceedings of Evalita’09at AI*IA, Reggio Emilia (2009)19. Zanzotto, F.M.: Lost in Grammar Translation. Intelligenza Artificiale 12 (2007)

The Evalita 2011 Parsing Task: the Dependency Track

https://iris.unito.it/bitstream/2318/95278/1/DEP_PARS_ORGANIZERS-1_4aperto_680595.pdf

The Evalita 2011 Parsing Task: the Dependency Track

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin