Alessandro, Mazzei

Cristina, Bosco

English

Bosco, Cristina

Mazzei, Alessandro

Institutional Research Information System University of Turin

This is an author version of the contribution published on:Cristina BOSCO, Alessandro MAZZEIThe Evalita 2011 Parsing Task: the constituency trackEditor: CELCT a r.l.2012inWorking Notes of EVALITA 20111 - 6Evalita 2011Roma24-25 gennaio 2012The definitive version is available at:http://www.evalita.it/2011/working_notesThe Evalita 2011 Parsing Task: the constituencytrackCristina Bosco and Alessandro MazzeiDipartimento di Informatica, Universita` di TorinoCorso Svizzera 185, 101049 Torino, Italy{bosco,mazzei}@di.unito.itAbstract. The aim of Evalita Parsing Task is at defining and extendingItalian state of the art parsing by encouraging the application of exist-ing models and approaches. As in the Evalita’07 and ’09, the Task isorganized around two tracks, i.e. Dependency Parsing and ConstituencyParsing. In this paper, we describe only the Constituency Parsing trackby presenting the data sets for development and testing, and reportingthe results, which positively compare with those obtained for this sametrack held in the Evalita’07 and ’09.Keywords: Constituency Parsing, Evaluation, Italian1 MotivationThe general aim of the Evalita Parsing evaluation campaign is at defining andextending Italian state of the art parsing with reference to existing resources, byencouraging the application of existing models to this language. As in previouseditions, in 2007 [7, 5] and 2009 [4, 3], the focus is mainly on the application toItalian language of various parsing approaches, i.e. rule–based and statistical,and paradigms, i.e. constituency and dependency–based. The aim is in fact atcontributing to the literature on parsing results giving information about thebehavior of parsing models on Italian, which is a morphologically rich languagecurrently less-resourced with respect e.g. to English or German.In previous Evalita editions, the results for dependency parsing have been evalu-ated as no far from the state of the art for English, while those for constituencyshowed a higher distance from it. Instead, in the current edition the major im-provement has to be referred to constituency parsing, where scores meaningfullymore proximate to the state of the art for English have been achieved. Never-theless, these results confirm that the scores published for English (of aroundF 92.1 [13]) using the Penn Treebank, remain currently irreproducible for Ital-ian. By proposing again the constituency track, we aim at contributing to theinvestigation on the causes of this irreproducibility and giving the same datafor development and testing (annotated in dependency and constituency for-mat) in both tracks of the parsing task, we make available new materials for thedevelopment of cross-paradigm analyses about Italian parsing.In this paper we will analyze the constituency track of the competition, whilethe dependency track is described in [2]. The paper is organized as follows. Inthe following sections, first we describe the task, then, we show the developmentand test data sets and the measures applied in the evaluation procedure. Weconclude with the presentation of participation results and a brief discussion.2 Definition of the TaskAs described in the CoNLL competitions [8, 14], the parsing task is defined as theactivity of assigning a syntactic structure to a given set of PoS tagged sentences,called development set. A large set of syntactically fully annotated sentences isgiven to the participants in order to train and tune their parsers. The evalua-tion for this task is based on a manually syntactically annotated smaller set ofsentences called gold standard test set.Since the Evalita Parsing task is articulated in two tracks, for each track a testset and a development set is given by organizers. Moreover, in order to allowfor a meaningful and direct comparison between the results achieved in the twotracks, all these datasets for dependency have been built by including exactlythe same sentences as for constituency. Nevertheless, even if the organizers en-couraged the participation to both tracks, only one participant submitted runsfor both dependency and constituency.3 DatasetsThe data proposed for the training and development of parsing systems (i.e. thedevelopment set) are from TUT, the treebank for Italian developed by the Nat-ural Language Processing group of the Department of Computer Science of theUniversity of Turin1. TUT has been newly released for the last time in 2011, afterautomatic and manual revisions, in an improved version where both the consis-tency of the annotation and the size of the treebank are improved with respectto the previous releases. In particular, for what concerns size, TUT is currentlysimilar to the other Italian resources, i.e. VIT and ISST–TANL. Moreover, TUTmakes available more annotation formats [6] that allowed for a larger variety oftraining and testing for parsing systems and for meaningful comparisons withtheoretical linguistic frameworks, e.g. the native TUT, the TUT-Penn, and theCCG-TUT which is an application to Italian of the Combinatory CategorialGrammar [1].3.1 Development SetFor the constituency parsing track, the data format adopted is the TUT–Penn,as in previous Evalita contests, which is an application of the Penn Treebankformat to the Italian language [3]. The kind and structure of the constituents1 For the free download of the resource, see http://www.di.unito.it/∼tutreeb.are the same as in Penn Treebank for English, but the inventory of functionaltags is enriched with some relations needed to represent e.g. the subject in post-verbal position. Moreover, in order to describe the rich inflectional system ofItalian language, the TUT–Penn format adopts a different and richer set of Partof Speech tags with respect to the Penn Treebank.The development set for both tracks includes the same sentences, namely3,452 sentences belonging to the five different text genres which are currentlyrepresented in the subcorpora of the treeebank:– NEWS and VEDCH, two collections of sentences from Italian newspaper(700 + 400 sentences)– CODCIV, a collection of sentences from the Italian Civil Law Code (1,100sentences)– EUDIR, a collection of declarations of the European Community from theItalian section of the JRC-Acquis Multilingual Parallel Corpus2 (201 sen-tences)– Wikipedia, a collection of sentences from the Italian section of Wikipedia(459 sentences)– COSTITA, the full collection of sentences of the Costituzione Italiana (682sentences)We can observe that in 2009 the development set for the constituency track wassmaller and consisted in 2,200 sentences (i.e. 64,193 annotated tokens in TUTnative format) representing only two text genres3, which are now all included inthe Evalita 2011 development set. Moreover, while in the past only a portion ofthe development set proposed for the training of dependency parsers was madeavailable also for the constituency track in TUT–Penn format, this year, for thefirst time, the development set is exactly the same for both tracks.3.2 Test SetThe test set for both the tracks is composed by the same 300 sentences which rep-resent around the same balancement of the development set: 150 sentences fromCivil Law Code, 75 sentences from newspapers and 75 sentences from Wikipedia.In Evalita 2009, the test set included only 200 sentences, namely 100 from news-papers and 100 from the Civil Law Code.4 Evaluation MeasuresAs in previous editions of the contest, we exploited for the evaluation of con-stituency parsing results the standard metric EVALB (http://nlp.cs.nyu.edu/evalb/,[10]): it is a bracket scoring program that reports labelled precision (LP), recall2 http://langtech.jrc.it/JRC-Acquis.html3 The development set in 2009 was composed by the Italian newspapers (1,100 sen-tences) and the Civil Law Code (1,100 sentences) subcorpora.(LR), F-score (LF), non crossing and tagging accuracy for given data. Note thatthe official measure for the final result is the F-score.As usual we did not score the TOP label and the functional labels too; moreover,in contrast with usual, in order to have a direct comparison with dependencysubtask we do use punctuation in scoring.5 Participation ResultsWe had only one participant to the constituency track, i.e. FBKirst Lavelli CPAR4,whose parser adopts probabilistic context-free grammars model, namely theBerkeley one. The author participated also to previous editions of the task, andto the dependency track of Evalita’11. Confirming the trend seen in previous edi-tions of the task, the number of participants for constituency parsing has beensmaller than for dependency, but in previous editions we had two participantsalso for the constituency track [3, 7].The evaluation of the participation results for the constituency track is presentedin table 1. With respect to the subcorpora that represent the text genres of TUTTable 1. Constituency parsing evaluation on test set full (300 sentences).Bracketing F Bracketing Recall Bracketing Precision Participant82.96 82.97 82.94 FBKirst Lavelli CPARwithin the test set, we can observe that the best results have been achieved onthe set of sentences extracted from the Civil Law Code (see table 2).Table 2. Constituency parsing evaluation on subcorpora.Subcorpus Size Bracketing F Bracketing Recall Bracketing PrecisionCODCIV 150 87.27 87.41 87.14NEWS 75 77.46 78.22 76.72WIKIPEDIA 75 78.38 77.49 79.304 The name of each system that participated to the contest is composed according tothe following pattern: institution author XPAR, where X is D for dependency andC for constituency.6 DiscussionThe results obtained in the constituency parsing positively compare with previ-ous experiences in this area, but still far from the state of the art for Englishconstituency parsing.Due to the fact that the task is based on the same (revised) treebank of Evalita’09and Evalita’07, the more obvious comparison that we can develop is with this ex-perience. With respect to the previous editions of the constituency parsing taskin the Evalita’07 and ’09, there is an impressive improvement of the results: thebest F-score was in fact 72.15 in 2007, and 78.73 in 2009 (80.66 on the sentencesfrom the Civill Law Code only) by Lavelli but by exploiting different parsers [9,12].As far as the text genre is concerned, we can observe that, in the dependencytrack of this year and in previous editions of the constituency parsing track,the best scores have been achieved on the Civil Law sentences. However, thedifferences between the performances of the participant parser on various genresare higher than in the case of dependency. Nevertheless, it is difficult to find themotivations of these differences because of the too limited participation to thetrack. We can only formulate two hypotheses. First, the parser adopted need fora larger amount of data to be trained successfully, and its performance on CivilLaw is the best one because of the larger training set of data available. Second,the legal text genre is intrinsically less hard to parse with respect to newspapersor Wikipedia regardless of the amount of training data. Further experimentswith other parsers and larger amount of data can help us in deciding if the firstor second hypothesis applies.References1. Bos, J., Bosco, C., Mazzei, A.: Converting a Dependency-Based Treebank to aCategorial Grammar Treebank for Italian. In: Proceedings of the 8th Workshop onTreebanks and Linguistic Theories, Milano (2009)2. Bosco, C., Mazzei, A.: The Evalita 2011 Parsing Task: the dependency track. InWorking Notes of EVALITA 2011, 24th-25th January 2012, Rome (2012)3. Bosco, C., Mazzei, A., Lombardo, V.: Evalita Parsing Task 2009: constituencyparsing and a Penn format for Italian. In: Proceedings of Evalita’09 at AI*IA,Reggio Emilia (2009)4. Bosco, C., Montemagni, S., Mazzei, A., Lombardo, V., Dell’Orletta, F., Lenci,A.: Evalita’09 Parsing Task: comparing dependency parsers and treebanks. In:Proceedings of Evalita’09 at AI*IA, Reggio Emilia (2009)5. Bosco, C., Mazzei, A., Lombardo, V., Attardi, G., Corazza, A., Lavelli, A., Lesmo,L., Satta, G., Simi, M.: Comparing Italian parsers on a common treebank: theEvalita experience. In: Proceedings of LREC’08, Marrakesh (2008)6. Bosco C.: Multiple-step treebank conversion: from dependency to Penn format. In:Proceedings of the Workshop on Linguistic Annotation at the ACL’07 (2007)7. Bosco, C., Mazzei, A., Lombardo, V.: Evalita Parsing Task: an analysis of the firstparsing system contest for Italian. Intelligenza Artificiale 12 (2007)8. Buchholz S., Marsi E.: CoNLL-X Shared Task on Multilingual Dependency Pars-ing. In: Proceedings of the CoNLL-X (2006)9. Corazza, A., Lavelli, A., Satta, G.: Phrase Based Statistical Parsing. IntelligenzaArtificiale 12 (2007)10. Collins M.: A New Statistical Parser Based on Bigram Lexical Dependencies. In:Proceedings of ACL96 (1996)11. Ku¨bler S., McDonald R., Nivre J.: Dependency parsing. Morgan and ClaypoolPublishers (2009)12. Lavelli, A., Corazza, A.: The Berkeley Parser at the EVALITA 2009 ConstituencyParsing Task. In: Proceedings of Evalita’09 at AI*IA, Reggio Emilia (2009)13. McClosky D., Charniak E., Johnson M.: When is self-training effettive for parsing?In: Proceedings of CoLing (2008)14. Nivre J., Hall J., Ku¨bler S., McDonald R., Nilsson J., Riedel S., Yuret D.: TheCoNLL 2007 Shared Task on Dependency Parsing. In: Proceedings of the EMNLP-CoNLL (2007)

The Evalita 2011 Parsing Task: the constituency track

https://iris.unito.it/bitstream/2318/96494/1/CONS_PARS_ORGANIZERS_4aperto_680597.pdf

The Evalita 2011 Parsing Task: the constituency track

Abstract

Similar works

Full text

Available Versions

Institutional Research Information System University of Turin