Rens, Bod

Sangati, Federico

Willem, Zuidema

English

Università degli Studi di Napoli L'Orientale: CINECA IRIS

Proceedings of the 11th International Conference on Parsing Technologies (IWPT), pages 238–241,Paris, October 2009. c©2009 Association for Computational LinguisticsA generative re-ranking model for dependency parsingFederico Sangati, Willem Zuidema and Rens BodInstitute for Logic, Language and ComputationUniversity of AmsterdamScience Park 904, 1098 XH Amsterdam, The Netherlands{f.sangati,zuidema,rens.bod}@uva.nlAbstractWe propose a framework for dependencyparsing based on a combination of dis-criminative and generative models. Weuse a discriminative model to obtain a k-best list of candidate parses, and subse-quently rerank those candidates using agenerative model. We show how this ap-proach allows us to evaluate a variety ofgenerative models, without needing differ-ent parser implementations. Moreover, wepresent empirical results that show a smallimprovement over state-of-the-art depen-dency parsing of English sentences.1 IntroductionProbabilistic generative dependency models de-fine probability distributions over all valid depen-dency structures, and thus provide a useful inter-mediate representation that can be used for manyNLP tasks including parsing and language mod-eling. In recent evaluations of supervised de-pendency parsing, however, generative approachesare consistently outperformed by discriminativemodels (Buchholz et al., 2006; Nivre et al.,2007), which treat the task of assigning the cor-rect structure to a given sentence as a classifica-tion task. In this category we include both transi-tion based (Nivre and Hall , 2005) and graph basedparsers (McDonald, 2006).In this paper, we explore a reranking approachthat combines a generative and a discrimativemodel and tries to retain the strengths of both.The idea of combining these two types of modelsthrough re-ranking is not new, although it has beenmostly explored in constituency parsing (Collinset al., 2002). This earlier work, however, used thegenerative model in the first step, and trained thediscriminative model over its k-best candidates. Inthis paper we reverse the usual order of the twomodels, by employing a generative model to re-score the k-best candidates provided by a discrim-inative model. Moreover, the generative model ofthe second phase uses frequency counts from thetraining set but is not trained on the k-best parsesof the discriminative model.The main motivation for our approach is thatit allows for efficiently evaluating many gener-ative models, differing from one another on (i)the choice of the linguistic units that are gener-ated (words, pairs of words, word graphs), (ii) thegeneration process (Markov process, top-down,bottom-up), and (iii) the features that are consid-ered to build the event space (postags/words, dis-tance). Although efficient algorithms exist to cal-culate parse forests (Eisner, 1996a), each choicegives rise to different parser instantiations.1.1 A generative model for re-rankingIn our re-ranking perspective, all the generativemodel has to do is to compute the probability ofk pre-generated structures, and select the one withmaximum probability. In a generative model, ev-ery structure can be decomposed into a series ofindependent events, each mapped to a correspond-ing conditioning event. As an example, if a gener-ative model chooses D as the right dependent of acertain word H , conditioned uniquely on their rel-ative position, we can define the event as D is theright dependent of H , and the conditioning eventas H has a right dependent.As a preprocessing step, every sentence struc-ture in the training corpus is decomposed into a se-ries of independent events, with their correspond-ing conditioning events. During this process, ourmodel updates two tables containing the frequencyof events and their conditioning counterparts.In the re-ranking phase, a given candidate struc-ture can be decomposed into independent events(e1, e2, . . . , en) and corresponding conditioningevents (c1, c2, . . . , cn) as in the training phase.238The probability of the structure can then be cal-culated asn∏i=1f(ei)f(ci)(1)where f(x) returns the frequency of x previouslystored in the tables.It is important to stress the point that the onlyspecificity each generative model introduces is inthe way sentence structures are decomposed intoevents; provided a generic representation for the(conditioning) event space, both training phaseand probability calculation of candidate structurescan be implemented independently from the spe-cific generative model, through the implementa-tion of generic tables of (conditioning) events.In this way the probabilities of candidate struc-tures are exact probabilities, and do not suf-fer from possible approximation techniques thatparsers often utilize (i.e., pruning). On the otherhand the most probable parse is selected from theset of the k candidates generated by the discrimi-native model, and it will equal with the most prob-able parse among all possible structures, only forsufficiently high k.2 MST discriminative modelIn order to generate a set of k-candidate struc-tures for every test sentence, we use a state-of-the-art discriminative model (McDonald, 2006).This model treats every dependency structure asa set of word-dependent relations, each describedby a high dimensional feature representation. Forinstance, if in a certain sentence word i is thehead of word j, v(i, j) is the vector describingall the features of such relation (i.e., labels of thetwo words, their postag, and other informationincluding words in between them, and ancestralnodes). During the training phase the model learnsa weight vector w which is then used to find thebest dependency structure y for a given test sen-tence x. The score that needs to be maximized isdefined as∑(i,j)∈y w ·v(i, j), and the best candi-date is called the maximum spanning tree (MST).Assuming we have the weight vector and weonly consider projective dependency structures,the search space can be efficiently computed byusing a dynamic algorithm on a compact repre-sentation of the parse forest (Eisner, 1996a). Thetraining phase is more complex; for details we re-fer to (McDonald, 2006). Roughly, the model em-ploys a large-margin classifier which iterates overthe structures of the training corpus, and updatesthe weight vector w trying to keep the score of thecorrect structure above the scores of the incorrectones by an amount which is proportional to howmuch they differ in accuracy.3 Generative model3.1 Eisner modelAs a generative framework we have chosen to usea variation of model C in (Eisner, 1996a). Inthis approach nodes are generated recursively ina top-down manner starting from the special sym-bol EOS (end of sentence). At any given node, leftand right children are generated as two separateMarkov sequences of nodes1, each conditioned onancestral and sibling information (which, for now,we will simply refer to as context).One of the relevant variations with respect tothe original model is that in our version the direc-tion of the Markov chain sequence is strictly leftto right, instead of the usual inside outwards.More formally, given a dependency structure T ,and any of its node N , the probability of generat-ing the fragment T (N) of the dependency struc-ture rooted in N is defined as:P (T (N)) =L∏l=1P (N2l)|context) · P (T (N2l))×R∏r=1P (N3r)|context) · P (T (N3r)) (2)where L and R are the number of left and rightchildren of N in T (L, R > 0), N2l is the leftdaughter of N at position l in T (analogously forright daughters). The probability of the entire de-pendency structure T is computed as P (T (EOS)).In order to illustrate how a dependency struc-ture can be decomposed into events, we presentin table 1 the list of events and the correspond-ing conditioning events extracted from the depen-dency structure illustrated in figure 1. In this sim-ple example, each node is identified with its word,and the context is composed of the direction withrespect to the head node, the head node, and thepreviously chosen daughter (or NONE if it is thefirst). While during the training phase the eventtables are updated with these events, in the testphase they are looked-up to compute the structureprobability, as in equation 1.1Every sequence ends with the special symbol EOC.239NObamaVwonDtheJpresidentialNelectionEOSFigure 1: Dependency tree of the sentence“Obama won the presidential election”.3.2 Model extensionIn equation 2 we have generically defined theprobability of choosing a daughter D based onspecific features associated with D and the con-text in which it occurs. In our implementation,this probability is instantiated as in equation 3.The specific features associated with D are: thedistance2 dist(H,D) between D and its head H ,the flag term(D) which specifies whether D hasmore dependents, and the lexical and postag repre-sentation of D. The context in which D occurs isdefined by features of the head node H , the previ-ously chosen sister S, the grandparent G, and thedirection dir (left or right).Equation 3 is factorized in four terms, each em-ploying an appropriate backoff reduction list re-ported in descending priority3.P (D|context) = (3)P (dist(H,D), term(D), word(D), tag(D)|H,S, G, dir) =P (tag(D)|H,S, G, dir)reduction list:wt(H), wt(S), wt(G), dirwt(H), wt(S), t(G), dir{wt(H), t(S), t(G), dirt(H), wt(S), t(G), dirt(H), t(S), t(G), dir× P (word(D)|tag(D), H, S, G, dir)reduction list:wt(H), t(S), dirt(H), t(S), dir× P (term(D)|word(D), tag(D), H, S, G, dir)reduction list:tag(D), wt(H), t(S), dirtag(D), t(H), t(S), dir× P (dist(P,D)|term(D), word(D), tag(D), H, S, G, dir)reduction list:word(D), tag(D), t(H), t(S), dirtag(D), t(H), t(S), dir2In our implementation distance values are grouped in 4categories: 1, 2, 3− 6, 7−∞.3In the reduction lists, wt(N) stands for the string in-corporating both the postag and the word of N , and t(N)stands for its postag. This second reduction is never appliedto closed class words. All the notation and backoff parame-ters are identical to (Eisner, 1996b), and are not reported herefor reasons of space.4The counts are extracted from a two-sentence corpuswhich also includes “Obama lost the election.”Events Freq. Conditioning Events Freq.won L EOS NONE 1 L EOS NONE 2EOC L EOS won 1 L EOS won 1EOC R EOS NONE 2 R EOS NONE 2Obama L won NONE 1 L won NONE 1EOC L won Obama 1 L won Obama 1election R won NONE 1 R won NONE 1EOC R won election 1 R won election 1EOC L Obama NONE 2 L Obama NONE 2EOC R Obama NONE 2 R Obama NONE 2the L election NONE 2 L election NONE 2presidential L election the 1 L election the 2EOC L election presidential 1 L election presidential 1EOC R election NONE 2 R election NONE 2EOC L the NONE 2 L the NONE 2EOC R the NONE 2 R the NONE 2EOC L presidential NONE 1 L presidential NONE 1EOC R presidential NONE 1 R presidential NONE 1Table 1: Events occurring when generating the de-pendency structure in figure 1, for the event space(dependent | direction, head, sister). According tothe reported frequency counts4, the structure has aassociated probability of 1/4.4 ResultsIn our investigation, we have tested our modelon the Wall Street Journal corpus (Marcus et al.,1993) with sentences up to 40 words in length,converted to dependency structures. Althoughseveral algorithms exist to perform such a conver-sion (Sangati and Zuidema, 2008), we have fol-lowed the scheme in (Collins, 1999). Section 2-21was used as training, and section 22 as test set.The MST discriminative parser was provided withthe correct postags of the words in the test set, andit was run in second-order5 and projective mode.Results are reported in table 2, as unlabeled attach-ment score (UAS). The MST dependency parserobtains very high results when employed alone(92.58%), and generates a list of k-best-candidateswhich can potentially achieve much better results(an oracle would score above 95% when selectingfrom the first 5-best, and above 99% from the first1000-best). The decrease in performance of thegenerative model, as the number of the candidateincreases, suggests that its performance would belower than a discriminative model if used alone.On the other hand, our generative model is able toselect better candidates than the MST parser, whentheir number is limited to a few dozens, yielding amaximum accuracy for k = 7 where it improvesaccuracy on the discriminative model by a 0.51%(around 7% error reduction).5The features of every dependency relation include infor-mation about the previously chosen sister of the dependent.240k-best Oracle best Oracle worst Reranked1 92.58 92.58 92.582 94.22 88.66 92.893 95.05 87.04 93.024 95.51 85.82 93.025 95.78 84.96 93.026 96.02 84.20 93.067 96.23 83.62 93.098 96.40 83.06 93.029 96.54 82.57 92.9710 96.64 82.21 92.96100 98.48 73.30 92.321000 99.34 64.86 91.4791.00%92.00%93.00%94.00%95.00%96.00%97.00%98.00%99.00%100.00%1 2 3 4 5 6 7 8 9 10 100 1000Oracle-BestRerankedMSTFigure 2: UAS accuracy of the MST discriminative and re-ranking parser on section 22 of the WSJ.Oracle best: always choosing the best result in the k-best, Oracle worst: always choosing the worst,Reranked: choosing the most probable candidate according to the generative model.5 ConclusionsWe have presented a general framework for depen-dency parsing based on a combination of discrim-inative and generative models. We have used thisframework to evaluate and compare several gener-ative models, including those of Eisner (1996) andsome of their variations. Consistently with earlierresults, none of these models performs better thanthe discriminative baseline when used alone. Wehave presented an instantiation of this frameworkin which our newly defined generative model leadsto an improvement of the state-of-the-art parsingresults, when provided with a limited number ofbest candidates. This result suggests that discrim-inative and generative model are complementary:the discriminative model is very accurate to filterout “bad” candidates, while the generative modelis able to further refine the selection among thefew best candidates. In our set-up it is now pos-sible to efficiently evaluate many other generativemodels and identify the most promising ones forfurther investigation. And even though we cur-rently still need the input from a discriminativemodel, our promising results show that pessimismabout the prospects of probabilistic generative de-pendency models is premature.Acknowledgments We gratefully acknowledgefunding by the Netherlands Organization forScientific Research (NWO): FS and RB arefunded through a Vici-grant “Integrating Cogni-tion” (277.70.006) to RB, and WZ through a Veni-grant “Discovering Grammar” (639.021.612) ofNWO. We also thank 3 anonymous reviewers foruseful comments.ReferencesS. Buchholz, and E. Marsi. 2006. CoNLL-X SharedTask on Multilingual Dependency Parsing. In Proc.of the 10th CoNLL Conference, pp. 149–164.M. Collins. 1999. Head-Driven Statistical Models forNatural Language Parsing. Ph.D. thesis, Universityof Pennsylvania.M. Collins, N. Duffy, and F. Park. 2002. New RankingAlgorithms for Parsing and Tagging: Kernels overDiscrete Structures, and the Voted Perceptron. In InProceedings of the ACL 2002, pp. 263–270.J. Eisner. 1996a. Three New Probabilistic Models forDependency Parsing: An Exploration. In Proc. ofthe 16th International Conference on ComputationalLinguistics (COLING-96), pp. 340–345.J. Eisner. 1996b. An Empirical Comparison of Proba-bility Models for Dependency Grammar. TechnicalReport number IRCS-96-11, Univ. of Pennsylvania.M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz.1993. Building a large annotated corpus of English:The Penn Treebank. In Computational Linguistics,19(2), pp. 313–330.R. McDonald. 2006. Discriminative Learning andSpanning Tree Algorithms for Dependency Parsing.Ph.D. thesis, University of Pennsylvania.J. Nivre and J. Hall. 2005. MaltParser: A Language-Independent System for Data-Driven DependencyParsing. In Proc. of the Fourth Workshop on Tree-banks and Linguistic Theories, pp. 137–148.J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nils-son,S. Riedel, and D. Yuret. 2007. The CONLL2007 shared task on dependency parsing. In Proc.of the CoNLL 2007 Shared Task Session, pp. 915–932.F. Sangati and W. Zuidema. 2008. UnsupervisedMethods for Head Assignments. In Proc. of theEACL 2009 Conference, pp. 701–709.241

A generative re-ranking model for dependency parsing

https://unora.unior.it/bitstream/11574/191035/1/Sangati_2009_A%20generative.pdf

A generative re-ranking model for dependency parsing

Abstract

Similar works

Full text

Available Versions

Università degli Studi di Napoli L'Orientale: CINECA IRIS