In this article, we describe the submissions

of Team Szeged for the EPE 2017 shared

task. We introduce three approaches and

first experiments to exploit the opportunities

of the general dependency graph representation

of the shared task

Farkas, Richárd

Szántó, Zsolt

Repository of the Academy's Library

Team Szeged @ EPE 2017 shared task -First experiments in a generalized syntactic parsing frameworkZsolt Szanto´, Richa´rd FarkasUniversity of Szeged, Institute of Informatics{szantozs, rfarkas}@inf.u-szeged.huAbstractIn this article, we describe the submissionsof Team Szeged for the EPE 2017 sharedtask. We introduce three approaches andfirst experiments to exploit the opportuni-ties of the general dependency graph rep-resentation of the shared task.1 IntroductionThe goal of the First Shared Task on ExtrinsicParser Evaluation (EPE 2017) was to estimate “therelative utility of different types of dependencyrepresentations for a variety of downstream ap-plications that depend heavily on the analysis ofgrammatical structure.”To enable different types of dependency repre-sentations, the organizers of the shared task in-troduced a very general graph-based representa-tion of ’relational’ structure reflecting syntactic-semantic analysis. The nodes of this graph cor-respond to lexical units, and its edges repre-sent labeled directed relations between two nodes.Nodes can be defined in any terms of (in principlearbitrary) sub-strings of the surface form of the in-put sentence. This representation allows overlap-ping and empty (i.e. zero-span) node sub-stringsas well. Moreover, nodes and edges are labeled byattributevalue maps without any restriction on theattribute set.This very general graph-based representationopens brand new ways for expressing syntactic orsemantic information besides the standard depen-dency tree formalism. We understood the call ofthe shared task in a generalized way and came upwith ideas which aim to leverage the opportunitiesof the general representation beyond dependencyparse trees. We experimented with couple of suchideas (instead of trying to achieve high scores inthe shared task) and we shall introduce them inthis paper.The contribution of this work consists of threeindependent pack of experiments in the EPE 2017setting. We emphasize that these are only first ex-perimental results and there is a plenty of spacefor analyzing these approaches and also to comeup with other ideas leveraging the broadly under-stood task specification.In the first set of experiments (§2), we start fromthe classic dependency parsing approach but in-stead of a single dependency parse we express thedistribution of possible dependency parses givena sentence in the graph-based general representa-tion. In Section 3, we introduce a possible solutionfor enriching the dependency parse by constituentinformation given by a standard phrase-structureparser. In this way, various syntactic representa-tions can be represented in the graph and infor-mation is not lost because the downstream appli-cation can only accept a single dependency parsetree. Furthermore, in the EPE 2017 setting we cansend a blended relational structure to the down-stream task, like a parse distribution and blendedversion of different syntactic approaches, and thedownstream application is able to machine learnwhich type of syntactic structure or phenome oreven which combination of syntactic informationis useful for itself.Our last batch of experiment (§4) is a conse-quence of this objective, i.e. the relational repre-sentation has to be useful for the downstream ap-plication. Here, we tried to automatically recog-nize which dependency parse labels are useful to adownstream task and collapsed the useless ones.(a) (b) (c)Figure 1: Alternatives for representing constituent trees in the general dependecy graph format.2 Parse Distribution as Input forDownstream ApplicationsStandard dependency parsers output a single de-pendency parse tree. Our hypothesis was that adownstream application could profit from havingaccess to the distribution of possible parses andnot just to the most likely parse tree. The distribu-tion over possible parses estimated by the parsingmodel might be useful for a downstream applica-tion because it might reveal that edges or their la-bels are less confident or also point out relativelyhighly probably dependencies which are not partof the best single parse tree. The general graph-based representation of EPE 2017 enables to ex-press the distribution over possible edges and la-bels, i.e. possible parses.Getting out the density estimation from a par-ticular parser is usually complicated because ofboth theoretical and practical (software implemen-tation) issues. Hence we decided to use an approx-imation of edge and label likelihoods based on top-K parses for our first experiments. Our assump-tion here is that the top K output of a parser modelcontains most of the useful bi-lexical dependen-cies and the frequency of a particular dependencycounted among the K parses is a good enough ap-proximation for its likelihood (this idea is similarto constituent-level strategy of the Berkeley prod-uct parser (Petrov, 2010)).We added each edges from the k-best trees of aparser to the general dependency graph. We alsoadded a new label to all edges whose value is thefrequency of the same edge label pairs among thek parses. For these experiments we used the MST-Parser (McDonald et al., 2005) what we trainedon Universal Dependencies v2 (Nivre et al., 2015)and we asked for the 10-best trees with default pa-rameters.3 Constituents in the (Bi-Lexical)Relational RepresentationConstituency parsers focus on thephrases/constituents and phrase structure ofthe sentence, i.e. follow a non bi-lexical syntacticrepresentation. Several applications might preferbi-lexical representations (like the ones basedon predicate-argument structures) while othersmight prefer constituency (like scope detection).Fortunately, the general graph representation ofEPE 2017 enables us to put both the dependencyand constituency parse output into a blendedsyntactic graph. Hence we do not have to choosebetween the two approaches but the downstreamapplication can machine learn which syntacticphenomena is useful for itself or even can learnpatterns in the graph consisting of informationfrom both constituency and dependency. Coupleof previous work has shown that the two syntacticrepresentation and their parsers can work togetherefficiently cf. (Farkas and Bohnet, 2012). Webelieve that is especially true for using themjointly in downstream applications.There are many possible ways how we can rep-resent a constituent tree in the general dependencygraph format. Although these representations con-tain the same information because of the featureextractors of the downstream applications they canhave different effect in practice.An interesting opportunity of the EPE 2017general graph representaton is that it enables thecreation of virtual nodes. This feature gives theposibility to create a new node for each non-terminal in a constituent tree. Our three proposaldiffers in the way these virtual nodes are linked tothe overt nodes in the graph.1. In the first setup, shown by the Figure 1a, weconnect each of the children to their directPRNNMODPRP LOCAPPOCOORD CONJSUBADVOBJDEPLGSVCOPRDAMODPPRDIMMNR49.9949.849.8749.9850.1549.9649.9150.0449.9549.9849.9150.2450.150.0649.9449.7750.12Figure 2: Label adjustment graph.parents. In this way our graphs will be verysimilar to a constituent tree. In this examplethe Word 2 and Word 3 are connected to anNP, that NP and Word 1 is connected to theanother NP.2. Another posibility is when each of the nodesare connected to all ancestor non-terminals(Figure 1b). In that case there is a direct rela-tion between a constituent and their descen-dants. In the current example the higher levelNP directly contains the children (Word 2,Word 3) of its child. This representation hasthe hope that the feature extractor of down-stream applications can directly generate fea-tures about the ancestors without recursiverules3. A different approach is where we give thecovering area for each new non-terminal(Figure 1c). In these cases like in the previ-ous we have not got direct information aboutthe connection between the nonterminals. Onthe other hand, it can help for an applicationwhich uses the position of a node.For constituent parsing we used the BerkeleyParser (Slav Petrov and Klein, 2006) with de-fault parameters and pretrained model (eng-sm6).In our submission we used the second and thirdmethods in the dependency graph format.4 Label Set Adjustment Driven byDownstream ApplicationsDifferent downstream applications might utilizedifferent type of grammatical patterns. The sim-plest case is that a downstream application mightextract important features from particular edge la-bels while features over other edge labels are neg-ligible in its machine learnt model. Moreover, dif-ferent applications might utilize different type ofdependencies, see for example event recognitionversus negation scope detection.We propose a simple procedure to recognizeedge labels which can be collapsed into other edgelabels because their discrimination does not giveany added value to the downstream application inquestion. We start from the full set of edge labelsand systematically check what is the effect of col-lapsing two particular labels evaluated through thedownstream application.We calculated for all label pairs what is goingto happen if we replace one dependency label toanother. For our experiments we used the TEESsystem, but because we did not have enough timeto retrain the TEES system for all combination, wetrained it once with the full label set and we didthe prediction part separatly to each dependencylabel pairs. In this prediction part we replacedeach of the labels with each of another labels onthe full development set. We got a complete di-rected graph where the nodes are the labels andedges contains the scores from the TEES systemwith the merged labels. For each node we kept theoutcoming edge with maximum weight i.e. whenthe replace was the most efficient. When therewere two edges between two node we removed thesmaller.Figure 2 contains the graph we got. (Whenwe ran the TEES system with default parame-ters we got 49.76 with original labels). By us-ing this graph we started replacing the nodes fromthe highest edge weight to the lowest. We evalu-ated the new labelset in every step and we foundthe best result after three steps, 50.36, which isslightly better than the best merged pair. After thatwe did the three replace steps in the full dataset.Because of lack of time, we cannot make the re-placement in the full dataset and retrain the TEESbefore the submission deadline. We sent the treeswith collapsed labelset. We found lower scoresthan the baseline on the TEES test set. We alsoEvent Extraction Negation Resolution Opinion Analysismate - baseline 47.84 61.98 65.87mate + label adjustment 47.37 60.53 66.33mate + constituent 46.71 61.26 63.13mate + mst - baseline 46.69 59.78 63.25mate + mst - k-best 45.96 59.05 62.5Table 1: Final result in evalutaion set.could not use this method for the other down-stream applications.5 ResultsThe Table 1 shows our official results achived onthe shared task. The baseline - mate is one ofour baselines where we just run the mate parser(Bohnet, 2010) with pretrained model. The secondand third rows contain the result of label adjust-ment and constituent parsing experiments. Thefourth row contains another baseline when we ap-plied the MSTParser and the last row is shows thescores of our k-best experiment (we used MST-Parser here).Unfortunatelly, the deeper analysis of results isleft for future work. For example, the reason whythe combination of k-best parse get lower resultin every task than the baseline-MST is maybe thefeature extractors were prepared for dependencyrepresentations and not for distribution graphs.5.1 Event ExtractionIn the event extraction task we can not beat ourbaselines, all of our modifications – including thelabel adjustment which is optimized for this task– get negative effect. The dependency label merg-ing mechanism what we directly developed on thistask also failed.5.2 Negation ResolutionOne of the main motivation of the constituentbased approach was the negation resolution task.The scope of the negations are usually a closewhat can we identify with constituent parsers.This constituency-based system got better result inthree out of four scope-focused evaluation metricthan our baseline. Table 2 shows the detailed com-parison of the baseline and the constituent system.The following example shows how can the con-stituent parse help:“I join in it because there is no other way in theworld by which justice can be gained.”baseline mate + constdev test dev testScope Match 78.42 80.00 77.98 81.14Scope Tokens 86.64 89.17 87.38 89.27Event Match 75.47 67.90 72.90 65.20Full Negation 62.15 61.98 59.91 61.26Table 2: Detailed results of the baseline - mate andthe mate + constituent systems in negation resolu-tion task.The scope of the no negation clue starts from thethere and end with the gained word. Our baselinesystem marked the negation from the there to thejustice, but the constituent based method found thecorrect scope. If we look at the constituent tree wesee the full scope is covered by a constituent withS label. Instead of scope detection the constituentbased information can’t help in the event detectionsubtask.5.3 Opinion AnalysisIn the opinion analysis task the label adjustmentmethod imporved 0.5 percetage point against themate-baseline and got the best results in the sharedtask in Holders (In Vitro) metric. It seems the la-bel collapsions what our method found in the eventextraction task is more general than we expected.On the other hand, it is still an open question whythis label collapsion did not work at the event ex-traction task’s evalaution set.6 ConclusionsWe introduced the contribution of team Szeged tothe EPE 2017 Shared Task. We proposed threeapproaches for relational representation of syntaxbeyond the canonical dependency parse tree ap-proach. Although these experiments are only thevery first tries for such representation we hope thatthis might give ideas about the important topic ofsyntactic and semantic parser solutions which aimto be (automatically) fine-tuned for a particulardownstream applications demands.AcknowledgmentsWe thank to the organizers of the EPE shared tasktheir hard work on extending the feature extrac-tor of downstream applications according to ourweired ideas.The research work of Richrd Farkas has beensupported by the Jnos Bolyai scholarship of theHungarian Academy of Sciences.ReferencesBernd Bohnet. 2010. Very high accuracy and fast de-pendency parsing is not a contradiction. In Pro-ceedings of the 23rd International Conference onComputational Linguistics. Association for Compu-tational Linguistics, Stroudsburg, PA, USA, COL-ING ’10, pages 89–97.Richa´rd Farkas and Bernd Bohnet. 2012. Stacking ofdependency and phrase structure parsers. In Pro-ceedings of COLING 2012. The COLING 2012 Or-ganizing Committee, Mumbai, India, pages 849–866.Ryan McDonald, Fernando Pereira, Kiril Ribarov, andJan Hajicˇ. 2005. Non-projective dependency pars-ing using spanning tree algorithms. In Proceedingsof the Conference on Human Language Technologyand Empirical Methods in Natural Language Pro-cessing. Association for Computational Linguistics,Stroudsburg, PA, USA, HLT ’05, pages 523–530.Joakim Nivre, Zˇeljko Agic´, Maria Jesus Aranzabe,Masayuki Asahara, Aitziber Atutxa, Miguel Balles-teros, John Bauer, Kepa Bengoetxea, Riyaz AhmadBhat, Cristina Bosco, et al. 2015. Universal depen-dencies 1.2 .Slav Petrov. 2010. Products of random latent vari-able grammars. In Human Language Technologies:The 2010 Annual Conference of the North AmericanChapter of the Association for Computational Lin-guistics. Association for Computational Linguistics,Stroudsburg, PA, USA, HLT ’10, pages 19–27.Leon Barrett Romain Thibaux Slav Petrov and DanKlein. 2006. Learning accurate, compact, andinterpretable tree annotation. In Proceedings ofCOLING-ACL. Association for Computational Lin-guistics, Sydney, Australia, pages 433–440.

Team Szeged @ EPE 2017 shared task - First experiments in a generalized syntactic parsing framework

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library