We describe DCU’s LFG dependencybased

metric submitted to the shared evaluation

task of WMT-MetricsMATR 2010.

The metric is built on the LFG F-structurebased

approach presented in (Owczarzak

et al., 2007). We explore the following

improvements on the original metric: 1)

we replace the in-house LFG parser with

an open source dependency parser that

directly parses strings into LFG dependencies;

2) we add a stemming module

and unigram paraphrases to strengthen the

aligner; 3) we introduce a chunk penalty

following the practice of METEOR to reward

continuous matches; and 4) we introduce

and tune parameters to maximize the

correlation with human judgement. Experiments

show that these enhancements improve

the dependency-based metric's correlation

with human judgement

Du, Jinhua

He, Yifan

van Genabith, Josef

Way, Andy

English

We describe DCU’s LFG dependencybased
metric submitted to the shared evaluation
task of WMT-MetricsMATR 2010.
The metric is built on the LFG F-structurebased
approach presented in (Owczarzak
et al., 2007). We explore the following
improvements on the original metric: 1)
we replace the in-house LFG parser with
an open source dependency parser that
directly parses strings into LFG dependencies;
2) we add a stemming module
and unigram paraphrases to strengthen the
aligner; 3) we introduce a chunk penalty
following the practice of METEOR to reward
continuous matches; and 4) we introduce
and tune parameters to maximize the
correlation with human judgement. Experiments
show that these enhancements improve
the dependency-based metric\u27s correlation
with human judgement

Irish Universities

The DCU dependency-based metric in WMT-MetricsMATR 2010

We describe DCU’s LFG dependencybased\ud
metric submitted to the shared evaluation\ud
task of WMT-MetricsMATR 2010.\ud
The metric is built on the LFG F-structurebased\ud
approach presented in (Owczarzak\ud
et al., 2007). We explore the following\ud
improvements on the original metric: 1)\ud
we replace the in-house LFG parser with\ud
an open source dependency parser that\ud
directly parses strings into LFG dependencies;\ud
2) we add a stemming module\ud
and unigram paraphrases to strengthen the\ud
aligner; 3) we introduce a chunk penalty\ud
following the practice of METEOR to reward\ud
continuous matches; and 4) we introduce\ud
and tune parameters to maximize the\ud
correlation with human judgement. Experiments\ud
show that these enhancements improve\ud
the dependency-based metric's correlation\ud
with human judgement

Name not available

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 349–353,Uppsala, Sweden, 15-16 July 2010. c©2010 Association for Computational LinguisticsThe DCU Dependency-Based Metric in WMT-MetricsMATR 2010Yifan He Jinhua Du Andy Way Josef van GenabithCentre for Next Generation LocalisationSchool of ComputingDublin City UniversityDublin 9, Ireland{yhe,jdu,away,josef}@computing.dcu.ieAbstractWe describe DCU’s LFG dependency-based metric submitted to the shared eval-uation task of WMT-MetricsMATR 2010.The metric is built on the LFG F-structure-based approach presented in (Owczarzaket al., 2007). We explore the followingimprovements on the original metric: 1)we replace the in-house LFG parser withan open source dependency parser thatdirectly parses strings into LFG depen-dencies; 2) we add a stemming moduleand unigram paraphrases to strengthen thealigner; 3) we introduce a chunk penaltyfollowing the practice of METEOR to re-ward continuous matches; and 4) we intro-duce and tune parameters to maximize thecorrelation with human judgement. Exper-iments show that these enhancements im-prove the dependency-based metric’s cor-relation with human judgement.1 IntroductionString-based automatic evaluation metrics such asBLEU (Papineni et al., 2002) have led directlyto quality improvements in machine translation(MT). These metrics provide an alternative to ex-pensive human evaluations, and enable tuning ofMT systems based on automatic evaluation results.However, there is widespread recognition inthe MT community that string-based metrics arenot discriminative enough to reflect the translationquality of today’s MT systems, many of whichhave gone beyond pure string-based approaches(cf. (Callison-Burch et al., 2006)).With that in mind, a number of researchers havecome up with metrics which incorporate more so-phisticated and linguistically motivated resources.Examples include METEOR (Banerjee and Lavie,2005; Lavie and Denkowski, 2009) and TERP(Snover et al., 2010), both of which now uti-lize stemming, WordNet and paraphrase informa-tion. Experimental and evaluation campaign re-sults have shown that these metrics can obtain bet-ter correlation with human judgements than met-rics that only use surface-level information.Given that many of today’s MT systems incor-porate some kind of syntactic information, it wasperhaps natural to use syntax in automatic MTevaluation as well. This direction was first ex-plored by (Liu and Gildea, 2005), who used syn-tactic structure and dependency information to gobeyond the surface level matching.Owczarzak et al. (2007) extended this line ofresearch with the use of a term-based encoding ofLexical Functional Grammar (LFG:(Kaplan andBresnan, 1982)) labelled dependency graphs intounordered sets of dependency triples, and calculat-ing precision, recall, and F-score on the triple setscorresponding to the translation and reference sen-tences. With the addition of partial matching andn-best parses, Owczarzak et al. (2007)’s methodconsiderably outperforms Liu and Gildea’s (2005)w.r.t. correlation with human judgement.The EDPM metric (Kahn et al., 2010) im-proves this line of research by using arc labelsderived from a Probabilistic Context-Free Gram-mar (PCFG) parse to replace the LFG labels,showing that a PCFG parser is sufficient for pre-processing, compared to a dependency parser in(Liu and Gildea, 2005) and (Owczarzak et al.,2007). EDPM also incorporates more informationsources: e.g. the parser confidence, the Porterstemmer, WordNet synonyms and paraphrases.Besides the metrics that rely solely on the de-pendency structures, information from the depen-dency parser is a component of some other metricsthat use more diverse resources, such as the textualentailment-based metric of (Pado et al., 2009).In this paper we extend the work of (Owczarzaket al., 2007) in a different manner: we use an349adapted version of the Malt parser (Nivre et al.,2006) to produce 1-best LFG dependencies andallow triple matches where the dependency la-bels are different. We incorporate stemming, syn-onym and paraphrase information as in (Kahn etal., 2010), and at the same time introduce a chunkpenalty in the spirit of METEOR to penalize dis-continuous matches. We sort the matches accord-ing to the match level and the dependency type,and weight the matches to maximize correlationwith human judgement.The remainder of the paper is organized as fol-lows. Section 2 reviews the dependency-basedmetric. Sections 3, 4, 5 and 6 introduce our im-provements on this metric. We report experimen-tal results in Section 7 and conclude in Section 8.2 The Dependency-Based MetricIn this section, we briefly review the metric pre-sented in (Owczarzak et al., 2007).2.1 C-Structure and F-Structure in LFGIn Lexical Functional Grammar (Kaplan and Bres-nan, 1982), a sentence is represented as both a hi-erarchical c-(onstituent) structure which capturesthe phrasal organization of a sentence, and a f-(unctional) structure which captures the functionalrelations between different parts of the sentence.Our metric currently only relies on the f-structure,which is encoded as labeled dependencies in ourmetric.2.2 MT Evaluation as Dependency TripleMatchingThe basic method of (Owczarzak et al., 2007) canbe illustrated by the example in Table 1.The metric in (Owczarzak et al., 2007) performstriple matching over the Hyp- and Ref-Triples andcalculates the metric score using the F-score ofmatching precision and recall. Let m be the num-ber of matches, h be the number of triples in thehypothesis and e be the number of triples in thereference. Then we have the matching precisionP = m/h and recall R = m/e. The score of thehypothesis in (Owczarzak et al., 2007) is the F-score based on the precision and recall of match-ing as in (1):Fscore =2PRP +R(1)Table 1: Sample Hypothesis and ReferenceHypothesisrice will be held talks in egypt next weekHyp-Triplesadjunct(will, rice)xcomp(will, be)adjunct(talks, held)xcomp(be, talks)adjunct(talks, in)obj(in, egypt)adjunct(week, next)adjunct(talks, week)Referencerice to hold talks in egypt next weekRef-Triplesobl(rice, to)obj(hold, to)adjunct(week, talks)adjunct(talks, in)obj(in, egypt)adjunct(week, next)obj(hold, week)2.3 Details of the Matching Strategy(Owczarzak et al., 2007) uses several techniquesto facilitate triple matching. First of all, consider-ing that the MT-generated hypotheses have vari-able quality and are sometimes ungrammatical,the metric will search the 50-best parses of boththe hypothesis and reference and use the pair thathas the highest F-score to compensate for parsernoise.Secondly, the metric performs complete or par-tial matching according to the dependency labels,so the metric will find more matches on depen-dency structures that are presumably more infor-mative.More specifically, for all except the LFGPredicate-Only labeled triples of the formdep(head, modifier), the method does notallow a match if the dependency labels (deps)are different, thus enforcing a complete match.For the Predicate-Only dependencies, par-tial matching is allowed: i.e. two triples are con-sidered identical even if only the head or themodifier are the same.Finally, the metric also uses linguistic resourcesfor better coverage. Besides using WordNet syn-onyms, the method also uses the lemmatized out-put of the LFG parser, which is equivalent to using350an English lemmatizer.If we do not consider these additional lin-guistic resources, the metric would find the fol-lowing matches in the example in Table 1:adjunct(talks, in), obj(in, egypt)and adjunct(week, next), as these threetriples appear both in the reference and in the hy-pothesis.2.4 Points for ImprovementWe see several points for improvement from Table1 and the analysis above.• More linguistic resources: we can use morelinguistic resources than WordNet in pursuitof better coverage.• Using the 1-best parse instead of 50-bestparses: the parsing model we currently usedoes not produce k-best parses and using onlythe 1-best parse significantly improves thespeed of triple matching. We allow ‘soft’triple matches to capture the triple matcheswhich we might otherwise miss using the 1-best parse.• Rewarding continuous matches: itwould be more desirable to reflectthe fact that the 3 matching triplesadjunct(talks, in), obj(in,egypt) and adjunct(week, next)are continuous in Table 1.We introduce our improvements to the metricin response to these observations in the followingsections.3 Producing and Matching LFGDependency Triples3.1 The LFG ParserThe metric described in (Owczarzak et al., 2007)uses the DCU LFG parser (Cahill et al., 2004)to produce LFG dependency triples. The parseruses a Penn treebank-trained parser to producec-structures (constituency trees) and an LFG f-structure annotation algorithm on the c-structureto obtain f-structures. In (Owczarzak et al., 2007),triple matching on f-structures produced by thisparadigm correlates well with human judgement,but this paradigm is not adequate for the WMT-MetricsMatr evaluation in two respects: 1) the in-house LFG annotation algorithm is not publiclyavailable and 2) the speed of this paradigm is notsatisfactory.We instead use the Malt Parser1 (Nivre et al.,2006) with a parsing model trained on LFG de-pendencies to produce the f-structure triples. Ourcollaborators2 first apply the LFG annotation algo-rithm to the Penn Treebank training data to obtainf-structures, and then the f-structures are convertedinto dependency trees in CoNLL format to trainthe parsing model. We use the liblinear (Fan etal., 2008) classification module to for fast parsingspeed.3.2 Hard and Soft Dependency MatchingCurrently our parser produces only the 1-bestoutputs. Compared to the 50-best parses in(Owczarzak et al., 2007), the 1-best parse limitsthe number of triple matches that can be found. Tocompensate for this, we allow triple matches thathave the same Head and Modifier to consti-tute a match, even if their dependency labels aredifferent. Therefore for triples Dep1(Head1,Mod1) and Dep2(Head2, Mod2), we allowthree types of match: a complete match ifthe two triples are identical, a partial match ifDep1=Dep2 and Head1=Head2, and a softmatch if Head1=Head2 and Mod1=Mod2.4 Capturing Variations in LanguageIn (Owczarzak et al., 2007), lexical variations atthe word-level are captured by WordNet. Weuse a Porter stemmer and a unigram paraphrasedatabase to allow more lexical variations.With these two resources combined, there arefour stages of word level matching in our sys-tem: exactmatch, stemmatch,WordNetmatch andunigram paraphrase match. The stemming mod-ule uses Porter’s stemmer implementation3 and theWordNet module uses the JAWS WordNet inter-face.4 Our metric only considers unigram para-phrases, which are extracted from the paraphrasedatabase in TERP5 using the script in the ME-TEOR6 metric.1http://maltparser.org/index.html2O¨zlem C¸etinog˘lu and Jennifer Foster at the NationalCentre for Language Technology, Dublin City University3http://tartarus.org/˜martin/PorterStemmer/4http://lyle.smu.edu/˜tspell/jaws/index.html5http://www.umiacs.umd.edu/˜snover/terp/6http://www.cs.cmu.edu/˜alavie/METEOR/3515 Adding Chunk Penalty to theDependency-Based MetricThe metric described in (Owczarzak et al., 2007)does not explicitly consider word order and flu-ency. METEOR, on the other hand, utilizes this in-formation through a chunk penalty. We introducea chunk penalty to our dependency-based metricfollowing METEOR’s string-based approach.Given a reference r = wr1...wrn, we denotewri as ‘covered’ if it is the head or modifier ofa matched triple. We only consider the wris thatappear as head or modifier in the referencetriples. After this notation, we follow METEOR’sapproach by counting the number of chunks inthe reference string, where a chunk wrj ...wrk isa sequence of adjacent covered words in the refer-ence. Using the hypothesis and reference in Ta-ble 1 as an example, the three matched triplesadjunct(talks, in), obj(in, egypt)and adjunct(week, next) will cover a con-tinuous word sequence in the reference (under-lined), constituting one single chunk:rice to hold talks (in) egypt next weekBased on this observation, we introduce a simi-lar chunk penalty Pen as in METEOR in our met-ric, as in 2:Pen = γ · ( #chunks#matches)β (2)where β and γ are free parameters, which we tunein Section 6.2. We add this penalty to the depen-dency based metric (cf. Eq. (1)), as in Eq. (3).score = (1− Pen) · Fscore (3)6 Parameter Tuning6.1 Parameters of the MetricIn our metric, dependency triple matches can becategorized according to many criteria. We as-sume that some matches are more critical thanothers and encode the importance of matches byweighting them differently. The final match willbe the sum of weighted matches, as in (4):m =∑λtmt (4)where λt and mt are the weight and number ofmatch category t. We categorize a triple match ac-cording to three perspectives: 1) the level of matchL={complete, partial}; 2) the linguistic resourceused in matchingR={exact, stem, WordNet, para-phrase}; and 3) the type of dependency D. Toavoid too large a number of parameters, we onlyallow a set of frequent dependency types, alongwith the type other, which represents all the othertypes and the type soft for soft matches. We haveD={app, subj, obj, poss, adjunct, topicrel, other,soft}.Therefore for each triple matchm, we can havethe type of the match t ∈ L×R×D.6.2 TuningIn sum, we have the following parameters to tunein our metric: precision weight α, chunk penaltyparameters β, γ, and the match type weightsλ1...λn. We perform Powell’s line search (Press etal., 2007) on the sufficient statistics of our metricto find the set of parameters that maximizes Pear-son’s ρ on the segment level. We perform the op-timization on the MT06 portion of the NIST Met-ricsMATR 2010 development set with 2-fold crossvalidation.7 ExperimentsWe experiment with four settings of the metric:HARD, SOFT, SOFTALL and WEIGHTED in or-der to validate our enhancements. The first twosettings compare the effect of allowing/not al-lowing soft matches, but only uses WordNet asin (Owczarzak et al., 2007). The third setting ap-plies our additional linguistic features and the finalsetting tunes parameter weights for higher correla-tion with human judgement.We report Pearson’s r, Spearman’s ρ andKendall’s τ on segment and system levels on theNIST MetricsMATR 2010 development set usingSnover’s scoring tool.7Table 2: Correlation on the Segment Levelr ρ τHARD 0.557 0.586 0.176SOFT 0.600 0.634 0.213SOFTALL 0.633 0.662 0.235WEIGHTED 0.673 0.709 0.277Table 2 shows that allowing soft triple matchesand using more linguistic features all leadto higher correlation with human judgement.Though the parameters might somehow overfit on7http://www.umiacs.umd.edu/˜snover/terp/scoring/352the data set even if we apply cross validation, thiscertainly confirms the necessity of weighing de-pendency matches according to their types.Table 3: Correlation on the System Levelr ρ τHARD 0.948 0.905 0.786SOFT 0.964 0.905 0.786SOFTALL 0.975 0.976 0.929WEIGHTED 0.989 1.000 1.000When considering the system-level correlationin Table 3, the trend is very similar to that of thesegment level. The improvements we introduce alllead to improvements in correlation with humanjudgement.8 Conclusions and Future WorkIn this paper we describe DCU’s dependency-based MT evaluation metric submitted to WMT-MetricsMATR 2010. Building upon the LFG-based metric described in (Owczarzak et al.,2007), we use a publicly available parser insteadof an in-house parser to produce dependency la-bels, so that the metric can run on a third partymachine. We improve the metric by allowing morelexical variations and weighting dependency triplematches depending on their importance accordingto correlation with human judgement.For future work, we hope to apply this methodto languages other than English, and performmorerefinement on dependency type labels and linguis-tic resources.AcknowledgementsThis research is supported by the Science Foundation Ireland(Grant 07/CE/I1142) as part of the Centre for Next Gener-ation Localisation (www.cngl.ie) at Dublin City University.We thank O¨zlem C¸etinog˘lu and Jennifer Foster for providingus with the LFG parsing model for the Malt Parser, as well asthe anonymous reviewers for their insightful comments.ReferencesSatanjeev Banerjee and Alon Lavie. 2005. METEOR: Anautomatic metric for MT evaluation with improved corre-lation with human judgments. In Proceedings of the ACLWorkshop on Intrinsic and Extrinsic Evaluation Measuresfor Machine Translation and/or Summarization, pages65–72, Ann Arbor, MI.Aoife Cahill, Michael Burke, Ruth O’Donovan, Josef vanGenabith, and Andy Way. 2004. Long-distance depen-dency resolution in automatically acquired wide-coveragePCFG-based LFG approximations. In Proceedings of the42nd Meeting of the Association for Computational Lin-guistics (ACL-2004), pages 319–326, Barcelona, Spain.Chris Callison-Burch, Miles Osborne, and Philipp Koehn.2006. Re-evaluation the role of bleu in machine trans-lation research. In Proceedings of 11th Conference of theEuropean Chapter of the Association for ComputationalLinguistics, pages 249–256, Trento, Italy.Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-RuiWang, and Chih-Jen Lin. 2008. Liblinear: A library forlarge linear classification. Journal of Machine LearningResearch, 9:1871–1874.Jeremy G. Kahn, Matthew Snover, and Mari Ostendorf.2010. Expected dependency pair match: predicting trans-lation quality with expected syntactic structure. MachineTranslation.Ronald M. Kaplan and Joan Bresnan. 1982. Lexical-functional grammar: A formal system for grammaticalrepresentation. The mental representation of grammaticalrelations, pages 173–281.Alon Lavie andMichael J. Denkowski. 2009. he meteor met-ric for automatic evaluation of machine translation. Ma-chine Translation, 23(2-3).Ding Liu and Daniel Gildea. 2005. Syntactic featuresfor evaluation of machine translation. In Proceedings ofthe ACL Workshop on Intrinsic and Extrinsic EvaluationMeasures for Machine Translation and/or Summarization,pages 25–32, Ann Arbor, MI.Joakim Nivre, Johan Hall, and Jens Nilsson. 2006. Malt-parser: A data-driven parser-generator for dependencyparsing. In In The fifth international conference on Lan-guage Resources and Evaluation (LREC-2006), pages2216–2219, Genoa, Italy.Karolina Owczarzak, Josef van Genabith, and Andy Way.2007. Labelled dependencies in machine translation eval-uation. In Proceedings of the Second Workshop on Statis-tical Machine Translation, pages 104–111, Prague, CzechRepublic.Sebastian Pado, Michel Galley, Dan Jurafsky, and Christo-pher D. Manning. 2009. Robust machine translationevaluation with entailment features. In Proceedings ofthe Joint Conference of the 47th Annual Meeting of theACL and the 4th International Joint Conference on Natu-ral Language Processing of the AFNLP, pages 297–305,Suntec, Singapore.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-JingZhu. 2002. Bleu: a method for automatic evaluationof machine translation. In Proceedings of 40th AnnualMeeting of the Association for Computational Linguistics(ACL-2002), pages 311–318, Philadelphia, PA.William H. Press, Saul A. Teukolsky, William T. Vetterling,and Brian P. Flannery. 2007. Numerical Recipes 3rd Edi-tion: The Art of Scientific Computing. Cambridge Univer-sity Press, New York, NY.Matthew Snover, Nitin Madnani, Bonnie Dorr, and RichardSchwartz. 2010. Ter-plus: paraphrase, semantic, andalignment enhancements to translation edit rate. MachineTranslation.353

DCU Online Research Access Service

Bleu: a method for automatic evaluation of machine translation.

Expected dependency pair match: predicting translation quality with expected syntactic structure. Machine Translation.

he meteor metric for automatic evaluation of machine translation.

Labelled dependencies in machine translation evaluation.

Lexicalfunctional grammar: A formal system for grammatical representation. The mental representation of grammatical relations,

Liblinear: A library for large linear classiﬁcation.

Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations.

Maltparser: A data-driven parser-generator for dependency parsing. In

METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.

Numerical Recipes 3rd Edition: The Art of Scientiﬁc Computing.

Re-evaluation the role of bleu in machine translation research.

Robust machine translation evaluation with entailment features.

Syntactic features for evaluation of machine translation.

Ter-plus: paraphrase, semantic, and alignment enhancements to translation edit rate. Machine Translation.

http://doras.dcu.ie/15796/1/The_DCU_Dependency-Based_Metric_in_WMT-MetricsMATR_2010.pdf

The DCU dependency-based metric in WMT-MetricsMATR 2010

Abstract

Similar works

Full text

Available Versions

Irish Universities

Name not available

DCU Online Research Access Service