In this paper, we give a description of the machine translation system developed at DCU that was used for our participation in the NLP Tools Contest of the International

Conference on Natural Language Processing (ICON 2008). This was our first ever attempt at working on any Indian language. In this participation, we focus on various techniques for word and phrase alignment to improve system quality. For the English-Hindi translation task we exploit

source-language reordering. We also carried out experiments combining both in-domain and out-of-domain data to improve

the system performance and, as a post-processing step we transliterate out-of-vocabulary items

Haque, Rejwanul

Naskar, Sudip Kumar

Srivastava, Ankit Kumar

Way, Andy

English

DCU Online Research Access Service

MaTrEx: The DCU Machine Translation System for ICON 2008Ankit Kumar Srivastava, Rejwanul Haque, Sudip Kumar Naskar, Andy WayCentre for Next Generation LocalisationSchool of ComputingDublin City UniversityDublin, Ireland{asrivastava, rhaque, snaskar, away}@computing.dcu.ieAbstractIn this paper, we give a description of themachine translation system developed atDCU that was used for our participationin the NLP Tools Contest of the Inter-national Conference on Natural LanguageProcessing (ICON 2008). This was ourfirst ever attempt at working on any Indianlanguage. In this participation, we focuson various techniques for word and phrasealignment to improve system quality. Forthe English–Hindi translation task we ex-ploit source-language reordering. We alsocarried out experiments combining bothin-domain and out-of-domain data to im-prove the system performance and, as apost-processing step we transliterate out-of-vocabulary items.1 IntroductionIn this paper, we describe some basic experimentscarried out on the English–Hindi translation taskas well as test the data-driven hybrid MT systemdeveloped at DCU, MATREX (Machine Transla-tion using examples) (Stroppa and Way, 2006), onthe English–Hindi language pair.The remainder of the paper is organized as fol-lows. Section 2 describes the various componentsof the MATREX system. In Section 3, we presentthe system configuration for both the EILMT andTIDES data. We describe the experiments con-ducted in Section 4, reporting the results in Sec-tion 5, followed by discussion and avenues for fu-ture research in Section 6.2 The MATREX SystemThe MATREX system is a modular hybrid data-driven MT system which exploits aspects of boththe EBMT and SMT paradigms. It consists of anumber of extendible and re-implementable mod-ules and features, some of which are Word Align-ment Module, Word packing (Ma et al., 2007),Chunking and Chunk Alignment Module (Tins-ley et al., 2008), Treebank-based phrase extrac-tion (Tinsley et al., 2007), Supertagging (Has-san et al., 2007), Source-context features (Stroppaet al., 2007), and Decoder.In some cases, these modules may comprisewrappers around pre-exisiting software. For ex-ample, our system configuration for the translationtask incorporates a wrapper around GIZA++ (Ochand Ney, 2003) for word alignment and a wrapperaround Moses (Koehn et al., 2007) for decoding.The system also includes language-specific exten-sions such as taggers, parsers, etc. used in pre-processing and post-processing modules describedin the next section.The MATREX system makes use of marker-based chunking, which is based on the MarkerHypothesis (Green, 1979), a psycholinguistic con-straint which posits that all languages are markedfor surface syntax by a specific closed set of lex-emes or morphemes which signify context. Usinga set of closed-class (or “marker”) words, such asdeterminers, conjunctions, prepositions, posses-sive and personal pronouns, aligned source-targetsentences are segmented into chunks (Gough andWay, 2004) during a pre-processing step. A chunkis created at each new occurrence of a markerword, with the restriction that each chunk mustcontain at least one content (or non-marker) word.In order to align the chunks obtained by the chunk-“Proceedings of ICON-2008: 6th International Conference on Natural Language Processing, MacmillanPublishers, India. Also accessible from http://ltrc.iiit.ac.in/proceedings/ICON-2008”ing procedures, the system makes use of an “edit-distance style” dynamic programming alignmentalgorithm.Many features available in our system which,for one reason or another, were not exploited forthe purposes of this translation task.3 System DescriptionThe following section describes the system setupfor the TIDES and EILMT data.3.1 Pre-processingWe carried out an analysis of the training data.We filtered out sentence pairs based on length(>100 words) and fertility (2:1 word ratio). Wefound the EILMT corpus to be cleaner than theTIDES corpus (highest E–H word ratio 4.7 vs.16.5, lowest E–H word ratio .17 vs. .08). Detailsof these statistics are given in Table 1.TIDES EILMTTotal sentences 50000 7000|En| >100 or, |Hn| >100 123 1Fertility>2 314 32Highest E–H word ratio 16.5 4.7Lowest E–H word ratio .08 .17Highest E–H char ratio 82.5 5.6Lowest E–H char ratio .17 .23Table 1: Summary of pre-processing of trainingdataReordering of phrases during translation is typ-ically managed by distortion models, which havenot proved to be entirely satisfactory (Collins etal., 2006), especially for language pairs that dif-fer a lot in terms of word order. The English–Hindi language pair exhibits longer distance SOV-SVO syntactic divergence. In this work, targetlanguage-specific heuristics are applied to reorderEnglish syntatic trees in the pre-processing step.This reduces, and often eliminates, the distor-tion load during training. An open source proba-bilistic parser (Stanford Parser1 version 1.6.1) wasused to parse the English sentences. The parsetree is traversed for transformation of the Eng-lish phrase pattern (part-of-speech tag and phrase-based schema form) to the Hindi phrase pattern.English-to-Hindi phrase transformation rules areapplied to the intermediate nodes (representing1http://nlp.stanford.edu/software/lex-parser.shtml/phrases). A CRF-based chunker2 was used forproper reordering of VPs (usually for complex andcompound sentences). We also augment the train-ing data with an English–Hindi lexicon of 40K en-tries.3.2 System ConfigurationAs mentioned in Section 2, our word alignmentmodule employs a wrapper around GIZA++.We built a 5-gram language model based on thetarget side of the training data. This was done us-ing the SRI Language Modelling toolkit (Stolcke,2002) employing linear interpolation and modi-fied Kneser-Ney discounting (Chen and Goodman,1996).Our phrase-table comprised of word alignment-based phrase pairs extracted using the grow-diag-final method (Koehn et al., 2003), with a maxi-mum phrase length of 7 words. Phrase transla-tion probabilities were estimated by relative fre-quency over all phrase pairs and were combinedwith other features, such as a reordering model,in a log-linear combination of features. We tunedour system on the development set provided usingminimum error-rate training (Och, 2003) to opti-mise the BLEU score. Finally, we carried out de-coding using a wrapper around the Moses decoder.3.3 Post-processingThe translated output contained many OOV wordsincluding named entities and unseen words. TheOOV words are transliterated using a modifiedjoint source channel-based transliteration module(Ekbal et al., 2006) in a bid to maximize matcheswith the reference.4 ExperimentsDue to time constraints, we could not run the sameset of experiments on both data sets. In case of ex-periments on TIDES data, the 1K development setwas divided into a 400-sentence development setand a 600-sentence test set for the first three exper-iments; only the last experiment was carried out onthe released 1K test set and tuned on the the entiredevelopment set. For the EILMT data, we first ranthe baseline MOSES (VANILLA) on the test datawithout any pre-processing or post-processing.Then we incrementally applied filtering (FIL), re-ordering of the source side (REO), augmentation2http://crfchunker.sourceforge.net/Experiment BLEU NIST METEOR WER PERVANILLA 15.68 4.90 32.05 79.89 54.92+ TRA 15.79 4.94 53.91 79.73 54.61FIL (Baseline) 16.35 4.99 31.6 78.92 54.32REO 17.06 4.88 32.11 76.20 55.56+ TRA 17.14 4.92 51.61 76.02 55.27REO + FAC 13.36 4.59 30.25 75.16 57.55REO + LEX 17.33 4.98 35.83 76.03 54.93+ TRA 17.41 5.02 53.64 75.85 54.67Table 2: Summary of the results on EILMT test dataExperiment BLEU NIST METEOR WER PERVANILLA 4.87 2.69 22.86 93.83 76.65REO 8.70 3.81 35.47 87.42 68.45REO + LEX 8.63 3.84 35.45 87.22 68.16REO + FAC 8.94 4.38 28.93 82.10 62.92FIL + REO + LEX 10.47 4.23 31.50 87.61 6.34FIL + REO + LEX + TRA 10.49 4.24 37.10 87.57 66.27Table 3: Summary of the results on TIDES test dataof the training set with lexicon (LEX), and translit-eration (TRA) to factor out the individual effects ofeach of these techniques. We also tested factoredmodels (Koehn and Hoang, 2007)(FAC) with dif-ferent factors (word, root/lemma (4 length), Part-of-Speech, morph). These were obtained using theStanford Parser and WordNet 3 on the source side,and the IIIT Shallow Parser 4 on the target side.5 ResultsWe initially carried out the same experiments onthe Hindi data in both wx format and utf format,in which the latter format always gave us better re-sults. Therefore we conducted all the experimentson utf Hindi data only. System performance isevaluated with respect to BLEU, NIST, METEOR,WER and PER. Results on the EILMT data andTIDES data are given in Tables 2 and 3 respec-tively. As expected, each of the pre-processingand post-processing modules proved to be use-ful in improving performance. MOSES withoutany pre-processing or post-processing produces aBLEU score of 15.68 (opposed to 17.70 reportedby the organiser) on the EILMT data set. Sincefiltering the training data improves performanceacross all evaluation metrics for all the experi-ments, MOSES on filtered data serves as our new3http://wordnet.princeton.edu/4http://ltrc.iiit.ac.in/analyzer/baseline. Our best results (17.41 BLEU score onEILMT data and 10.49 on TIDES data) are ob-tained when we apply all the pre-processing andpost-processing modules collectively. On the con-trary, factored models fall way short of the base-line system. Results on the TIDES corpus areatleast 6 BLEU points lower than the results ob-tained on the EILMT corpus.6 DiscussionWe conducted a number of experiments with dif-ferent settings, and observed that data in the utfformat always gives better results. It is also ev-ident that filtering out noise from the trainingdata helps and that reordering is an important pre-processing step which improved the baseline sys-tem performance. In the post-processing phase,transliteration is performed which increases theaccuracy slightly. All the pre-processing and post-processing modules improve system performaceindividually and the best result is obtained whenthey are applied collectively. Poor results on theTIDES data perhaps indicate the domain depen-decy of SMT systems. Another interesting ob-servation is that even with 4 features the factoredmodel failed to reach the baseline accuracy ob-tained by MOSES.Furthermore, on manual evaluation of a randomset of 60 sentences, it was found that the outputtranslations look better than these scores suggest,i.e. given the relatively free word order (used withemphasis and complex structures) in Hindi, pro-viding just one reference translation set for evalu-ation is somewhat arbitrary.Due to time constraints we could not apply theEBMT module (which is a unique feature of theMATREX system) in the experiments, but we hopeto report results using this method at the confer-ence itself.AcknowledgmentsThis work is supported by Science Foundation Ire-land (grant number: 07/CE/I1142).ReferencesChen, S. F. and Goodman, J. (1996). An EmpiricalStudy of Smoothing Techniques for Language Mod-eling. In Proceedings of the Thirty-Fourth AnnualMeeting of the Association for Computational Lin-guistics, pages 310–318, San Francisco, CA.Collins, M., Koehn, P., and Kucerova, I. (2006). ClauseRestructuring for Statistical Machine Translation. InProceedings of ACL-2006, pages 531–540, Sydney,Australia.Ekbal, A., Naskar, S.K., and Bandyopadhyay, S.(2006). A modified joint source-channel model fortransliteration. In Proceedings of the Coling/ACL2006 Main Conference Poster Sessions,, pages 191–198, Sydney, Australia.Green, T. (1979). The Necessity of Syntax Markers.Two experiments with artificial languages. Journalof Verbal Learning and Behavior, 18:481–496.Gough, N. and Way, A. (2004). Robust Large-ScaleEBMT with Marker-Based Segmentation. In Pro-ceedings of the 10th International Conference onTheoretical and Methodological Issues in MachineTranslation (TMI-04), pages 95–104, Baltimore,MD.Hassan, H., Sima’an, K., and Way, A. (2007). Su-pertagged Phrase-based Statistical Machine Trans-lation. In Proceedings of the 45th Annual Meet-ing of the Association for Computational Linguistics(ACL’07), pages 288–295, Prague, Czech Republic.Koehn, P., Och, F. J., and Marcu, D. (2003). Statis-tical Phrase-Based Translation. In Proceedings ofthe 2003 Conference of the North American Chapterof the Association for Computational Linguistics onHuman Language Technology (NAACL ’03), pages48–54, Edmonton, Canada.Koehn, P., Hoang, H., Birch, A., Callison-Burch, C.,Federico, M., Bertoldi, N., Cowan, B., Shen, W.,Moran, C., Zens, R., Dyer, C., Bojar, O., Con-stantin, A., and Herbst, E. (2007). Moses: OpenSource Toolkit for Statistical Machine Translation.In Annual Meeting of the Association for Compu-tational Linguistics (ACL), demonstration session,pages 177–180, Prague, Czech Republic.Koehn, P. and Hoang, H. (2007). Factored Trans-lation Models. In Proceedings of the 2007 JointConference on Empirical Methods in Natural Lan-guage Processing and Computational Natural Lan-guage Learning (EMNLP-CoNLL ’07), pages 868–876, Prague, Czech Republic.Ma, Y., Stroppa, N., and Way, A. (2007). Boostrap-ping Word Alignment via Word Packing. In Pro-ceedings of the 45th Annual Meeting of the Associa-tion for Computational Linguistics (ACL’07), pages304–311, Prague, Czech Republic.Och, F. (2003). Minimum error rate training in sta-tistical machine translation. In Proceedings of the41st Annual Meeting of the Association for Com-putational Linguistics (ACL), pages 160–167, Sap-poro, Japan., Sapporo, Japan.Och, F. J. and Ney, H. (2003). A Systematic Compari-son of Various Statistical Alignment Models. Com-putational Linguistics, 29(1):19–51.Stolcke, A. (2002). SRILM - An Extensible Lan-guage Modeling Toolkit. In Proceedings of the In-ternational Conference Spoken Language Process-ing, Denver, CO.Stroppa, N. and Way, A. (2006). MaTrEx: the DCUmachine translation system for IWSLT 2006. InProceedings of the International Workshop on Spo-ken Language Translation, pages 31–36, Kyoto,Japan.Stroppa, N., van den Bosch, A., and Way, A.(2007). Exploiting Source Similarity for SMT us-ing Context-Informed Features. In Proceedings ofthe 11th International Conference on Theoreticaland Methodological Issues in Machine Translation(TMI-07), pages 231–240, Sko¨vde, Sweden.Tinsley, J., Hearne, M., and Way, A. (2007). Exploit-ing Parallel Treebanks to Improve Phrase-Based Sta-tistical Machine Translation. In Proceedings of theSixth International Workshop on Treebanks and Lin-guistic Theories (TLT-07), pages 175–187, Bergen,Norway.Tinsley, J., Ma, Y., Ozdowska S., and Way, A. (2008).MaTrEx: the DCU MT System for WMT 2008.In Proceedings of the Third Workshop on Statisti-cal Machine Translation, ACL 2008, pages 171–174,Columbus, OH.

A modied joint source-channel model for transliteration.

A Systematic Comparison of Various Statistical Alignment Models.

An Empirical Study of Smoothing Techniques for Language Modeling.

Boostrapping Word Alignment via Word Packing.

Exploiting Source Similarity for SMT using Context-Informed Features.

ExploitingParallelTreebankstoImprovePhrase-BasedStatistical Machine Translation.

Factored Translation Models.

I.(2006). Clause RestructuringforStatisticalMachineTranslation.

MaTrEx: the DCU machine translation system for IWSLT

MaTrEx: the DCU MT System for WMT

Minimum error rate training in statistical machine translation.

Moses: Open Source Toolkit for Statistical Machine Translation.

Robust Large-Scale EBMT with Marker-Based Segmentation.

SRILM - An Extensible Language Modeling Toolkit.

Statistical Phrase-Based Translation.

Supertagged Phrase-based Statistical Machine Translation.

The Necessity of Syntax Markers. Two experiments with articial languages.

MaTrEx: the DCU machine translation system for ICON 2008

In this paper, we give a description of the machine translation system developed at DCU that was used for our participation in the NLP Tools Contest of the International\ud
Conference on Natural Language Processing (ICON 2008). This was our first ever attempt at working on any Indian language. In this participation, we focus on various techniques for word and phrase alignment to improve system quality. For the English-Hindi translation task we exploit\ud
source-language reordering. We also carried out experiments combining both in-domain and out-of-domain data to improve\ud
the system performance and, as a post-processing step we transliterate out-of-vocabulary items

Name not available

In this paper, we give a description of the machine translation system developed at DCU that was used for our participation in the NLP Tools Contest of the International
Conference on Natural Language Processing (ICON 2008). This was our first ever attempt at working on any Indian language. In this participation, we focus on various techniques for word and phrase alignment to improve system quality. For the English-Hindi translation task we exploit
source-language reordering. We also carried out experiments combining both in-domain and out-of-domain data to improve
the system performance and, as a post-processing step we transliterate out-of-vocabulary items

Irish Universities

http://doras.dcu.ie/15200/1/SrivastavaEtAl_icon_08.pdf

MaTrEx: the DCU machine translation system for ICON 2008

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Name not available

Irish Universities