In this paper we describe recent improvements to components and methods used in our statistical machine translation system for Chinese-English used in the January 2008 GALE evaluation. Main improvements are results of consistent data processing, larger statistical models and a POS-based word reordering approach

Bach, Nguyen

Gao, Qin

Hewavitharana, Sanjika

Hildebrand, Almut Silja

Notari, Timothy

Rottmann, Kay

Vogel, Stephan

English

KITopen

Proceedings of ACL-08: HLT, Short Papers (Companion Volume), pages 77–80,Columbus, Ohio, USA, June 2008. c©2008 Association for Computational LinguisticsRecent Improvements in theCMU Large Scale Chinese-English SMT SystemAlmut Silja Hildebrand, Kay Rottmann, Mohamed Noamany, Qin Gao,Sanjika Hewavitharana, Nguyen Bach and Stephan VogelLanguage Technologies InstituteCarnegie Mellon UniversityPittsburgh, PA 15213, USAsilja, kayrm, mfn, qing, sanjika, nbach, vogel+@cs.cmu.eduAbstractIn this paper we describe recent improvementsto components and methods used in our statis-tical machine translation system for Chinese-English used in the January 2008 GALE eval-uation. Main improvements are results ofconsistent data processing, larger statisticalmodels and a POS-based word reordering ap-proach.1 IntroductionBuilding a full scale Statistical Machine Transla-tion (SMT) system involves many preparation andtraining steps and it consists of several components,each of which contribute to the overall system per-formance. Between 2007 and 2008 our system im-proved by 5 points in BLEU from 26.60 to 31.85for the unseen MT06 test set, which can be mainlyattributed to two major points.The fast growth of computing resources overthe years make it possible to use larger and largeramounts of data in training. In Section 3 we showhow parallelizing model training can reduce trainingtime by an order of magnitude and how using largertraining data as well as more extensive models im-prove translation quality.Word reordering is still a difficult problem inSMT. In Section 4 we apply a Part Of Speech (POS)based syntactic reordering model successfully to ourlarge Chinese system.1.1 DecoderOur translation system is based on the CMUSMT decoder as described in (Hewavitharana etal., 2005). Our decoder is a phrase-based beamsearch decoder, which combines multiple modelse.g. phrase tables, several language models, a dis-tortion model ect. in a log-linear fashion. In orderto find an optimal set of weights, we use MER train-ing as described in (Venugopal et al., 2005), whichuses rescoring of the top n hypotheses to maximizean evaluation metric like BLEU or TER.1.2 EvaluationIn this paper we report results using the BLEU met-ric (Papineni et al., 2002), however as the evaluationcriterion in GALE is HTER (Snover et al., 2006), wealso report in TER (Snover et al., 2005).We used the test sets from the NIST MT evalua-tions from the years 2003 and 2006 as developmentand unseen test data.1.3 Training DataIn translation model training we used the Chinese-English bilingual corpora relevant to GALE avail-able through the LDC1. After sentence alignmentthese sources add up to 10.7 million sentences with301 million running words on the English side. Ourpreprocessing steps include tokenization on the En-glish side and for Chinese: automatic word segmen-tation using the revised version of the Stanford Chi-nese Word Segmenter2 (Tseng et al., 2005) from2007, replacement of traditional by simplified Chi-nese characters and 2-byte to 1-byte ASCII charac-ter normalization. After data cleaning steps like e.g.removal of sentence pairs with very unbalanced sen-1http://projects.ldc.upenn.edu/gale/data/catalog.html2http://nlp.stanford.edu/software/segmenter.shtml77tence length etc., we used the remaining 10 millionsentences with 260 million words (English) in trans-lation model training (260M system).2 Number TaggingSystematic tagging and pre-translation of numbershad shown significant improvements for our Arabic-English system, so we investigated this for Chinese-English. The baseline for these experiments was asmaller system with 67 million words (67M) bilin-gual training data (English) and a 500 million word3-gram LM with a BLEU score of 27.61 on MT06.First we pre-translated all numbers in the testdataonly, thus forcing the decoder to treat the numbers asunknown words. Probably because the system couldnot match longer phrases across the pre-translatednumbers, the overall translation quality degraded by1.6 BLEU to 26.05 (see Table 1).We then tagged all numbers in the training corpus,replaced them with a placeholder tag and re-trainedthe translation model. This reduced the vocabu-lary and enabled the decoder to generalize longerphrases across numbers. This strategy did not lead tothe expected result, the BLEU score for MT06 onlyreached 25.97 BLEU.System MT03 MT0667M baseline 31.45/60.93 27.61/62.18test data tagged – 26.06/63.36training data tagged 29.07/62.52 25.97/63.39Table 1: Number tagging experiments, BLEU/TERAnalysing this in more detail, we found, the rea-son for this degradation in translation quality couldbe the unbalanced occurrence of number tags in thetraining data. From the bilingual sentence pairs,which contain number tags, 66.52% do not containthe same number of tags on the Chinese and the En-glish side. As a consequence 52% of the phrase pairsin the phrase table, which contain number tags hadto be removed, because the tags were unbalanced.This hurts system performance considerably.3 Scaling up to Large Data3.1 Language ModelDue to the availability of more computing resources,we were able to extend the language model historyfrom 4- to 5-gram, which improved translation qual-ity from 29.49 BLEU to 30.22 BLEU for our largescale 260M system (see Table 2). This shows, thatlonger LM histories help if we are able to use enoughdata in model training.System MT03 MT06260M, 4gram 31.20/61.00 29.49/61.00260M, 5gram 32.20/60.59 30.22/60.81Table 2: 4- and 5-gram LM,260M system, BLEU/TERThe language model was trained on the sourcesfrom the English Gigaword Corpus V3, which con-tains several newspapers for the years between 1994to 2006. We also included the English side of thebilingual training data, resulting in a total of 2.7 bil-lion running words after tokenization.We trained separate open vocabulary languagemodels for each source and interpolated them usingthe SRI Language Modeling Toolkit (Stolcke, 2002).Table 3 shows the interpolation weights for the dif-ferent sources. Apart from the English part of thebilingual data, the newswire data from the ChineseXinhua News Agency and the Agence France Presshave the largest weights. This reflects the makeup ofthe test data, which comes in large parts from thesesources. Other sources, as for example the UN par-lamentary speeches or the New York Times, differsignificantly in style and vocabulary from the testdata and therefore get small weights.xin 0.30 cna 0.06 nyt 0.03bil 0.26 un 0.07 ltw 0.01afp 0.21 apw 0.05Table 3: LM interpolation weights per source3.2 Speeding up Model TrainingTo accelerate the training of word alignmentmodels we implemented a distributed version ofGIZA++ (Och and Ney, 2003), based on the latestversion of GIZA++ and a parallel version developedat Peking University (Lin et al., 2006). We divide thebilingual training data in equal parts and distribute itover several processing nodes, which perform align-ment independently. In each iteration the nodes readthe model from the previous step and output all nec-essary counts from the data for the models, e.g. the78co-occurrence or fertility model. A master processcollects the counts from the nodes, normalizes themand outputs the intermediate model for each itera-tion.This distributed GIZA++ version finished trainingthe word alignment up to IBM Model 4 for both lan-guage directions on the full bilingual corpus (260million words, English) in 39 hours. On averageabout 11 CPUs were running concurrently. In com-parison the standard GIZA++ implementation fin-ished the same training in 169 hours running on 2CPUs, one for each language direction.We used the Pharaoh/Moses package (Koehn etal., 2007) to extract and score phrase pairs using thegrow-diag-final extraction method.3.3 Translation ModelWe trained two systems, one on the full data and onewithout the out-of-domain corpora: UN parlament,HK hansard and HK law parallel texts. These parla-mentary sessions and law texts are very different ingenre and style from the MT test data, which con-sists mainly of newspaper texts and in recent yearsalso of weblogs, broadcast news and broadcast con-versation. The in-domain training data had 3.8 mil-lion sentences and 67 million words (English). The67 million word system reached a BLEU score of29.65 on the unseeen MT06 testset. Even though thefull 260M system was trained on almost four timesas many running words, the baseline score for MT06only increased by 0.6 to 30.22 BLEU (see Table 4).System MT03 MT0667M in-domain 32.42/60.26 29.65/61.22260M full 32.20/60.59 30.22/60.81Table 4: In-domain only or all training data, BLEU/TERThe 67M system could not translate 752 Chinesewords out of 38937, the number of unknown wordsdecreased to 564 for the 260M system. To increasethe unigram coverage of the phrase table, we addedthe lexicon entries that were not in the phrase tableas one-word translations. This lowered the numberof unknown words further to 410, but did not effectthe translation score.4 POS-based ReorderingAs Chinese and English have very different wordorder, reordering over a rather limited distance dur-ing decoding is not sufficient. Also using a simpledistance based distortion probability leaves it essen-tially to the language model to select among dif-ferent reorderings. An alternative is to apply auto-matically learned reordering rules to the test sen-tences before decoding (Crego and Marino, 2006).We create a word lattice, which encodes many re-orderings and allows long distance reordering. Thiskeeps the translation process in the decoder mono-tone and makes it significantly faster compared toallowing long distance reordering at decoding time.4.1 Learning Reordering RulesWe tag both language sides of the bilingual corpuswith POS information using the Stanford Parser3and extract POS based reordering patterns fromword alignment information. We use the context inwhich a reordering pattern is seen in the training dataas an additional feature. Context refers to the wordsor tags to the left or to the right of the sequence forwhich a reordering pattern is extracted.Relative frequencies are computed for every rulethat has been seen more than n times in the trainingcorpus (we observed good results for n > 5).For the Chinese system we used only 350k bilin-gual sentence pairs to extract rules with length ofup to 15. We did not reorder the training corpusto retrain the translation model on modified Chineseword order.4.2 Applying Reordering RulesTo avoid hard decisions, we build a lattice struc-ture for each source sentence as input for our de-coder, which contains reordering alternatives consis-tent with the previously extracted rules.Longer reordering patterns are applied first.Thereby shorter patterns can match along new paths,creating short distance reordering on top of long dis-tance reordering. Every outgoing edge of a node isscored with the relative frequency of the pattern usedon the following sub path (For details see (Rottmannand Vogel, 2007)). These model scores give this re-3http://nlp.stanford.edu/software/lex-parser.shtml79ordering approach an advantage over a simple jumpmodel with a sliding window.System MT03 MT06260M, standard 32.20/60.59 30.22/60.81260M, lattice 33.53/59.74 31.74/59.59Table 5: Reordering lattice decoding in BLEU/TERThe system with reordering lattice input outper-forms the system with a reordering window of 4words by 1.5 BLEU (see Table 5).5 SummaryThe recent improvements to our Chinese-EnglishSMT system (see Fig. 1) can be mainly attributed toa POS based word reordering method and the possi-bility to work with larger statistical models.We used the lattice translation functionality of ourdecoder to translate reordering lattices. They arebuilt using reordering rules extracted from taggedand aligned parallel data. There is further potentialfor improvement in this approach, as we did not yetreorder the training corpus and retrain the translationmodel on modified Chinese word order.Improvements in BLEU242526272829303132332007 67M+3gr 260M+3gr 260M+4gr 260M+5gr 260M+ROFigure 1: Improvements for MT06 in BLEUWe modified GIZA++ to run in parallel, which en-abled us to include especially longer sentences intotranslation model training. We also extended our de-coder to use 5-gram language models and were ableto train an interpolated LM from all sources of theEnglish GigaWord Corpus.AcknowledgmentsThis work was partly funded by DARPA underthe project GALE (Grant number #HR0011-06-2-0001).ReferencesJosep M. Crego and Jose B. Marino. 2006. ReorderingExperiments for N-Gram-Based SMT. Spoken Lan-guage Technology Workshop, Palm Beach, Aruba.Sanjika Hewavitharana, Bing Zhao, Almut Silja Hilde-brand, Matthias Eck, Chiori Hori, Stephan Vogel andAlex Waibel. 2005. The CMU Statistical MachineTranslation System for IWSLT 2005. IWSLT 2005,Pittsburgh, PA.Philipp Koehn, Hieu Hoang, Alexandra Birch, ChrisCallison-Burch, Marcello Federico, Nicola Bertoldi,Brooke Cowan, Wade Shen, Christine Moran, RichardZens, Chris Dyer, Ondrej Bojar, Alexandra Con-stantin, Evan Herbst. 2007. Moses: Open SourceToolkit for Statistical Machine Translation. ACL 2007,Demonstration Session, Prague, Czech Republic.Xiaojun Lin, Xinhao Wang, and Xihong Wu. 2006.NLMP System Description for the 2006 NIST MTEvaluation. NIST 2006 MT Evaluation.Franz Josef Och and Hermann Ney. 2003. A SystematicComparison of Various Statistical Alignment Models.Computational Linguistics, 29(1):19–51.Kishore Papineni, Salim Poukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Eval-uation of Machine Translation. ACL 2002, Philadel-phia, USA.Kay Rottmann and Stephan Vogel. 2007. Word Reorder-ing in Statistical Machine Translation with a POS-based Distortion Model. TMI-2007: 11th Interna-tional Conference on Theoretical and MethodologicalIssues in MT, Skvde, Sweden.Mathew Snover, Bonnie Dorr, Richard Schwartz, JohnMakhoul, Linnea Micciula and Ralph Weischedel.2005. A Study of Translation Error Rate with Tar-geted Human Annotation. LAMP-TR-126, Universityof Maryland, College Park and BBN Technologies.Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin-nea Micciulla and John Makhoul. 2006. A Studyof Translation Edit Rate with Targeted Human Anno-tation. 7th Conference of AMTA, Cambridge, Mas-sachusetts, USA.Andreas Stolcke. 2002. SRILM - An Extensible Lan-guage Modeling Toolkit. ICSLP, Denver, Colorado.Huihsin Tseng, Pichuan Chang, Galen Andrew, DanielJurafsky and Christopher Manning. 2005. A Con-ditional Random Field Word Segmenter. FourthSIGHAN Workshop on Chinese Language Processing.Ashish Venugopal, Andreas Zollman and Alex Waibel.2005. Training and Evaluation Error MinimizationRules for Statistical Machine Translation. ACL 2005,WPT-05, Ann Arbor, MI80

Recent improvements in the CMU large-scale Chinese-English SMT system

https://publikationen.bibliothek.kit.edu/1000166382/152264210

Recent improvements in the CMU large-scale Chinese-English SMT system

Abstract

Similar works

Full text

Available Versions

KITopen