Junczys-Dowmunt, Marcin

Pinero, Blanca

Pouliquen, Bruno

Ziemski, Michal

This paper presents a machine translation tool -- based on Moses -- developed for the International Maritime Organization (IMO) for the automatic translation of documents from Spanish, French, Russian and Arabic to/from English. The main challenge lies in the insufficient size of in-house corpora (especially for Russian and Arabic). The United Nations (UN) granted IMO the right to use UN resources and we describe experiments and results we obtained with different translation model combination techniques. While BLEU results remain inconclusive for combinations, we also analyze user preferences for certain models (when choosing between IMO only or combined with UN). The combined models are perceived by translators as being much better for general texts while IMO only models seem better for technical texts

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      SMT at the International Maritime Organization: Experiences withCombining In-house Corpora with Out-of-domain CorporaCitation for published version:Pouliquen, B, Junczys-Dowmunt, M, Pinero, B & Ziemski, M 2015, SMT at the International MaritimeOrganization: Experiences with Combining In-house Corpora with Out-of-domain Corpora. in EuropeanAssociation for Machine Translation 2015. pp. 202-205.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:European Association for Machine Translation 2015General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019SMT at the International Maritime Organization: Experiences withCombining In-house Corpora with Out-of-domain CorporaBruno PouliquenWorld Intellectual Property Organization34, chemin des ColombettesCH-1211 Geneva, Switzerlandbruno.pouliquen@wipo.intMarcin Junczys-DowmuntAdam Mickiewicz Universityul. Umultowska 8761-614 Poznan´, Polandjunczys@amu.edu.plBlanca PineroInternational Maritime Organization4 Albert EmbankmentLondon SE17SR, United Kingdombpinero@imo.orgMichał ZiemskiUnited Nations8-14, Avenue de la PaixCH-1211 Geneva, Switzerlandmziemski@unog.chAbstractThis paper presents a machine transla-tion tool – based on Moses – developedfor the International Maritime Organiza-tion (IMO) for the automatic translationof documents from Spanish, French, Rus-sian and Arabic to/from English. The mainchallenge lies in the insufﬁcient size of in-house corpora (especially for Russian andArabic). The United Nations (UN) grantedIMO the right to use UN resources andwe describe experiments and results weobtained with different translation modelcombination techniques. While BLEUresults remain inconclusive for combina-tions, we also analyze user preferences forcertain models (when choosing betweeenIMO only or combined with UN). Thecombined models are perceived by transla-tors as being much better for general textswhile IMO only models seem better fortechnical texts.1 IntroductionThis paper describes the installation and trainingof TAPTA, an MT tool, for the automatic trans-lation of IMO documents. TAPTA has been pre-viously installed at other international organiza-tions (Pouliquen et. al 2013). IMO is a special-ized agency of the United Nations system deal-ing with safe and secure seas and the protectionc� 2015 The authors. This article is licensed under a CreativeCommons 3.0 licence, no derivative works, attribution, CC-BY-ND.of the marine environment. It has three workinglanguages (English, French and Spanish) with par-allel corpora of ca. 60 million words each. A muchsmaller number of documents (conventions and re-ports totaling ca. 6 million words per language)are translated into the other ofﬁcial languages ofthe United Nations (Arabic, Chinese1, Russian).IMO felt that the introduction of MT would helptranslators in their daily work, given similar ex-periences in other UN agencies and repetitive na-ture of their documentation due to periodic report-ing. While building the SMT corpora, the spe-ciﬁc terminology used in the maritime domain andthe “house style” were also important considera-tions. The large imbalance in the number of paral-lel documents between language poses a problemwhich we try to solve by integrating larger par-allel corpora which “complete” the IMO models(especially for Russian and Arabic)2. The corporaprovided by the United Nations Secretariat werethought to be ideal for this purpose as the languagepairs are the same and working practices in bothtranslation services are very similar. Authorizationwas granted to merge the corpora.2 Data and preprocessingThe International Maritime Organization has 6 of-ﬁcial languages (Arabic, English, Spanish, French,1Work on Chinese data has been postponed and is not de-scribed in this paper.2Documents were provided by the Documentation Division(New York) of the Department for General Assembly andConference Management, the main entity of the United Na-tions Secretariat charged with the production of parliamentarydocumentation.202Russian and Chinese), which means that, if suchan organization wanted a translation tool for alllanguage pair combinations, it would require 42translation engines. A rule-based translation sys-tem would be extremely costly to build and main-tain. A data-driven approach is usually more suit-able when a big parallel corpus exists, therefore wefocused on SMT.Moses (Koehn et al. 2007) has been trainedwith a parallel corpora extracted consisting of IMOdocuments translated between January 2000 andOctober 2014 (ca. 20,000 documents for English,French and Spanish, about 400 documents for Rus-sian/Arabic, see Table 1). The provided corporahave been extracted from original Word or PDFdocuments, identical IDs between languages allowto align documents for each language pair. Weuse an in-house (WIPO) sentence aligner. Thetool processes each parallel text document and pro-duces a set of aligned sentences after applying thefollowing steps:• Sentence splitting• Tokenization• Sentence alignment with our sentencesaligned (based on Champollion (Ma 2006))— produces an “aligned-segment-matching-score”• ﬁltering out whole documents with anaverage-segment-matching-score below agiven threshold• ﬁltering out sets of consecutive segments hav-ing a low scores• ﬁltering out sets of consecutive segments thatare sorted by alphabetical order3• ﬁltering out sentences having only one wordor more than 80 words, or a source/targetword ratio more than 93 SMT system3.1 Baseline systemThe baseline SMT system consists of an extendedMoses (Koehn et al. 2007) conﬁguration. Dur-rani et al. (2013) report on improvements for var-ious language pairs when an Operation Sequence3In both IMO and UN, it is very common to sort enumerationsof countries, persons, organizations, etc. by alphabetical or-der, which will of course often be different between languagesand results in very noisy sentence alignment.Lang. IMO corpus UNpair Docs Words Segments Wordsen-fr 17132 53.8 M 2.60 M 316 Men-es 16213 54.0 M 2.50 M 295 Men-ru 318 5.6 M 0.30 M 296 Men-ar 296 4.1 M 0.23 M 304 Men-zh [not available yet] 280 MTable 1: Size of the parallel corpora used for train-ing. The fourth and ﬁfth columns show the trainingsize (in millions of English words) for IMO andUN corpus.Model (OSM) is added to the phrase-based de-coder. Class-based language models seem to bea good compromise between increased n-gramlength and total model size. We use automati-cally calculated word cluster ids as classes. Wehad good experience with word2vec (Mikolov etal. 2012) in the context of larger SMT modelsand use this tool to compute 200 word classesfrom the target language data. The target lan-guage corpora are mapped to sequences of classesand 9-gram language model are estimated. Theﬁnal phrase-tables of the larger models (English-French, English-Spanish) have been signiﬁcancepruned (Johnson et al. 2007) for size reduction. Inour experiments signiﬁcance pruning results in noquality loss while reducing translation model sizeby a factor of 5. The standard 5-gram languagemodels and the 9-gram word-class models areestimated with Modiﬁed Kneser-Ney smoothing(Chen and Goodman 1996, Heaﬁeld et al. 2013).To reduce size requirements, we use heavily quan-tized binary models with no noticeable quality re-duction. Pruning is applied to all singleton n-grams with n equal to or greater than 3.3.2 Attempts at domain adaptationWe explore two model combination methods forboth, translation models and language models: lin-ear and log-linear interpolation. Log-linear modelinterpolation is natively supported in Moses viaits feature function framework. Translation mod-els and language models can be log-linearly inter-polated just by adding them to the Moses conﬁg-uration ﬁles. Parameter tuning then chooses theappropriate interpolation weights which are actu-ally feature weights. Linear interpolation, thougha standard method for language models, is moreinvolved. In the case of language models, we203compute a new static linearly interpolated lan-guage model from IMO and UN data target lan-guage data. Interpolation weights are optimizedon the dev set. In the case of translation modelswe use a new feature function available in Mosesthat allows for setting up virtual phrase tables thatare in fact linearly interpolated translation models(Sennrich 2012). We use the same interpolationweights as previously determined for linear lan-guage model interpolation. The two interpolatedtranslation models are the original IMO and UNtranslation models as used in stand-alone transla-tors. Results are mixed, we report the best re-sults for our experiments (see Table 2, Section 5.1).Log-linear interpolation is downright harmful (andtherefore omitted), for the larger language pairs(en-fr and en-es) any of the interpolation methodsseem to be unhelpful, improvements for en-es arewithin the range of optimizer instability. For thesmaller models (en-ar, en-ru) we observe quite sig-niﬁcant improvements that stem mainly from lin-ear translation model interpolation.4 Translating4.1 Server conﬁgurationThe server has been installed on a virtual machinerunning Ubuntu, the same machine is being usedfor training and decoding. Server speciﬁcationsare: 12 cores, 64 GB RAM and 1 TB of disk space.The server runs several Moses decoders (one de-coder is a Moses single-thread executable). Eachdecoder is encapsulated in a Java RMI interfaceserver which allows to operate several concurrentdecoders. Each sentence submitted is queued andsent to the next free decoder. Since both, thephrase table as well as all the language models,rely on memory mapping and shared memory, hav-ing several independent workers instead of a multi-threaded architecture does not represent much of amemory problem. Common data is shared auto-matically between processes. Thanks to our expe-rience in installing the tool, we were able to in-stall and conﬁgure the server in 2 days (not includ-ing research model parameters and speciﬁc exper-iments with model combinations). Training oneIMO model takes ca. 20 hours.4.2 User interface4.2.1 Web interface: gist translationA web interface allows users to submit shorttexts and access the corresponding automaticF3Figure 1: Translating with the “auto hotkey”Figure 2: Concordancer for term “coral cover”,the graph shows the term usage over years, nextthe most used translations are display, then the fullparallel segments with links to original documents.translation (with highlighting of parallel segmentsor words).4.2.2 “Auto hotkey” accessTranslators in IMO use speciﬁc software (Mul-tiTrans Prism) and do not wish to copy-pastetexts in order to use the tool. So we decidedto use the “auto hotkey” open source software(http://www.autohotkey.com), which al-lows users to call the tool with a keystroke, transla-tions are then copied to the clipboard and users canpaste it into the translation in-progress (see Fig-ure 1 for an example screen shot).4.2.3 ConcordancerUsers can access the concordancer using a Webinterface or through a different “hotkey”. The con-cordancer is based on a Lucene index containingthe word aligned corpus. This concordancer dis-plays segments containing the search term and thecorresponding aligned words. A ﬁrst window dis-204IMO only Combined Googleen-fr 54.24 54.03 32.58en-es 52.68 52.99 35.18en-ru 58.77 60.20 20.56en-ar 41.20 44.18 16.58en-zh [not available]Table 2: BLEU scores for each language pair, com-pared with a combined model and with Googletranslate.plays the usage of the term by year, a second win-dow displays the aligned words by order of fre-quency, the user can immediately see which trans-lation is the most common (see Figure 2 for an ex-ample).5 Results/Evaluation5.1 Automatic evaluationBLEU scores (Papineni et al. 2002) were used tocompare human translations with automatic trans-lations (one reference) on a set slightly more than2000 sentences which have been set apart beforemodel training.5.2 Human perceptionIt is always difﬁcult to measure user acceptance,especially at this early stage. However we can nowobserve that, on average, more than 1500 wordsare translated every day using our tool. Some users“jump” between various models (eg. users pre-fer IMO-only models for English-to-Spanish, butnevertheless use the combined model in more than10% of the cases). Even though the automatic eval-uation scores do not show signiﬁcant improvementwith combined models, translators judged com-bined models to be better for general texts whileIMO-only models work better for more technicaltexts. Additional functionality such as the con-cordancer are readily embraced and found usefulalongside the pure translation function.6 Conclusion and future workDuring our experiments, we had to face both, ascarcity problem (small IMO corpora for some lan-guages) and a scalability problem (large UN cor-pora). However, our experience shows that opensource solutions can sometimes provide better re-sults than generic commercial products. Moreover,sharing the tool between these organizations fa-cilitates sharing of corpora and the spread of MTin international organizations. User comments in-clude that the Web interface is intuitive and the“auto-hotkey” is an easy and fast way of access-ing translations; integration like this requires verylittle training and this training can be done inter-nally. Future work includes: better integration intothe users’ environment and a biannual retraining ofall the models. We believe the model combinationtechnique can still be improved. An area to explorewould be to “automatically” choose the best modelto translate a given document/sentence.AcknowledgementsThe authors wish to thank Mrs Olga O’Neil, Di-rector, Conference Division, IMO, for making thiscollaboration possible, and Ms Cecilia Elizalde,UNHQ, for her invaluable assistance in obtainingthe UN corpora.ReferencesPouliquen B., C. Elizalde, M. Junczys-Dowmunt, C.Mazenc, and J. Garca-Verdugo. 2013. Large-scaleMultiple Language Translation Accelerator at theUnited Nations, Proc. of MT Summit 2013.Chen S. F. and J. Goodman. 1996. An Empirical Studyof Smoothing Techniques for Language Modeling,in Proc. of ACL 1996.Durrani, N., A. Fraser, H. Schmid, H. Hoang, and P.Koehn. 2013. Can Markov Models over MinimalTranslation Units Help Phrase-based SMT? in ACL.Heaﬁeld, K, I. Pouzyrevsky, J. H. Clark, and P.Koehn. 2013. Scalable Modiﬁed Kneser-Ney Lan-guage Model Estimation, in Proc. of ACL 2013.Johnson, H., J. D. Martin, G. F. Foster, and R. Kuhn.2007. Improving Translation Quality by DiscardingMost of the Phrasetable, Proc. of EMNLP 2007.Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris C.Burch, Marcello Federico, Nicola Bertoldi, BrookeCowan, Wade Shen, Christine Moran, Richard Zens,Chris Dyer, Ondej Bojar, Alexandra Constantin,Evan Herbst. 2007. Moses: Open Source Toolkit forStatistical Machine Translation. Proc. of ACL 2007.Koehn, Phillip. 2010. Statistical Machine Translation.Textbook, Cambridge University Press.Ma, Xiaoyi. 2006. Champollion: A Robust ParallelText Sentence Aligner. Proc. of LREC-2006.Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013.Efﬁcient Estimation ofWord Representations in Vec-tor Space, CoRR, vol. abs/1301.3781.Papineni, K., S. Roukos, T. Ward, and WJ Zhu. 2002.BLEU: a Method for Automatic Evaluation of Ma-chine Translation, Proc. of ACL 2002.Sennrich R. 2012, Perplexity Minimization for Trans-lation Model Domain Adaptation in Statistical Ma-chine Translation. Proc. of EACL 2012.205

SMT at the International Maritime Organization: Experiences with Combining In-house Corpora with Out-of-domain Corpora

https://www.pure.ed.ac.uk/ws/files/25567605/W15_4927.pdf

SMT at the International Maritime Organization: Experiences with Combining In-house Corpora with Out-of-domain Corpora

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer