Bhattacharyya, Pushpak

Ekbal, Asif

Gupta, Kamal Kumar

Sen, Sukanta

English

This paper describes the systems submitted by the IITP-MT team to WAT 2018 multilingual Indic languages shared task. We submit two multilingual neural machine translation (NMT) systems (Indic-to-English and English-to-Indic) based on Transformer architecture and our approaches are similar to many-to-one and one-to-many approaches of Johnson et al. (2017). We also train separate bilingual models as baselines for all translation directions involving English. We evaluate the models using BLEU score and find that a single multilingual NMT model performs better (up to 14.81 BLEU) than separate bilingual models when the target is English. However, when English is the source language, multi-lingual NMT model improves only for low-resource language pairs (up to 11.60 BLEU) and degrades for relatively high-resource language pairs over separate bilingual models

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      IITP-MT at WAT2018: Transformer-based Multilingual Indic-English Neural Machine Translation SystemCitation for published version:Sen, S, Gupta, KK, Ekbal, A & Bhattacharyya, P 2018, IITP-MT at WAT2018: Transformer-basedMultilingual Indic-English Neural Machine Translation System. in Proceedings of the 32nd Pacific AsiaConference on Language, Information and Computation: 5th Workshop on Asian Translation. Associationfor Computational Linguistics, Hong Kong, pp. 1003-1007, The 5th Workshop on Asian Translation @PACLIC 32, Hong Kong, 3/12/18. <https://www.aclweb.org/anthology/Y18-3012>Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation: 5th Workshop onAsian TranslationGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 30. Jun. 2022IITP-MT at WAT2018: Transformer-based Multilingual Indic-EnglishNeural Machine Translation SystemSukanta Sen, Kamal Kumar Gupta, Asif Ekbal, Pushpak BhattacharyyaDepartment of Computer Science and EngineeringIndian Institute of Technology Patna{sukanta.pcs15,kamal.pcs17,asif,pb}@iitp.ac.inAbstractThis paper describes the systems submittedby the IITP-MT team to WAT 2018 multilin-gual Indic languages shared task. We sub-mit two multilingual neural machine trans-lation (NMT) systems (Indic-to-English andEnglish-to-Indic) based on Transformer ar-chitecture and our approaches are similar tomany-to-one and one-to-many approaches ofJohnson et al. (2017). We also train separatebilingual models as baselines for all transla-tion directions involving English. We evaluatethe models using BLEU score and find that asingle multilingual NMT model performs bet-ter (up to 14.81 BLEU) than separate bilingualmodels when the target is English. However,when English is the source language, multi-lingual NMT model improves only for low-resource language pairs (up to 11.60 BLEU)and degrades for relatively high-resource lan-guage pairs over separate bilingual models.1 IntroductionIn this paper, we describe our submission to mul-tilingual Indic languages shared task at 5th Work-shop on Asian Translation (WAT 2018) (Nakazawaet al., 2018). This task covers 7 Indic languages(Bengali, Hindi, Malayalam, Tamil, Telugu, Sin-halese and Urdu) and English. The objective ofthis shared task is to build translation models forXX-EN language pairs. By XX, we denote the setof 7 Indic languages. In this task, we submit two(single models for Indic-to-English and English-to-Indic) multilingual neural machine translation sys-tems to translate between Indic languages and En-glish. Unlike the European languages, most of theIndian languages do not have enough-sized paral-lel English translations. The parallel corpora usedin this shared task have 22k to 521k parallel sen-tences (see Table 1), which is insufficient for NMTtraining. NMT is a data hungry approach and itis not possible to have sufficient amount of paral-lel training data for all language pairs. So build-ing multilingual translation model by means of shar-ing parameters with high-resource languages is acommon practice to improve the performance oflow-resource language pairs. Sharing of parametersbetween low-resource and high-resource languagepairs helps low-resource pairs to learn better modelcompared to model trained separately. However, ithas been seen that training multiple languages to-gether sometimes degrades the performance of somelanguage pairs compared to a separate single bilin-gual model as languages may have different linguis-tic properties.Recent success of end-to-end bilingual NMT sys-tems (Kalchbrenner and Blunsom, 2013; Cho et al.,2014; Sutskever et al., 2014; Bahdanau et al., 2015)quickly gave the rise of multilingual NMT in vari-ous ways (Dong et al., 2015; Firat et al., 2016; John-son et al., 2017). Most of the existing multilingualNMT involve non-Indic languages and are based onattentional encoder-decoder approach. We use theTransformer architecture (Vaswani et al., 2017) withsubword (Sennrich et al., 2016) as basic transla-tion unit. We develop two multilingual translationmodels: one is for XX→EN (7 Indic languages toEnglish) and another is for EN→XX (English to 7Indic languages). We also train separate bilingualPACLIC 32 - WAT 20181003 32nd Pacific Asia Conference on Language, Information and Computation The 5th Workshop on Asian Translation Hong Kong, 1-3 December 2018 Copyright 2018 by the authorsmodel as a baseline for each translation direction in-volving English. We evaluate the multilingual mod-els against the bilingual models using BLEU (Pa-pineni et al., 2002) metric. We found that multi-lingual NMT is better than bilingual models for allXX→EN directions, however for EN→XX direc-tions, multilingual NMT performs better than bilin-gual NMT for low-resource language pairs only.In the next section, we briefly mention some no-table multilingual NMT works. We describe oursubmitted systems in section 3 which includes de-scription on datasets, preprocessing, experimentalsetup. Results are described in section 4. Finally,the work is concluded in section 5.2 Related WorksDong et al. (2015) implemented a system with one-to-many mapping of languages. They translated asource language to multiple target languages whereeach target language decoder deals with its own at-tention network. Firat et al. (2016) used a single at-tentional network that was shared among all source-target language pairs. They used separate encoderdecoder for each source and target language. Thus,the number of parameters increases as the numberof language increases. Johnson et al. (2017) cameup with a simple but effective approach for mul-tilingual translation. They mixed all parallel dataand trained a standard attentional encoder-decoderNMT model without any change. They used an ad-ditional token before each source sentence to specifyits target language. We apply this simple approachof combining training data and then we train trans-former based NMT models for building multilingualtranslation systems (many-to-one and one-to-many)for Indic languages.3 System DescriptionIn this section, we describe datasets, preprocessingof data and experimental setup of our systems.3.1 DatasetsWe use the Indic Languages Multilingual ParallelCorpus1 consisting of the following languages: Ben-gali, Hindi, Malayalam, Tamil, Telugu, Sinhalese,1http://lotus.kuee.kyoto-u.ac.jp/WAT/indic-multilingual/indic languages corpus.tar.gzUrdu and English. It contains 7 parallel corpora for7 Indic languages (translated into English), and 8monolingual corpora. These corpora have been col-lected from OPUS2 and belongs to the spoken lan-guage (OpenSubtitles) domain. For experiments, weuse parallel corpora only. Training data size is pre-sented in Table 1. For each language pair, devel-opment set and test set have 500 and 1,000 parallelsentences, respectively. Before feeding the data fortraining, we tokenize, truecase, subword the origi-nal corpora as preprocessing. We tokeninze Englishdata using Moses tokenizer3 and the Indic NLP li-brary4 tool is used for tokenizing Indic languagedata. Tokenized English sentences are truecased us-ing Moses truecaser script. There is no need to true-case Indic languages as they are case-insensitive.Language Pair #SentencesBengali (BN) - English 337,428Hindi (HI) - English 84,557Malayalam (ML) - English 359,423Tamil (TA) - English 26,217Telugu (TE) - English 22,165Urdu (UR) - English 26,619Sinhalese (SI) - English 521,726Table 1: Training data size for each language pair.3.2 Subword UnitNMT works with fixed vocabulary size. To deal withlarge vocabulary of Indic languages, we subwordeach bilingual corpora independently. Sennrich et al.(2016) introduced Byte-pair-encoding (BPE) basedsubword unit for dealing with rare words problem inNMT. It helps to decrease the vocabulary size andto deal with unseen tokens at training and test time.With variable size of training data (see Table 1) andmorphological variations among languages, vocabu-lary size for each language is also different. Originalvocabulary size, number of BPE merge, and vocab-ulary size after applying BPE are shown in Table 2.2http://opus.nlpl.eu3https://github.com/moses-smt/mosesdecoder/blob/RELEASE-3.0/scripts/tokenizer/tokenizer.perl4https://bitbucket.org/anoopk/indic nlp libraryPACLIC 32 - WAT 20181004 32nd Pacific Asia Conference on Language, Information and Computation The 5th Workshop on Asian Translation Hong Kong, 1-3 December 2018 Copyright 2018 by the authorsData Pair Source TargetOriginal Vocab Merge Final Vocab Original Vocab Merge Final VocabBN-EN 90,482 8,000 8,394 56,498 5,000 5,248HI-EN 24,470 4,000 4,286 24,380 4,000 4,150ML-EN 253,360 10,000 10,351 58,320 5,000 5,273SI-EN 169,603 9,000 9,392 72,093 7,000 7,417TA-EN 18,723 3,500 3,675 18,723 2,000 2,114TE-EN 12,728 2,000 2,230 9,929 1,500 1,633UR-EN 13,581 3,000 3,268 12,854 2,000 2,126Table 2: Original vocabulary size, number of BPE merge and final vocabulary size after applying BPE for each trainingdata pair. We decided the BPE merge values without any rigorous exploration.3.3 Experimental SetupWe train 2 multilingual models namely XX→EN(Indic languages to English) and EN→XX (Englsihto Indic languages) and 14 bilingual models (7 forIndic languages to English, and 7 for English to In-dic languages). All of these models are based onTransformer (Vaswani et al., 2017) network. Fortraining the models, we use Sockeye (Hieber et al.,2017), a toolkit for NMT. Each token in training,development and test sets are split in subword unitsin preprocessing stage. Along with that an addi-tional token5 indicating which Indic language a sen-tence pair belong to is added at the beginning ofevery source6 sentence. Then parallel data of allpairs are appended in one parallel corpus with In-dic languages in one side and English on other side,for training a single multilingual model for eachof EN→XX and XX→EN directions. These to-kens are added with development and test sets tooand likewise, development sets are also appendedin a single development set. We set embedding di-mension of 512, hidden dimension of 512, learningrate of 0.0002, dropout rate of 0.2. We use Adam(Kingma and Ba, 2015) optimizer. We keep mini-batch size of 2000 words7, and maximum sentencelength is restricted to 50. Rest of the hyperparame-ters are set to the default values of Sockeye. Train-ing is completed on meeting early-stopping criteria5We use the followings tokens: BN##, HI##, ML##, SI##,TA##, TE##, UR##6Source can be either English or any Indic language depend-ing on translation direction.7Sockeye supports word based batching too.(BLEU based, 10 patience) on development set. Fi-nally, the best model is used for translating the testsets.System Bi Multi NBN 18.24 20.05 +1.81HI 27.11 32.95 +5.84ML 10.56 19.94 +9.38SI → EN 18.22 21.35 +3.13TA 11.58 22.42 +10.84TE 16.15 30.96 +14.81UR 20.02 26.56 +6.54BN 13.38 13.27 -0.11HI 24.25 26.60 +2.35ML 20.92 13.50 -7.42EN → SI 12.75 10.64 -2.11TA 11.88 18.81 +6.93TE 14.21 25.81 +11.60UR 18.73 21.48 +2.75Table 3: BLEU scores of our {BN, HI, ML, SI, TA, TE,UR}→EN and EN→{BN, HI, ML, SI, TA, TE, UR} sys-tems; Bi: Bilingual Model; Multi: Multilingual Model;N denotes improvement of multilingual model over bilin-gual model.4 ResultsBLEU scores of bilingual and Multilingual systemsare shown in Table 3. For Indic languages to En-glish (XX→EN), BLEU score increases in each pairof multilingual system compared to bilingual sys-tem of that pair. Here, at target side decoder hasPACLIC 32 - WAT 20181005 32nd Pacific Asia Conference on Language, Information and Computation The 5th Workshop on Asian Translation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors(a) EN→HI (b) HI→EN(c) EN→TA (d) TA→ENFigure 1: Official bar charts showing Adequacy, Pairwise Human Evaluation, BLEU, RIBES and AM-FM scores oftop systems for multilingual Indic languages shared task at WAT 2018.to deal with only one language i.e. English (EN).It shows sharing of parameters improves the perfor-mance of high-resource language pairs ({BN, ML,SI}→EN) as well as low-resource language pairs({HI, TA, TE, UR}→EN). Unlike XX→EN, BLEUscores of EN→XX improve only for low-resourcelanguage pairs (EN→{HI, TA, TE, UR}) at the costof BLEU scores of high-resource language pairs(EN→{BN, ML, SI}). For EN→XX multilingualsystem, a single decoder has to deal with multi-ple languages with different vocabulary and differ-ent linguistic features; that is why it is difficult for asingle decoder to handle information of each targetlanguage.For multilingual shared task, the official PairwiseHuman evaluation and Adequacy scores (Nakazawaet al., 2018) were released for four translation di-rections only: EN→HI, HI→EN, EN→TA andTA→EN. Figure 1 shows the comparison of Pair-wise Human evaluation and Adequacy scores alongwith BLEU, RIBES (Isozaki et al., 2010), AM-FM(Banchs et al., 2015) scores of top three systems foreach of the four translation directions.5 ConclusionIn this paper, we described our submission to WAT2018 multilingual Indic languages shared task. Wesubmitted two multilingual NMT models: many-to-one (7 Indic languages to English) and one-to-many(English to 7 Indic languages). Our multilingualNMT is based on Transformer architecture. We eval-uated our models using BLEU score and found thatmultilingual NMT performs better than separatelytrained bilingual NMT models when the target sidehas only one language (English) and the improve-ment is higher for low-resource languages (up to14.81 BLEU points). However, performance of mul-tilingual NMT degrades compared to bilingual mod-PACLIC 32 - WAT 20181006 32nd Pacific Asia Conference on Language, Information and Computation The 5th Workshop on Asian Translation Hong Kong, 1-3 December 2018 Copyright 2018 by the authorsels for the relatively high-resource languages whenthe target has many languages.ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. International Confer-ence on Learning Representation(ICLR).Rafael E Banchs, Luis F D’Haro, and Haizhou Li. 2015.Adequacy-fluency metrics: Evaluating mt in the con-tinuous space model framework. IEEE/ACM Trans-actions on Audio, Speech and Language Processing(TASLP), 23(3):472–482.Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the proper-ties of neural machine translation: Encoder-decoderapproaches. In Proceedings of SSST-8, Eighth Work-shop on Syntax, Semantics and Structure in StatisticalTranslation, pages 103–111.Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, andHaifeng Wang. 2015. Multi-task learning for multi-ple language translation. In Proceedings of the 53rdAnnual Meeting of the Association for ComputationalLinguistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 1: LongPapers), pages 1723–1732. Association for Computa-tional Linguistics.Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016.Multi-way, multilingual neural machine translationwith a shared attention mechanism. In Proceedingsof the 2016 Conference of the North American Chap-ter of the Association for Computational Linguistics:Human Language Technologies, pages 866–875. As-sociation for Computational Linguistics.Felix Hieber, Tobias Domhan, Michael Denkowski,David Vilar, Artem Sokolov, Ann Clifton, and MattPost. 2017. Sockeye: A toolkit for neural machinetranslation. arXiv preprint arXiv:1712.05690.Hideki Isozaki, Tsutomu Hirao, Kevin Duh, KatsuhitoSudoh, and Hajime Tsukada. 2010. Automatic evalu-ation of translation quality for distant language pairs.In Proceedings of the 2010 Conference on EmpiricalMethods in Natural Language Processing, pages 944–952. Association for Computational Linguistics.Melvin Johnson, Mike Schuster, Quoc V. Le, MaximKrikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,Fernanda Viégas, Martin Wattenberg, Greg Corrado,Macduff Hughes, and Jeffrey Dean. 2017. Google’smultilingual neural machine translation system: En-abling zero-shot translation. Transactions of the Asso-ciation for Computational Linguistics, 5:339–351.Nal Kalchbrenner and Phil Blunsom. 2013. Recurrentcontinuous translation models. In Proceedings of the2013 Conference on Empirical Methods in NaturalLanguage Processing, pages 1700–1709.Diederik P Kingma and Jimmy Ba. 2015. Adam:A method for stochastic optimization. InternationalConference on Learning Representation(ICLR).Toshiaki Nakazawa, Shohei Higashiyama, ChenchenDing, Raj Dabre, Anoop Kunchukuttan, Win Pa Pa,Isao Goto, Hideya Mino, Katsuhito Sudoh, and SadaoKurohashi. 2018. Overview of the 5th workshop onasian translation. In Proceedings of the 5th Workshopon Asian Translation (WAT2018), Hong Kong, China,December.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Eval-uation of Machine Translation. In Proceedings of the40th annual meeting on association for computationallinguistics, pages 311–318, Philadelphia, Pennsylva-nia.Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural machine translation of rare words withsubword units. In Proceedings of the 54th AnnualMeeting of the Association for Computational Linguis-tics (Volume 1: Long Papers), pages 1715–1725. As-sociation for Computational Linguistics.Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural networks.In Advances in neural information processing systems,pages 3104–3112.Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in Neural Information Process-ing Systems, pages 5998–6008.PACLIC 32 - WAT 20181007 32nd Pacific Asia Conference on Language, Information and Computation The 5th Workshop on Asian Translation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors

IITP-MT at WAT2018: Transformer-based Multilingual Indic-English Neural Machine Translation System

https://www.research.ed.ac.uk/files/191923611/IITP_MT_at_WAT2018_SEN_DOA16112018_VOR_CC_BY.pdf

IITP-MT at WAT2018: Transformer-based Multilingual Indic-English Neural Machine Translation System

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer