We give an update on the Found in Translation (FoTran) project, focusing on the study of emerging language-agnostic representations from neural machine translation (NMT). We describe our attention-bridge model, a modular NMT model which connects language-specific components through a shared network layer. Our latest implementation supports distributed training over many nodes and GPUs in order to substantially scale up the number of languages that can be included in a modern neural translation architecture.Peer reviewe

Boggia, Michele

Grönroos, Stig-Arne

Loppi, Niki A.

Raganato, Alessandro

Tiedemann, Jörg

Vazquez , Raul

English

Helsingin yliopiston digitaalinen arkisto

https://helda.helsinki.fiþÿ L a t e s t   D e v e l o p m e n t   i n   t h e   F o T r a n   P r o j e c t      S c a l i n g   U pLanguage Coverage in Neural Machine Translation UsingDistributed Training with Language-Specific ComponentsVazquez , RaulEuropean Association for Machine Translation2022Vazquez , R , Boggia , M , Raganato , A , Loppi , N A , Grönroos , S-A & Tiedemann , Jþÿ 2 0 2 2   ,   L a t e s t   D e v e l o p m e n t   i n   t h e   F o T r a n   P r o j e c t      S c a l i n g   U p   L a n g u a g e   C o v e r a g e   i nNeural Machine Translation Using Distributed Training with Language-Specific Components. in H Moniz , L Macken & A Rufener, et al. (eds) , Proceedings of the 23rd AnnualConference of the European Association for Machine Translation . European Association forMachine Translation , Geneva , pp. 311-312 , Annual Conference of the EuropeanAssociation for Machine Translation , Ghent , Belgium , 01/06/2022 . <https://aclanthology.org/2022.eamt-1.45 >http://hdl.handle.net/10138/352028cc_by_nc_sapublishedVersionDownloaded from Helda, University of Helsinki institutional repository.This is an electronic reprint of the original article.This reprint may differ from the original in pagination and typographic detail.Please cite the original version.Latest Development in the FoTran Project – Scaling Up LanguageCoverage in Neural Machine Translation Using Distributed Training withLanguage-Specific ComponentsRaúl Vázquez♠ Michele Boggia♠ Alessandro Raganato♡Niki A. Loppi♢ Stig-Arne Grönroos♠♣ Jörg Tiedemann♠♠ University of Helsinki, Finland ♡ University of Milano-Bicocca, Italy♢ NVIDIA ♣ Silo.AI♠{name.surname}@helsinki.fi, ♡{name.surname}@unimib.it,♢nloppi@nvidia.comAbstractWe give an update on the Found in Trans-lation (FoTran) project, focusing on thestudy of emerging language-agnostic rep-resentations from neural machine transla-tion (NMT). We describe our attention-bridge model, a modular NMT modelwhich connects language-specific compo-nents through a shared network layer. Ourlatest implementation supports distributedtraining over many nodes and GPUs in or-der to substantially scale up the number oflanguages that can be included in a modernneural translation architecture.1 IntroductionThe FoTran project aims at developing models fornatural language understanding trained on implicitinformation given by large collections of humantranslations.1 It is funded by a European ResearchCouncil consolidation grant, running from 2018to 2023 within the language technology researchgroup at the University of Helsinki under coordi-nation of Prof. Jörg Tiedemann.Cross-lingual grounding, useful for resolvingambiguities through translation, is a guiding prin-ciple of the project. Consequently, we developed amodel for multilingual NMT specifically designedto obtain meaning representations injected withmultilingual data (Vázquez et al., 2020). Formerproject results pointed towards the improvementof both the translation quality and the abstractionsacquired by our model when including more lan-guages (Vázquez et al., 2019; Raganato et al.,© 2022 The authors. This article is licensed under a CreativeCommons 3.0 licence, no derivative works, attribution, CC-BY-ND.1http://www.helsinki.fi/fotran2019). Due to the use of language-specific mod-ules, the overall model architecture grows whenlanguages and translation directions are added.Doing this on a single device does not scale beyondthe memory limits of that specific computing node.This is a limitation for testing the project hypothe-sis that training on increasing amounts of linguisti-cally diverse data improves the abstractions foundby the model – eventually leading to language-independent meaning representations useful formachine translation and tasks that require semanticreasoning and inference. Here, we propose strate-gies to address those issues: (1) distribute mod-ules across several processing units, (2) efficientlytrain the network over many translation directions,and (3) reuse the trained modules without havingto load the entire network. These together delivera cost-effective multilingual NMT system that canfurther be used for extracting multilingual meaningrepresentations.2 Despite the high computationalresources needed to scale up the number of trans-lation directions when training the model, its mod-ularity allows to reuse the trained components onrelatively small processing units, making multilin-gual models more affordable and increasing theiravailability.2 MethodologyThe implementation follows an encoder–decoderarchitecture, incorporating language specific en-coders and decoders to enable multilingual train-ing. They are connected via a shared inner-attention layer that summarizes the encoderinformation in a fixed-size vector representa-tion (Vázquez et al., 2019), which in turn canbe applied to downstream tasks (Vázquez et al.,2https://github.com/Helsinki-NLP/FoTraNMT2020). We refer to all encoders, decoders and tothe shared layer as modules. Encoders and de-coders are language-specific as they only see train-ing data from specific translation directions.We distribute the model across multiple process-ing units by loading, in each device, encoders anddecoders for a subset of the training translation di-rections. The inner-attention layer is shared acrossall processing units. All modules that are presentin more than one device are initialized with thesame weights, and gradients for these parametersare communicated across devices to ensure thatthey remain synchronous.In general, allocating language pairs with com-mon source/target languages on the same de-vice decreases both the total memory footprintof the model and the amount of communica-tion needed to keep the modules synced. For-mally, we define the partition of the training lan-guage pairs Lij = (Sij , Tij) over N units asP = {(L11, L12, . . .), . . . , (LN1, LN2, . . .)},where the first subscripts indicate to which de-vice each pair is assigned, and the second is anincremental index over all pairs assigned to thesame device. Whenever Xij = Xkl for i ̸= k,with X ∈ {S, T} representing either a source ora target language, we need to load a copy of thesame module in devices i and k. This will also im-pact the training time as it requires communicatinggradients across devices to keep modules synced.However, when dealing with a high number oftranslation directions (and a limited number ofsource and target languages) it becomes impossi-ble to avoid this condition: gathering together lan-guage pairs based on the source (target) languagecould result in a scattered configuration based ontarget (source) languages. We address these prob-lems using two strategies. First, we solve an allo-cation problem to minimize inter-device commu-nication. Since in most cases the problem has nofeasible exact solutions, we approximate a solutionusing the Hungarian algorithm over a cost matrixthat makes it cheaper to assign the same languageto a given GPU. Second, we propose to schedulethe gradient updates to minimize the waiting timewhen inter-device communication happens.At each training step the ith device starts per-forming a forward pass over a training batch forthe language pair Li1 and accumulates gradientsover all the language pairs Lij , where j runs fromone to the number of language pairs assigned tothe processing unit. Afterwards, gradients of mod-ules that are present in multiple processing unitsare averaged across devices. Module weights arethen updated according to the computed gradients.We ensure that all copies of all modules have non-zero gradients that can be communicated, prevent-ing the training loop from hanging.We also save the modules individually to beloaded and used independently in an efficientway. This makes the system more portable anduser-friendly for further fine-tuning, generatingtranslations, and experimentation with multilin-gual sentence-representations.3 Final RemarksFoTran aims at testing and analyzing representa-tions obtained from massively multilingual NMTsystems, and we devised a model architecture thatis optimized for training large models (with a suf-ficiently large high-performance cluster). Aftertraining, it can also easily be used in non-resource-intensive settings due to its modular design. Next,we intend to systematically explore the effect ofincreasing language diversity and how the abstrac-tion capabilities of the inner representations are af-fected in different settings.AcknowledgmentsThis work is part of the FoTran project, funded bythe European Research Council (ERC) under theEU’s Horizon 2020 research and innovation program(agreement № 771113). We also thank the CSC-IT Center for Science Ltd., for computational resources andNVIDIA AI Technology Center (NVAITC) for the expertisein distributed training.ReferencesRaganato, Alessandro, Raúl Vázquez, Mathias Creutzand Jörg Tiedemann. 2019. An Evaluationof Language-Agnostic Inner-Attention-Based Rep-resentations in Machine Translation. Proceedingsof the 4th Workshop on Representation Learning forNLP (RepL4NLP-2019). ACL. 27–32.Vázquez, Raúl, Alessandro Raganato, Mathias Creutzand Jörg Tiedemann. 2019. Multilingual NMT witha Language-Independent Attention Bridge. Pro-ceedings of the 4th Workshop on RepresentationLearning for NLP (RepL4NLP-2019). ACL. 33–39.Vázquez, Raúl, Alessandro Raganato, Mathias Creutzand Jörg Tiedemann. 2020. A Systematic Study ofInner-Attention-Based Sentence Representations inMultilingual Neural Machine Translation. Compu-tational Linguistics, vol. 46(2):387–424.

Latest Development in the FoTran Project – Scaling Up Language Coverage in Neural Machine Translation Using Distributed Training with Language-Specific Components

https://helda.helsinki.fi/bitstream/10138/352028/1/2022.eamt_1.45.pdf

Latest Development in the FoTran Project – Scaling Up Language Coverage in Neural Machine Translation Using Distributed Training with Language-Specific Components

Abstract

Similar works

Full text

Available Versions

Helsingin yliopiston digitaalinen arkisto