Recently, translation systems based on neural networks are starting to compete with systems based on phrases. The systems which are based on neural networks use vectorial repre- sentations of words. However, one of the biggest challenges that machine translation still faces, is dealing with large vocabularies and morphologically rich languages. This work aims to adapt a neural machine translation system to translate from Chinese to Spanish, using as input different types of granularity: words, characters, bitmap fonts of Chinese characters or words. The fact of performing the interpretation of every character or word as a bitmap font allows for obtaining more informed vectorial representations. Best results are obtained when using the information of the word bitmap font.Postprint (published version

Aldón Mínguez, David

Rodríguez Fonollosa, José Adrián

Ruiz Costa-Jussà, Marta

English

UPCommons. Portal del coneixement obert de la UPC

Baltic J. Modern Computing, Vol. 0 (2016), No. 0, pp. 00–00Neural Machine Translation using Bitmap FontsDavid Aldo´n Mı´nguez, Marta R. Costa-jussa`, Jose´ A. R. FonollosaUniversitat Polite`cnica de Catalunya, Barcelonadavid.aldon@est.fib.upc.edu, {marta.ruiz,jose.fonollosa}@upc.eduAbstract. Recently, translation systems based on neural networks are starting to compete withsystems based on phrases. The systems which are based on neural networks use vectorial repre-sentations of words. However, one of the biggest challenges that machine translation still faces,is dealing with large vocabularies and morphologically rich languages. This work aims to adapt aneural machine translation system to translate from Chinese to Spanish, using as input differenttypes of granularity: words, characters, bitmap fonts of Chinese characters or words. The fact ofperforming the interpretation of every character or word as a bitmap font allows for obtainingmore informed vectorial representations. Best results are obtained when using the information ofthe word bitmap font.1 IntroductionDeep learning (or neural networks) allows to solve problems that require the processingof big amounts of data1. Neural networks try to simulate a common feature of humanbeings, the accumulated experience. In brief, neural networks are not more than a sim-plified and artificial model that try to emulate the features of a human brain, whichmostly consist of:1. Processing units that exchange data or information.2. Use them to recognize patterns, including bitmap fonts, manuscript and time se-quences.3. Have the ability of learning and improving its operation mode.1 http://blog.cit.upc.edu/?p=9862 David Aldo´n Mı´nguez, Marta R. Costa-jussa`, Jose´ A. R. FonollosaMachine translation (MT) can be defined a set of algorithms that aims at trans-forming a source language to a target language. Since the decade of the 90s, statisticaltranslation systems, among which phrase-based systems, have prevailed over others.These statistical translation systems maximize the probability of target phrases givensource phrases.Recently, systems based on neural networks, which use vector representations ofwords, have began to have a great relevance [Kalchbrenner and Blunsom2013, Cho etal.2014, Sutskever et al.2014]. The big obstacle that these systems arise is the inabilityto deal with large vocabularies. This problem originates because of the architecture ofthese systems, not to mention their computational cost. In this work, we are introducingthe bitmap font information of the input unit (either word or character) in order to pro-vide the neural MT system with an informed initialization instead of random [Bahdanauet al.2015].The rest of the paper is structured as follows. Section 2 reviews the related work,both in the specific task of Chinese-Spanish and in neural MT. Section 3 explains thebaseline neural MT system, and our contribution of integrating translation unit (wordsor characters) bitmap fonts. Section 4 describes the experimental framework and theresults obtained. Finally, Section 5 concludes.2 Related WorkThis section does a brief overview of works that approach Chinese-Spanish translationand previous works in neural MT.2.1 Chinese to SpanishSurprisingly, there are not many publications in the field of automatic translation be-tween the pair Chinese-Spanish despite being two of the most spoken languages in theworld, occupying the first position and the third respectively [Costa-jussa`2015].A work that was done was the creation of a pseudo corpus which is intended totranslate English to Chinese or to Spanish and create an artificial corpus for the as-sociation of Spanish-Chinese [Banchs et al.2006]. The problem was tried to resolveChinese-Spanish with Spanish-English and English-Chinese corpora. As reference sys-tem, they use a system based on n-grams that differs from the phrases mainly in thetranslation model.Neural Machine Translation using Bitmap Fonts 3In 2008, a IWSLT2 (International Workshop on Spoken Language Translation) eval-uation between these two languages was performed. There were two tasks for Spanish-Chinese. The first task was based on a direct translation for Spanish-Chinese. The sec-ond task was motivated by the fact that there is little corpus between Spanish-Chinese,but many among the Chinese-English and English-Spanish, so the task proposed con-sisted in translating from Chinese-Spanish by pivoting through English. As a final re-sult, the second task, the pivot technique, performed better than direct translation be-cause of the larger corpus provided. [Costa-jussa` et al.2012] show a comparison be-tween two types of standard pivots (pseudo corpus and cascade) using English and thedirect system. These results show that the pivot and direct techniques do not differ muchin their results, but that the technical pivot cascade is slightly better than the pseudo cor-pus.Differently, [Costa-jussa` and Centelles2016] presents the first rule-based MT sys-tem for Chinese to Spanish. Authors describe a hybrid method for constructing thissystem taking advantage of available resources such as parallel corpora that are used toextract dictionaries and lexical and structural transfer rules.Additionally, to all this research, there is the Chispa Android application and webservice3, that can be useful to tourists or traveling between countries, which use theselanguages [Centelles et al.2014].2.2 Neural Machine TranslationText translation via deep learning relies on an autoencoder structure [Bahdanau etal.2015] to translate from a source to a target language. The autoencoder is trainedusing translated texts. Source words are mapped to a small space. The new represen-tation of words is encoded in a summary vector (a representation of source sentence)using a recurrent neural network. Then, the summary vector is decoded into the targetlanguage. From 2013, there were different groups proposing competitive architecturesthat have progressed towards this new approach of neural MT [Kalchbrenner and Blun-som2013, Sutskever et al.2014]. And in 2015, [Bahdanau et al.2015] proposed to usegated recursive units (i.e. attention-based mechanism) that allows a better performanceon long sentences. This same attention-based mechanism is also used to describe thecontent of bitmap fonts [Xu et al.2015]. [Lamb and Xie2015] propose a general Convo-2 http://iwslt2010.fbk.eu3 http://www.chispa.me4 David Aldo´n Mı´nguez, Marta R. Costa-jussa`, Jose´ A. R. Fonollosalutional Neural Network (CNN) encoder model for MT that fits within in the frameworkof encoder-decoder models.3 Theoretical description of the systemIn this section, we introduce the neural MT baseline system together with the maintechnique that we propose to enhance it.3.1 Baseline SystemAs a baseline system, see Figure 1, we use the neural network system proposed by [Bah-danau et al.2015] which is mainly based on a encoder-decoder with the attention-basedmechanism. For simplification, the Figure does not show the attention-based mecha-nism.Fig. 1. Encoder/decoder neural MT scheme.The encoder takes a source sentence, and encodes each word in 1-of-K coding vec-tors, then trains word embeddings which will be codified into a summary vector throughthe recurrent network with attention. Then, the decoder applies the reverse process ob-taining the destination sentence.Neural Machine Translation using Bitmap Fonts 53.2 Adding Word Bitmap fontsIn this study, the previous system will be enhanced to be able to use word bitmap fonts toadd further information to the neural system. Given that the baseline representations ofvectors are random bits 0/1 level (with the 1-of-K coding), we propose to create a moreinformative representation. We represent Chinese words by means of 2-dimensionalbitmap which reflects the shape of the written word characters. See Figure 2 for illus-tration.Fig. 2. Chinese Word RepresentationLike this, we are converting Chinese words to bitmap fonts. Then we can get thevector of bits representing the bitmap fonts obtained from each word. In this way it is notonly providing more information to the system, but it is contributing with much smarterinformation than a set of random values, due to the characteristic that offer the neuralnetworks learning patterns. This new bitmap font vector becomes the initialization ofword embeddings used in the encoder. See the integration of this new encoding in thesystem in Figure 3.4 Experiments and ResultsIn this work, we use the Chinese-Spanish parallel corpus, United Nations Corpus (UN)[Rafalovitch and Dale2009]. Corpus statistics are shown in Table 1. Statistics for Chi-nese are shown both with word and character segmentations. In the case of word seg-mentation, the size of the vocabulary is similar to the target vocabulary, while in thecase of using Chinese characters, we have a much lower vocabulary.6 David Aldo´n Mı´nguez, Marta R. Costa-jussa`, Jose´ A. R. FonollosaFig. 3. New Encoder-DecoderL Set S W VESTrain 58.6K 2.3M 22.5KDev 990 43.4K 5.4kTest 1K 44.2K 5.5KZH WordsTrain 58.6K 1.6M 17.8KDev 990 33K 3.7KTest 1K 33.7K 3.8KZH CharactersTrain 58.6K 2.8M 3.8KDev 990 53.9K 1.7KTest 1K 55.1K 1.7KTable 1. Corpus details. Number of sentences (S),words (W), vocabulary (V). M stands for mil-lions and K stands for thousands.The neural-based system was built using the software available in github 4. Wegenerally used settings from previous work: networks have an embedding of 529 and adimension of 1024. We used a vocabulary size of 20000 in Spanish, 3500 for Chinesewhen using characters and 15000 for Chinese when using words.Given that we are experimenting with either Chinese words or characters, we alsotried with both word/character initialization. In the case of words, the bitmap fonts hadless resolution than characters because the size is the same. Table 2 shows the results interms of BLEU.4 http://github.com/nyu-dl/dl4mt-tutorial/Neural Machine Translation using Bitmap Fonts 7System BLEUCharacters 5.52Characters +Bitmap 5.72Words 5.55Words +Bitmap 8.49Table 2. BLEU results. In bold, best results.Results show that using bitmap fonts as initialization is much better than using arandom initialization, since much more information is provided to the neural system.When using character bitmap fonts the improvement is of 0.2 BLEU points, while usingword bitmap fonts the improvement is of almost of 3 BLEU points. In any case, it isobserved that it is better to use words than characters as translation units.Type Sentence1Src 55/14 . 大会议事规则第Words Convenio sobre la emprendidas tratado+Bitmap Reforma en la Asamblea GeneralRef reglamento de la Asamblea General2Src ( e )小口径弹药；Words e ) el Medio comunicacio´n Mundial ;+Bitmap e ) La informacio´n pequen˜o calibre ;Ref e ) Municiones de pequen˜o calibre ;3Src 58/147 . 妇女的家庭暴力行为Words ampliamente de los derechos humanos+Bitmap discriminacio´n contra la mujerRef violencia contra la mujer en el hogar4Src ( e )法律和司法体制；Words e ) acceso a la tecnologı´a y la gestio´n de la informacio´n;+Bitmap e ) la jurı´dicas o judiciales ;Ref e ) las instituciones jurı´dicas y judiciales ;Table 3. Example Sentences. Source (Src), Baseline (Words), Bitmap fonts (+Bitmap), Reference(Ref)Table 3 shows some examples of the kind of improvements that the neural MTsystem with the new initialization is capable of. Examples show how it improves theadequacy and fluency of the translations in general.8 David Aldo´n Mı´nguez, Marta R. Costa-jussa`, Jose´ A. R. Fonollosa5 ConclusionsThis work has presented an alternative to the representation of 1-of-K coding usingbitmap fonts instead. We have tried to take advantage of the representation of patterns.Neural performance in this task is far from state-of-the-art results [Costa-jussa` etal.2012]. The fact of not achieving comparable performance to the standard phrase-based system may be due to the fact that we are using a small dataset. However, thisstudy shows a significant improvement when using a smarter initialization of the neuralword vectors from standard neural MT system. The bitmap font initialization defini-tively provides more information to the neural systems than the random 1-of-K vectors.When comparing Chinese words or characters, the performance is similar, but it makesa big difference introducing bitmap fonts of words instead of bitmap fonts of characters.However, experiments on larger corpus would be required and are left for further work.Software for experiments reported in this paper is freely available in github5.AcknowledgementsThis work is supported by the 7th Framework Program of the European Commis-sion through the International Outgoing Fellowship Marie Curie Action (IMTraP-2011-29951) and also by the Spanish Ministerio de Economı´a y Competitividad and the Eu-ropean Regional Development Fund, contract TEC2015-69266-P (MINECO/FEDER,UE).ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation byjointly learning to align and translate. In CoRR.Rafael Banchs, Josep Maria Crego, Patrik Lambert, and Jose´ B. Marin˜o. 2006. A feasibility studyfor chinese-spanish statistical machine translation. In in Procedings of the 5th InternationalSymposium on Chinese Spoken Language Processing (ISCSLP).Jordi Centelles, Marta R. Costa-jussa`, and Rafael E. Banchs. 2014. Chispa on the go: A mobilechinese-spanish translation service for travellers in trouble. In Proceedings of the Demonstra-tions at the 14th Conference of the European Chapter of the Association for ComputationalLinguistics, pages 33–36.5 https://github.com/aldomin/NMTbitMapsNeural Machine Translation using Bitmap Fonts 9Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On theproperties of neural machine translation: Encoder–decoder approaches. In CoRR.Marta R. Costa-jussa` and Jordi Centelles. 2016. Description of the chinese-to-spanish rule-basedmachine translation system developed using a hybrid combination of human annotation andstatistical techniques. ACM Transactions on Asian and Low-Resource Language InformationProcessing, 15.Marta R. Costa-jussa`, Carlos A. Henrı´quez Q., and Rafael E. Banchs. 2012. Evaluating indirectstrategies for chinese-spanish statistical machine translation. Journal Of Artificial Intelli-gence Research, 45:762–780.Marta R. Costa-jussa`. 2015. Traduccio´n automa´tica estadı´stica entre chino y castellano. Kom-puter Sapiens, 1:16–36.Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for dis-course compositionality. In CoRR.Andrew Lamb and Michael Xie. 2015. Convolutional encoders for neural machine translation.In CoRR.Alexandre Rafalovitch and Robert Dale. 2009. United Nations General Assembly Resolutions:A Six-Language Parallel Corpus. In Proc. of the MT Summit XII, pages 292–299, Ottawa.Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neuralnetworks. In CoRR.Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov,Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image captiongeneration with visual attention. In CoRR.

Neural machine translation using bitmap fonts

Recently, translation systems based on neural networks are starting to compete with systems based on phrases. The systems which are based on neural networks use vectorial repre- sentations of words. However, one of the biggest challenges that machine translation still faces, is dealing with large vocabularies and morphologically rich languages. This work aims to adapt a neural machine translation system to translate from Chinese to Spanish, using as input different types of granularity: words, characters, bitmap fonts of Chinese characters or words. The fact of performing the interpretation of every character or word as a bitmap font allows for obtaining more informed vectorial representations. Best results are obtained when using the information of the word bitmap font

UPCommons

https://upcommons.upc.edu/bitstream/2117/102836/1/HyTra-5_paper4.pdf

Neural machine translation using bitmap fonts

Abstract

Similar works

Full text

Available Versions

UPCommons. Portal del coneixement obert de la UPC

UPCommons