Machine Translation (MT) for sign languages (SLs) can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individuals. In this paper, we discuss automatic translation from English to Irish SL (ISL) in the domain of airport information. We describe our data collection processes and the architecture of the MaTrEx system used for our translation work. This is followed by an outline of the additional animation phase that transforms the translated output into animated ISL. Through a set of experiments, evaluated both automatically and

manually, we show that MT has the potential to assist Deaf people by providing information in their first language

Morrissey, Sara

English

Machine Translation (MT) for sign languages (SLs) can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individuals. In this paper, we discuss automatic translation from English to Irish SL (ISL) in the domain of airport information. We describe our data collection processes and the architecture of the MaTrEx system used for our translation work. This is followed by an outline of the additional animation phase that transforms the translated output into animated ISL. Through a set of experiments, evaluated both automatically and\ud
manually, we show that MT has the potential to assist Deaf people by providing information in their first language

Name not available

Assistive Translation Technology for DeafPeople: Translating into and Animating IrishSign LanguageSara MorrisseyNational Centre for Language TechnologySchool of Computing, Dublin City UniversityDublin 9, IrelandAbstract. Machine Translation (MT) for sign languages (SLs) can fa-cilitate communication between Deaf and hearing people by translatinginformation into the native and preferred language of the individuals.In this paper, we discuss automatic translation from English to Irish SL(ISL) in the domain of airport information. We describe our data col-lection processes and the architecture of the MaTrEx system used forour translation work. This is followed by an outline of the additionalanimation phase that transforms the translated output into animatedISL. Through a set of experiments, evaluated both automatically andmanually, we show that MT has the potential to assist Deaf people byproviding information in their first language.1 IntroductionCommunication is the essence of human interaction. However, this natural meansof communication can prove a barrier in cases where languages differ. This isnowhere more evident than in the communication barrier between languages ofdifferent modalities, namely spoken and signed languages.SLs are fully formed, natural and eloquent languages. This coupled with theright for all humans to communicate in a means that is most natural to themsignifies that Deaf people should be able to communicate in their preferred lan-guage. But in Ireland, just over 1% of the Irish non-Deaf population know IrishSign Language (ISL) (Leeson, 2003). Furthermore, most Irish Deaf people arenot confident in their English language skills (Conroy, 2006). These can greatlyhamper daily communication between the Deaf and hearing worlds. In addition,there is little by way of public services available through SLs and most of whichis available is on a one-to-one basis (O´’Baoill & Matthews, 2000).SL interpreters (SLIs) play a valuable role in breaking down the barriersbut they are often difficult to come by, particularly in Ireland where the ratioof interpreters to Deaf people is about 1:250 (Leeson, 2003). SLIs can be anexpensive and impractical option if only required for a short period of time.There are also issues of confidentiality in legal or medical matters, for example.Technological advancements such as teletype systems and subtitling can helpovercome the auditory barrier, but they assume good literacy skills and speedof reading and understanding on the part of the users in their second language(Huenerfauth, 2006). In addition, often, for reasons of time and space they canbe simplified resulting in a loss of information.There is, however, one particular type of technology that can overcome theimpractical and confidential issues regarding SLIs and allow Deaf people to accessinformation and communicate through SLs, this is machine translation (MT).Over the last 60 years MT has been used to bridge communication between spo-ken languages, with data-driven approaches taking centre-stage in recent years.In this paper, we will discuss the development of our data-driven MT systemfor SLs, focussing on English–ISL translation for airport announcements. Weinclude an overview of the state of the art and discuss the expansion of our workto include a computer-generated signing avatar.2 State of the ArtData-driven MT is the dominant methodology in the field of automatic trans-lation, and allows for robust, extensible, broad-coverage translation. In general,linguistic information is eschewed in favour of probabilistic models collected fromlarge parallel corpora. MT for SLs has been predominantly rooted in rule-basedapproaches, with significant work in this area only emerging within the last20 years, including the Zardoz system (Veale et al., 1998) and the ViSiCASTproject (Marshall & Sa´fa´r, 2002).There has been little work by way of data-driven MT, the dominant method-ology for spoken language MT, for SLs. Three primary systems have been de-veloped:– (Bauer et al., 1999) employ a statistical approach translating German SLto German using gesture recognition technology. Their work falls somewhatoutside the scope of the MT described in this paper, focussing on the oppositetranslation direction.– (Stein et al., 2006) have attempted translation both to and from SLs usinga phrase-based statistical MT (SMT) system. While they have incorporatedsome extra technological features such as gesture recognition and avatarproduction, their focus remains on MT and investigating SMT approaches.– (Wu et al., 2007) combine rule-based and data-driven approaches in orderto achieve translations for Chinese to Taiwanese SLs. Through using a rule-based approach their system lacks extensibility to new language pairs anddirections. Also the system does not produce sign via avatar.3 MethodologyOur methodology comprises three parts: data collection, data-driven MT andanimation of output.3.1 Data CollectionA pre-requisite for data-driven MT is a bilingual corpus of data. Suitable dataare difficult to find and those available tend to focus on non-domain specificinformation, such as the children’s stories of the European Cultural HeritageOnline (ECHO) project corpus1, that is not suitable for MT. Furthermore, datathat contains a text-based representation of SLs are uncommon. Having per-formed previous experiments on the ECHO data with some success, we chose tocreate our own purpose-built corpus. We chose the Air Traffic Information Sys-tem (ATIS) corpus (Hemphill et al., 1990) consisting of 595 sentences of Englishdialogue for its practical domain and suitability for MT. In order to create anauthentic parallel ISL corpus we engaged the assistance of two native ISL signersto work in tandem translating, signing and monitoring the creation of our videocorpus. The videos were then manually annotated with semantic glosses to createa parallel ISL ‘text’. Due to the laborious annotation process, the videos wereonly annotated at the level of the hands meaning some important non-manualfeature detail is lost. The EUDICO Linguistic Annotator (ELAN)2 was usedfor annotation. An example of the annotated videos is shown in Figure 1. Theresulting bitext is then used to feed the next phase: automated translation.Fig. 1. MaTrEx Architecture1 http://www.let.kun.nl/sign-lang/echo/2 http://www.mpi.nl/tools/elan.html3.2 The MaTrEx Data-driven MT SystemIn order to translate English text into animated ISL, we employ the MaTrExdata-driven MT system (Stroppa & Way, 2006). It is a hybrid system, developedat the National Centre for Language Technology, Dublin City University, thatavails of both example-based MT (EBMT) and SMT approaches. A diagram ofthe system architecture is shown in Figure 2.Fig. 2. MaTrEx ArchitectureThe decoder is the main engine that takes an input sentence and produces acandidate translation. It is fed by three bilingual data resources, namely alignedsentences, sub-sentential chunks and words. Each is derived from the bilingualcorpus described in the previous section. The decoder employs these bilinguallyaligned resources as databases of translation examples against which it com-pares input data. Translation links are retrieved for the matches found and arerecombined to produce the candidate target language translation. In the caseof English to ISL translation, the output takes the form of glossed annotations.These are then fed into the final animation phase.3.3 The Animation ProcessFor an SL MT system to be of practical use to the Deaf community it requiresreal SL to be produced rather than annotation. Real human signing, while pre-ferred by most (Naqvi, 2007), is not a practical option for signed output. Wechose to use computer-generated signing mannequins to animate the ISL. Weused Poser Animation Software Version 6.03 to create our animated avatar.Individual videos were manually constructed for each sign using the positioning3 http://www.curiouslabs.de/poser6.html?&L=1tools. It was not possible to produce fluid, seamless transitions between eachsigned video during sentence construction for this work. For this reason, there isa short pause between each sign where the mannequin returns to a neutral faceand hand position. While we recognise this is somewhat unnatural, we hope torectify this in the future. The animated signing avatar is shown in Figure 3.Fig. 3. The Animated Avatar4 Experiments and ResultsIn the first phase, using the ATIS data described above, we feed the MaTrExsystem. Three sub-experiments were run using different chunking methodologiesin an attempt to improve translation. The baseline SMT system does not in-clude any additional EBMT chunks. Type 1 chunks are formed using the MarkerHypothesis and Type 2 chunks are formed using the Marker Hypothesis for theEnglish data and segmenting the ISL data according to the semantic sign glosses.In order to objectively assess the quality of the candidate translations producedindependent of the avatar module, we employed automatic evaluation metrics,namely BLEU, word error rate (WER) and position-independent WER (PER).The results are shown in Table 1. The higher the BLEU score the better, thelower the error rate scores the better.System BLEU WER PEREN–ISLMaTrEx Baseline 38.85 46.02 34.33Baseline + Type 1 Chunks 39.11 45.90 34.20Baseline + Type 2 Chunks 39.05 46.02 34.21Table 1. Evaluation Scores for EN–ISL experimentsWe can see here that Type 1 chunks perform the best for this language pairingby improving on the BLEU score and decreasing the error rates. A WER scoreof 45.90 for Type 1 chunks indicates that the system gets 54.10% of translatedwords correct and in the correct order.In the second phase of our experiments, we fed the annotated output fromthe MT system into the avatar module to produce animated sign. The pre-made avatar videos for each sign in the translation are joined together to forma signed sentence. We engaged Deaf native ISL signers to manually evaluate theanimated output for intelligibility and fidelity. 82% were considered intelligibleby the monitors and 72% were considered good-to-excellent translations.5 Impact and ContributionsThis work addresses an MT methodology that has never been applied to SL MTto date, namely EBMT. Furthermore this approach describes a complete Englishto animated ISL translation system that forms part of a bi-directional multilin-gual MT system to assist in communication and comprehension for members ofthe Irish Deaf community.6 Conclusions and Future WorkWe have shown through experiments and comprehensive automatic and manualevaluation, that EBMT is a suitable and successful approach for translating intoSLs in the practical domain of airport announcements. Also the addition of anavatar module enables the production of output in the preferred language of theIrish Deaf Community.Future work includes researching improved chunking methodologies to im-prove translations, increased avatar development to enhance naturalness of sign-ing and the extension of theMaTrEx system to a fully functioning bi-directionalSL MT system, including provision for SL–SL translation.ReferencesB. Bauer, S. Nießen and H. Heinz. 1999. Towards an Automatic Sign Language Transla-tion System. In Proceedings of the International Workshop on Physicality and Tan-gibility in Interaction: Towards New Paradigms for Interaction Beyond the Desktop,Siena, Italy.P. Conroy. 2006. Signing In & Signing Out: The Education and Employment Experi-ences of Deaf Adults in Ireland. Research Report. Irish Deaf Society, Dublin.C. Hemphill and J. Godfrey and G. Doddington. 1990. The ATIS Spoken LanguageSystems Pilot Corpus. In Proceedings of the Workshop on Speech and NaturalLanguage, Hidden Valley, PA, pp. 96–101.M. Huenerfauth. 2006. Generating American Sign Language Classifier Predicates forEnglish-to-ASL Machine Translation. PhD Thesis, University of Pennsylvania,Philadelphia, PA.L. Leeson. 2003. M. Cronin and C. O´ Cuilleana´in(Eds.), Languages of Ireland. FourCourts Press: Dublin, Ireland. Chapter 8, pp. 148–164.S. Naqvi. 2007. End-User Involvement in Assistive Technology Design for the Deaf AreArtificial Forms of Sign Language Meeting the Needs of the Target Audience? InProceedings of the Conference and Workshop on Assistive Technologies for Peoplewith Vision & Hearing Impairments (CVHI), Granada, Spain, [no page numbers].D. O´’Baoill and P. A. Matthews. 2000. The Irish Deaf Community (Volume 2): TheStructure of Irish Sign Language. The Linguistics Institute of Ireland, Dublin,Ireland.I. Marshall and E´. Sa´fa´r. 2002. Sign Language Generation using HPSG. In Proceedingsof the 9th International Conference on Theoretical and Methodological Issues inMachine Translation (TMI-02), Keihanna, Japan, pp.105–114.D. Stein, J. Bungeroth and H. Ney. 2006. The Architecture of an English Text-to-Sign Languages. In Proceedings of the 11th Annual Conference of the EuropeanAssociation for Machine Translation (EAMT, ’06), Oslo, Norway, pp. 169–177.N. Stroppa and A. Way. 2006. MaTrEx: DCU Machine Translation System for IWSLT2006. In Proceedings of the International Workshop on Spoken Language Transla-tion (IWSLT), Kyoto, Japan, pp. 31–36.T. Veale, A. Conway, and B. Collins. 2000. The Challenges of Cross-Modal Translation:English to Sign Language Translation in the Zardoz System. Machine Translation13(1):81–106.C-H. Wu and H-Y. Su and Y-H. Chiu and C-H. Lin. 2007. Transfer-Based StatisticalTranslation of Taiwanese Sign Language Using PCFG. ACM Transactions on AsianLanguage Information Processing (TALIP), 6(1):[no page numbers].

Assistive translation technology for deaf people: translating into and animating Irish sign language

DCU Online Research Access Service

Generating American Sign Language Classiﬁer Predicates for English-to-ASL Machine Translation.

Languages of Ireland. Four Courts Press:

MaTrEx: DCU Machine Translation System for IWSLT

Sign Language Generation using HPSG.

Signing In & Signing Out: The Education and Employment Experiences of Deaf Adults in Ireland. Research Report. Irish Deaf Society,

The Architecture of an English Text-toSign Languages.

The ATIS Spoken Language Systems Pilot Corpus.

The Challenges of Cross-Modal Translation: English to Sign Language Translation in the Zardoz System.

The Irish Deaf Community (Volume 2): The Structure of Irish Sign Language. The Linguistics Institute of Ireland,

Towards an Automatic Sign Language Translation System.

Transfer-Based Statistical Translation of Taiwanese Sign Language Using PCFG.

http://doras.dcu.ie/15199/1/Morrissey_icchp_08.pdf

Assistive translation technology for deaf people: translating into and animating Irish sign language

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service