Cleve, Anthony

Fink, Jérôme

Frénay, Benoît

English

Repository of the University of Namur

RESEARCH OUTPUTS / RÉSULTATS DE RECHERCHEAuthor(s) - Auteur(s) :Publication date - Date de publication :Permanent link - Permalien :Rights / License - Licence de droit d’auteur :Bibliothèque Universitaire Moretus PlantinInstitutional Repository - Research PortalDépôt Institutionnel - Portail de la Rechercheresearchportal.unamur.beDeep Learning Applied to Sign LanguageFink, Jérôme; Cleve, Anthony; Frenay, BenoîtPublished in:Reference AI &amp; ML Conference for Belgium, Netherlands &amp; Luxemburg 2019.Publication date:2019Document VersionPublisher's PDF, also known as Version of recordLink to publicationCitation for pulished version (HARVARD):Fink, J, Cleve, A & Frenay, B 2019, Deep Learning Applied to Sign Language. in K Beuls, B Bogaerts, GBo tempi, P Geurts, N Harley, B Lebichot, T Lenaerts, G Louppe & P Van Ecke (eds), Reference AI & MLConference for Belgium, Netherlands & Luxemburg 2019.: Proceedings of the 31st Benelux Conference onArtificial Intelligence (BNAIC 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn2019), Brussels, Belgium, November 6-8, 2019.. vol. 2491, CEUR Workshop Proceedings, CEUR WorkshopProceedings, BNAIC/BENELEARN 2019, Bruxelles, Belgium, 6/11/19.General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.            • Users may download and print one copy of any publication from the public portal for the purpose of private study or research.            • You may not further distribute the material or use it for any profit-making activity or commercial gain            • You may freely distribute the URL identifying the publication in the public portal ?Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 23. Jun. 2020Deep Learning Applied to Sign Language?Je´roˆme Fink1, Anthony Cle`ve1, and Benoˆıt Fre´nay1NADI Institute - PReCISE Research CenterUniversity of Namur - Faculty of Computer ScienceRue Grandgagnage 21, Namur - Belgium.1 ContextThe French Belgian Sign Language laboratory (LSFB-lab) recorded 150 hoursof conversations to create a corpus useful to analyze the Belgian sign languageand its usage [3]. The videos are captured in RGB without depth information.15 hours of these video have been annotated. They picture 52 different speakersand contains 670 signs with, at least, 10 occurrences. The LSFB-lab is interestedin the development of an algorithm able to translate sign language to french. Itcould help the deaf community during medical or administrative appointmentsas finding the right word is difficult in those specific contexts.This thesis aims to make a first step in that direction by exploring existingmethods developed in video recognition and analyzing the LSFB dataset.2 Related WorkDeep learning is successfully applied to image recognition challenges. To tacklethe problem of video recognition, methods for handling the temporal dimensionhave been developed. The mains methods identified are[1] :– LSTM networks: Each frame is presented to a network able to retain infor-mation about the evaluation of the previous frame. The retained informationcaptures the temporal dimension of the video.– Two stream networks: Two parallel networks focus on different aspects of theframes constituting a video. Usually, one network focuses on the temporalwhile the other extracts 2D information from one particular frame. Theirresults are then fused.– 3D Convolution layer : An extension of the 2D convolution able to handlethe temporal dimension induced by video.The state of the art architectures mix those methods.Other datasets have been considered to find the best for deep learning train-ing. Some identified datasets provide more examples per signs but the LSFBdataset has the advantage to capture more realistic signs as they are performedduring an active conversation. It is also one of the biggest dataset available interms of number of signs available. Each region has its sign language, thereforeit is impossible to fuse two datasets from different region of the world.? Copyright 2019 for this paper by its authors. Use permitted under Creative Com-mons License Attribution 4.0 International (CC BY 4.0).2 J. Fink et al.3 Preliminary Experimental ResultsTo reduce the scope of the problem, the thesis proposes and introduces the SignLanguage MNIST dataset with five classes shown by Fig 1.(a) Sign for 1 (b) Sign for 2 (c) Sign for 3 (d) Sign for 4 (e) Sign for 5Fig. 1: Example from the Sign Language MNIST dataset picturing digit from 1to 5. The angle of the camera hides one finger in the example for the number 5.Preliminary experiments have only considered one frame for each video. TheVGG16 and InceptionV3 models pre-trained on Imagenet have been tested. Theyhave been retrained using a triplet loss [2]. Fig 2 shows the result of this firstexperiment. VGG16 is able to capture more useful information than InceptionV3who is the state of the art in visual recognition.(a) InceptionV3 (b) VGG16Fig. 2: Confusion matrix for the Sign Language MNIST datasetEven if the results obtained in this master thesis are still preliminary, theyshow that deep learning is an interesting option for sign language recognition.This master thesis will be the starting point of new research in the context of aninterdisciplinary project, whose aim is to create a bilingual and context-sensitivedictionary for the french belgian sign language.References1. Joao Carreira and Andrew Zisserman, Quo Vadis, Action Recognition? A NewModel and the Kinetics Dataset. CVPR (2017)2. Herve Bredin, TristouNet: Triplet loss for speaker turn embedding. ICASSP (2017)3. Laurence Meurant, Corpus LSFB. First digital open access corpus of movies andannotations of French Belgian Sign Language (LSFB). (2015)

Deep Learning Applied to Sign Language

https://pure.unamur.be/ws/files/41628441/benelearn.pdf

Deep Learning Applied to Sign Language

Abstract

Similar works

Full text

Available Versions

Repository of the University of Namur