Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting

In this paper, we introduce a neural rendering pipeline for transferring the facial expressions, head pose, and body movements of one person in a source video to another in a target video. We apply our method to the challenging case of Sign Language videos: given a source video of a sign language user, we can faithfully transfer the performed manual (e.g., handshape, palm orientation, movement, location) and non-manual (e.g., eye gaze, facial expressions, mouth patterns, head, and body movements) signs to a target video in a photo-realistic manner. Our method can be used for Sign Language Anonymization, Sign Language Production (synthesis module), as well as for reenacting other types of full body activities (dancing, acting performance, exercising, etc.). We conduct detailed qualitative and quantitative evaluations and comparisons, which demonstrate the particularly promising and realistic results that we obtain and the advantages of our method over existing approaches.Comment: Accepted at AI4CC Workshop at CVPR 202

Language identification from suprasegmental cues: study on two Modern Greek idioms

Phonetic study and comparison of two idioms of Greek (Idiom of Agiasos of Lesvos and idiom of Athens). From the collection of six hours of recorded material of each of the two idioms, phrases were selected which constituted the material for the experimental study of this research. Using synthetic sounds that emerged from the digital processing of the original recordings, three kinds of "masks" were created which overlaid the timbre and words of the original recordings of the speakers of the two dialects, leaving only distinct suprasegmental elements, the rhythm and the melody of each sentence. The sentences from the two dialects were included in an acoustic experiment, the participants of which were asked to identify whether or not the sentences they heard came from the same dialect. The experiment was a success and the basic working hypothesis of the research, namely that rhythm and melody are recognizable elements capable of denoting the identity of these two dialects, was proved.Φωνητική μελέτη και σύγκριση δυο ιδιωμάτων της Ελληνικής (Ιδίωμα Αγιάσου Λέσβου και ιδίωμα Αθηνών). Από την συλλογή των έξι ωρών ηχογραφημένου υλικού καθενός από τα δύο ιδιώματα επιλέχθηκαν προτάσεις οι οποίες αποτέλεσαν το υλικό για την πειραματική μελέτη αυτής της έρευνας. Κάνοντας χρήση συνθετικών ήχων που προέκυψαν από την ψηφιακή επεξεργασία των αυθεντικών ηχογραφήσεων, δημιουργήθηκαν τριών ειδών «μάσκες» οι οποίες επικάλυψαν την χροιά και τα λόγια των αυθεντικών ηχογραφήσεων των ομιλητών των δύο διαλέκτων αφήνοντας μόνα διακριτά υπερτεμαχιακά στοιχεία, τον ρυθμό και την μελωδία κάθε πρότασης. Οι προτάσεις από τις δύο διαλέκτους περιλήφθηκαν σε ένα ακουστικό πείραμα, οι συμμετέχοντες του οποίου καλούνταν να αναγνωρίσουν κατά πόσο οι προτάσεις που άκουγαν προέρχονταν ή όχι από την ίδια διάλεκτο. Το πείραμα στέφθηκε με επιτυχία και η βασική υπόθεση εργασίας της έρευνας, ότι δηλαδή ο ρυθμός και η μελωδία είναι στοιχεία αναγνωρίσιμα και ικανά να δηλώσουν την ταυτότητα των συγκεκριμένων δύο διαλέκτων, αποδείχθηκε.Étude phonétique et comparaison de deux idiomes du grec (Idiome d’Agiasos de et idiome d’Athènes). À partir de la collection de six heures de matériel enregistré de chacun des deux idiomes, des phrases ont été sélectionnées qui constituaient le matériel pour l’étude expérimentale de cette recherche. En utilisant des sons synthétiques issus du traitement digital des enregistrements originaux, trois types de « masques » ont été créés qui superposent le timbre et les mots des enregistrements originaux des locuteurs des deux dialectes, ne laissant que des éléments suprasegmentaux distincts, le rythme et la mélodie de chaque phrase. Les phrases des deux dialectes ont été incluses dans une expérience acoustique, dont les participants ont été invités à identifier si les phrases qu’ils ont entendues provenaient ou non du même dialecte. L’expérience a été un succès et l’hypothèse de travail de base de la recherche, à savoir que le rythme et la mélodie sont des éléments reconnaissables capables de désigner l’identité de ces deux dialectes, a été prouvée

Organizing a bilingual lexicographic database with the use of WordNet

This paper reports on the restructuring of a bilingual (Greek Sign Language, GSL – Modern Greek) lexicographic database with the use of the WordNet semantic and lexical database. The relevant research was carried out by the Institute for Language and Speech Processing (ILSP) / Athena R.C. team within the framework of the European project Easier. The project will produce a framework for intelligent machine translation to bring down language barriers among several spoken/written and sign languages. This paper describes the experience of the ILSP team to contribute to a multilingual repository of signs and their corresponding translations and to organize and enhance a bilingual dictionary (GSL – Modern Greek) as a result of this mapping; this will be the main focus of this paper. The methodology followed relies on the use of WordNet and, more specifically, the Open Multilingual WordNet (OMW) tool to map content in GSL to WordNet synsets

Sign Language Technologies and the Critical Role of SL Resources in View of Future Internet Accessibility Services

In this paper, we touch upon the requirement for accessibility via Sign Language as regards dynamic composition and exchange of new content in the context of natural language-based human interaction, and also the accessibility of web services and electronic content in written text by deaf and hard-of-hearing individuals. In this framework, one key issue remains the option for composition of signed “text„, along with the ability for the reuse of pre-existing signed “text„ by exploiting basic editing facilities similar to those available for written text that serve vocal language representation. An equally critical related issue is accessibility of vocal language text by born or early deaf signers, as well as the use of web-based facilities via Sign Language-supported interfaces, taking into account that the majority of native signers present limited reading skills. It is, thus, demonstrated how Sign Language technologies and resources may be integrated in human-centered applications, enabling web services and content accessibility in the education and an everyday communication context, in order to facilitate integration of signer populations in a societal environment that is strongly defined by smart life style conditions. This potential is also demonstrated by end-user-evaluation results

What about synthetic signing? A methodology for signer involvement in the development of avatar technology with generative capacity

<p>Although signing avatar technology seems to be the only option currently available to serve sign language (SL) display in the context of applications which demand generative capacity from the part of the technology like in machine translation to SL, signing avatars have not yet been accepted by signers' communities. One major factor for this rejection is the feeling that technology is developed without the involvement of its actual users. Aiming to invite the signers' community into the process of signing avatar development, we have designed the shell methodological framework for signer-informed technology which is implemented as on-line surveys addressed to signer communities of different SLs. The surveys are communicated via focused on-line questionnaires with content of signing avatar performance that allows rating of various aspects of the produced SL synthetic signing by human signers. Here we report on the first survey application with content from the Greek Sign Language (GSL). The analysis of the obtained results is 2-fold: it highlights the significance of signer involvement and the provided feedback in the technological development of synthetic signing; in parallel it reveals those aspects of the survey setup that need fine-tuning before its next distribution cycles. The implementation of the first on-line survey can be found in: https://sign.ilsp.gr/slt-survey/.</p&gt

Signing avatar performance evaluation within the EASIER project

<p>The direct involvement of deaf users in the development and evaluation of signing avatars is imperative to achieve legibility and raise trust among synthetic signing technology consumers. A paradigm of constructive cooperation between researchers and the deaf community is the EASIER project1, where user driven design and technology development have already started producing results. One major goal of the project is the direct involvement of sign language (SL) users at every stage of development of the project's signing avatar. As developers wished to consider every parameter of SL articulation including affect and prosody in developing the EASIER SL representation engine, it was necessary to develop a steady communication channel with a wide public of SL users who may act as evaluators and can provide guidance throughout research steps, both during the project's end-user evaluation cycles and beyond. To this end, we have developed a questionnaire-based methodology, which enables researchers to reach signers of different SL communities on-line and collect their guidance and preferences on all aspects of SL avatar animation that are under study. In this paper, we report on the methodology behind the application of the EASIER evaluation framework for end-user guidance in signing avatar development as it is planned to address signers of four SLs -Greek Sign Language (GSL), French Sign Language (LSF), German Sign Language (DGS) and Swiss German Sign Language (DSGS)- during the first project evaluation cycle. We also briefly report on some interesting findings from the pilot implementation of the questionnaire with content from the Greek Sign Language (GSL).</p&gt

Dicta-Sign -Building a Multilingual Sign Language Corpus

International audienceThis paper presents the multilingual corpus of four European sign languages compiled in the framework of the Dicta-Sign project. Dicta-Sign researched ways to enable communication between Deaf individuals through the development of human-computer interfaces (HCI) for Deaf users, by means of sign language. Sign language resources were compiled to inform progress in the other research areas within the project, especially video recognition of signs, sign-to-sign translation, linguistic modelling, and sign generation. The aim for the corpus data collection was to achieve as high a level of naturalness as possible with semi-spontaneous utterances under lab conditions. At the same time the elicited data were supposed to be semantically close enough to be comparable both across individual informants and for all four sign languages. The sign language data were annotated using iLex and are now made available via a web portal that allows for different access options to the data

EASIER Multilingual Sign Language Wordnet

A wordnet for a number of sign languages target in the EASIER project

Introducing sign languages to a multilingual Wordnet: Bootstrapping corpora and lexical resources of Greek Sign Language and German Sign Language

<p>Wordnets have been a popular lexical resource type for many years. Their sense-based representation of lexical items and numerous relation structures have been used for a variety of computational and linguistic applications. The inclusion of different wordnets into multilingual wordnet networks has further extended their use into the realm of cross-lingual research. Wordnets have been released for many spoken languages. Research has also been carried out into the creation of wordnets for several sign languages, but none have yet resulted in publicly available datasets. This article presents our own efforts towards an inclusion of sign languages in a multilingual wordnet, starting with Greek Sign Language (GSL) and German Sign Language (DGS). Based on differences in available language resources between GSL and DGS, we trial two workflows with different coverage priorities. We also explore how synergies between both workflows can be leveraged and how future work on additional sign languages could profit from building on existing sign language wordnet data. The results of our work are made publicly available.</p&gt