A survey on mouth modeling and analysis for Sign Language recognition

© 2015 IEEE.Around 70 million Deaf worldwide use Sign Languages (SLs) as their native languages. At the same time, they have limited reading/writing skills in the spoken language. This puts them at a severe disadvantage in many contexts, including education, work, usage of computers and the Internet. Automatic Sign Language Recognition (ASLR) can support the Deaf in many ways, e.g. by enabling the development of systems for Human-Computer Interaction in SL and translation between sign and spoken language. Research in ASLR usually revolves around automatic understanding of manual signs. Recently, ASLR research community has started to appreciate the importance of non-manuals, since they are related to the lexical meaning of a sign, the syntax and the prosody. Nonmanuals include body and head pose, movement of the eyebrows and the eyes, as well as blinks and squints. Arguably, the mouth is one of the most involved parts of the face in non-manuals. Mouth actions related to ASLR can be either mouthings, i.e. visual syllables with the mouth while signing, or non-verbal mouth gestures. Both are very important in ASLR. In this paper, we present the first survey on mouth non-manuals in ASLR. We start by showing why mouth motion is important in SL and the relevant techniques that exist within ASLR. Since limited research has been conducted regarding automatic analysis of mouth motion in the context of ALSR, we proceed by surveying relevant techniques from the areas of automatic mouth expression and visual speech recognition which can be applied to the task. Finally, we conclude by presenting the challenges and potentials of automatic analysis of mouth motion in the context of ASLR

Alphabet Sign Language Recognition Using Leap Motion Technology and Rule Based Backpropagation-genetic Algorithm Neural Network (Rbbpgann)

Sign Language recognition was used to help people with normal hearing communicate effectively with the deaf and hearing-impaired. Based on survey that conducted by Multi-Center Study in Southeast Asia, Indonesia was on the top four position in number of patients with hearing disability (4.6%). Therefore, the existence of Sign Language recognition is important. Some research has been conducted on this field. Many neural network types had been used for recognizing many kinds of sign languages. However, their performance are need to be improved. This work focuses on the ASL (Alphabet Sign Language) in SIBI (Sign System of Indonesian Language) which uses one hand and 26 gestures. Here, thirty four features were extracted by using Leap Motion. Further, a new method, Rule Based-Backpropagation Genetic Al-gorithm Neural Network (RB-BPGANN), was used to recognize these Sign Languages. This method is combination of Rule and Back Propagation Neural Network (BPGANN). Based on experiment this pro-posed application can recognize Sign Language up to 93.8% accuracy. It was very good to recognize large multiclass instance and can be solution of overfitting problem in Neural Network algorithm

RoboTalk - Prototyping a Humanoid Robot as Speech-to-Sign Language Translator

Information science mostly focused on sign language recognition. The current study instead examines whether humanoid robots might be fruitful avatars for sign language translation. After a review of research into sign language technologies, a survey of 50 deaf participants regarding their preferences for potential reveals that humanoid robots represent a promising option. The authors also 3D-printed two arms of a humanoid robot, InMoov, with special joints for the index finger and thumb that would provide it with additional degrees of freedom to express sign language. They programmed the robotic arms with German sign language and integrated it with a voice recognition system. Thus this study provides insights into human–robot interactions in the context of sign language translation; it also contributes ideas for enhanced inclusion of deaf people into society

ALPHABET SIGN LANGUAGE RECOGNITION USING LEAP MOTION TECHNOLOGY AND RULE BASED BACKPROPAGATION-GENETIC ALGORITHM NEURAL NETWORK (RBBPGANN)

Sign Language recognition was used to help people with normal hearing communicate effectively with the deaf and hearing-impaired. Based on survey that conducted by Multi-Center Study in Southeast Asia, Indonesia was on the top four position in number of patients with hearing disability (4.6%). Therefore, the existence of Sign Language recognition is important. Some research has been conducted on this field. Many neural network types had been used for recognizing many kinds of sign languages. However, their performance are need to be improved. This work focuses on the ASL (Alphabet Sign Language) in SIBI (Sign System of Indonesian Language) which uses one hand and 26 gestures. Here, thirty four features were extracted by using Leap Motion. Further, a new method, Rule Based-Backpropagation Genetic Al-gorithm Neural Network (RB-BPGANN), was used to recognize these Sign Languages. This method is combination of Rule and Back Propagation Neural Network (BPGANN). Based on experiment this pro-posed application can recognize Sign Language up to 93.8% accuracy. It was very good to recognize large multiclass instance and can be solution of overfitting problem in Neural Network algorithm

Language switching in aviation : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Aviation at Massey University, Manawatū, New Zealand

Clear and precise communication between pilots and air traffic controllers is a precondition for safe operations. Communication has long been identified as a major element of the cockpit–controller interface, explaining one third of general aviation incidents (Etem & Patten, 1998). Yet, despite multilingualism with English as the lingua franca being a characteristic of aviation communication, little research appears to have investigated the efficiency of operation of bilinguals alternating between their dominant, usually native, language and English in a bilingual air traffic environment. The studies undertaken for this research sought to rectify this situation by examining the cognitive aspects of situation awareness during language switching in aviation. Quantitatively and qualitatively analysed responses to an online-distributed survey aimed at investigating the current bilingual situation in aviation revealed that while situation awareness for the majority (76%) of native-English speakers was adversely affected by bilingualism, almost 30% of bilinguals also reported their situation awareness being affected. Subsequent experimental analyses using a language switching paradigm investigated how participants recognize a target call sign, identify an error and predict in bilingual compared with monolingual English conditions. The effect of the language condition participants’ native Chinese only, English only, or a mix of both, varied across the three tasks. Call sign recognition performance was found to be faster in the English condition than in the bilingual condition, but accuracy did not differ, a finding that was attributed to the effect of call sign similarity. However, when the task was more complicated, the difference between the conditions diminished. No effect on performance was found for simultaneously listening to two speech sources, which is potentially analogous to cockpit communication and radio calls. The error analyses served to test for response bias by calculating sensitivity, d', and decision criterion C in accordance with Stanislaw and Todorov’s (1999) Signal Detection Theory calculations. Several cognitive implications for practice were proposed, for example, in Crew Resource Management (CRM) training and personal airmanship development, exploration of own behavioural biases might be used to adjust the placement of the criterion. The cognitive implications largely focused on affecting attitudes to increase awareness. Attention was focused on performance of bilinguals to identify which language condition facilitated faster and more accurate responses. The findings were unable to support any of the conditions, leaving the question: Would a universal language for communication on radio frequencies be worth considering, to allow everyone to understand what is said? Disentangling the effects of language switching on the performance of bilingual pilots and air traffic controllers remains a task for future studies

Hand Gesture Recognization Using Virtual Canvas

Computer vision based hand tracking can be used to interact with computers in a new innovative way. The input components of a normal computer system include keyboard, mouse, joystick are avoided. Gesture recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, fingers, arms, head, and/or body. It is of utmost importance in designing an intelligent and efficient human–computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation to virtual reality. In this paper, we provide a survey on gesture recognition with particular emphasis on hand gestures and facial expressions. Existing challenges and future research possibilities are also highlighted. Gestures are expressive, meaningful body motions involving physical movements of the fingers, hands, arms, head, face, or body with the intent of conveying meaningful information orinteracting with the environment. A gesture may also be perceived by the environment as a compression technique for the information to be transmitted elsewhere and subsequently reconstructed by the receive

A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation

Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. It is a form of communication that conveys information, emotions, attitudes, and intentions without the use of spoken or written words. It plays a crucial role in interpersonal interactions and can complement or even override verbal communication. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. The survey emphasizes their applications to BL generation and recognition. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH), and we have conducted an analysis and established the connections among these four BL for the first time. Their generation and recognition often involve multi-modal approaches. Benchmark datasets for BL research are well collected and organized, along with the evaluation of SOTA methods on these datasets. The survey highlights challenges such as limited labeled data, multi-modal learning, and the need for domain adaptation to generalize models to unseen speakers or languages. Future research directions are presented, including exploring self-supervised learning techniques, integrating contextual information from other modalities, and exploiting large-scale pre-trained multi-modal models. In summary, this survey paper provides a comprehensive understanding of deep multi-modal learning for various BL generations and recognitions for the first time. By analyzing advancements, challenges, and future directions, it serves as a valuable resource for researchers and practitioners in advancing this field. n addition, we maintain a continuously updated paper list for deep multi-modal learning for BL recognition and generation: https://github.com/wentaoL86/awesome-body-language

A Survey of Applications and Human Motion Recognition with Microsoft Kinect

Microsoft Kinect, a low-cost motion sensing device, enables users to interact with computers or game consoles naturally through gestures and spoken commands without any other peripheral equipment. As such, it has commanded intense interests in research and development on the Kinect technology. In this paper, we present, a comprehensive survey on Kinect applications, and the latest research and development on motion recognition using data captured by the Kinect sensor. On the applications front, we review the applications of the Kinect technology in a variety of areas, including healthcare, education and performing arts, robotics, sign language recognition, retail services, workplace safety training, as well as 3D reconstructions. On the technology front, we provide an overview of the main features of both versions of the Kinect sensor together with the depth sensing technologies used, and review literatures on human motion recognition techniques used in Kinect applications. We provide a classification of motion recognition techniques to highlight the different approaches used in human motion recognition. Furthermore, we compile a list of publicly available Kinect datasets. These datasets are valuable resources for researchers to investigate better methods for human motion recognition and lower-level computer vision tasks such as segmentation, object detection and human pose estimation

Scoring sign language vitality: Adapting a spoken language survey to target the endangerment factors affecting sign languages

This article explores factors affecting the endangerment levels of sign languages, and how these levels were assessed through an international collaboration using a systematic scoring scheme. This included adapting UNESCO’s Linguistic Vitality and Diversity survey and developing a system for determining endangerment levels based on the responses. The survey needed to be carefully revised because even though many spoken language procedures can be also used for sign languages, there are additional challenges and characteristics that uniquely affect sign language communities. In this paper, we present vitality scores for 15 languages, including both national and village sign languages, and the major factors threatening their vitality. The methodology of scoring based on averages is innovative, as is the workflow between the questionnaire respondents and scoring committee. Such methodological innovations can also be useful for spoken languages. In the future, the approach taken in this study might contribute to developing best practice models for promoting sign language vitality and compile diachronic data to monitor changes in endangerment status. The findings can also inform policy work to bring about legal recognition, greater communication access, and the protection of deaf signers’ linguistic and cultural identity