Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages.  This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems

Automatic recognition of Arabic alphabets sign language using deep learning

Technological advancements are helping people with special needs overcome many communications’ obstacles. Deep learning and computer vision models are innovative leaps nowadays in facilitating unprecedented tasks in human interactions. The Arabic language is always a rich research area. In this paper, different deep learning models were applied to test the accuracy and efficiency obtained in automatic Arabic sign language recognition. In this paper, we provide a novel framework for the automatic detection of Arabic sign language, based on transfer learning applied on popular deep learning models for image processing. Specifically, by training AlexNet, VGGNet and GoogleNet/Inception models, along with testing the efficiency of shallow learning approaches based on support vector machine (SVM) and nearest neighbors algorithms as baselines. As a result, we propose a novel approach for the automatic recognition of Arabic alphabets in sign language based on VGGNet architecture which outperformed the other trained models. The proposed model is set to present promising results in recognizing Arabic sign language with an accuracy score of 97%. The suggested models are tested against a recent fully-labeled dataset of Arabic sign language images. The dataset contains 54,049 images, which is considered the first large and comprehensive real dataset of Arabic sign language to the furthest we know

Use of Key Points and Transfer Learning Techniques in Recognition of Handedness Indian Sign Language

The most expressive way of communication for individuals who have trouble speaking or hearing is sign language. Normal people are unable to comprehend sign language. As a result, communication barriers are put up. Majority of people are right-handed. Statistics say that, an average population of left-handed person in the world is about 10%, where they use left hand as their dominating hand. In case of hand written text recognition, if the text is written by left-handed or right-handed person, then there would not be any problem in recognition neither for human and nor for computer. But same thing is not true for sign language and its detection using computer. When the detection is performed using computer vision and if it falls into the category of detection by appearance, then it might not detect correctly. In machine and deep learning, if the model is trained using just one dominating hand, let’s say right hand, then the predictions can go wrong if same sign is performed by left-handed person. This paper addresses this issue. It takes into account the signs performed by any type of signer: left-handed, right-handed or ambidexter. In proposed work is on Indian Sign Language (ISL). Two models are trained: Model I, is trained on one dominating hand and Model II, is trained on both the hands. Model II gives correct predictions regardless of any type of signer. It recognizes alphabets and numbers in ISL. We used the concept of Key points and Transfer Learning techniques for implementation. Using this approach, models get trained quickly and we could achieve validation accuracy of 99%

Emotional engineering of artificial representations of sign languages

The fascination and challenge of making an appropriate digital representation of sign language for a highly specialised and culturally rich community such as the Deaf, has brought about the development and production of several digital representations of sign language (DRSL). These range from pictorial depictions of sign language, filmed video recordings to animated avatars (virtual humans). However, issues relating to translating and representing sign language in the digital-domain and the effectiveness of various approaches, has divided the opinion of the target audience. As a result there is still no universally accepted digital representation of sign language. For systems to reach their full potential, researchers have postulated that further investigation is needed into the interaction and representational issues associated with the mapping of sign language into the digital domain. This dissertation contributes a novel approach that investigates the comparative effectiveness of digital representations of sign language within different information delivery contexts. The empirical studies presented have supported the characterisation of the prescribed properties of DRSL's that make it an effective communication system, which when defined by the Deaf community, was often referred to as "emotion". This has led to and supported the developed of the proposed design methodology for the "Emotional Engineering of Artificial Sign Languages", which forms the main contribution of this thesis

Assistive technologies for severe and profound hearing loss: beyond hearing aids and implants

Assistive technologies offer capabilities that were previously inaccessible to individuals with severe and profound hearing loss who have no or limited access to hearing aids and implants. This literature review aims to explore existing assistive technologies and identify what still needs to be done. It is found that there is a lack of focus on the overall objectives of assistive technologies. In addition, several other issues are identified i.e. only a very small number of assistive technologies developed within a research context have led to commercial devices, there is a predisposition to use the latest expensive technologies and a tendency to avoid designing products universally. Finally, the further development of plug-ins that translate the text content of a website to various sign languages is needed to make information on the internet more accessible

Evaluating the sign language phonology of sight words used to support deaf /hard of hearing students\u27 literacy development in Idioma de Señas de Nicaragua

Over the past 30-years linguists have been witnessing the birth and evolution of a language, Idioma de Señas de Nicaragua, in Nicaragua, and have initiated and documented to date the syntax and grammar of this new language. The purpose of this naturalistic comparative exploratory field study was to evaluate preprimer and primer Dolch sight words and sign language frequency and phonology—location, handshape, and movement—used to support deaf/hard of hearing students’ literacy development in Idioma de Señas de Nicaragua compared to American Sign Language. The research focused on the word and sign frequency and phonology or individual components of a Nicaraguan sign that gives it meaning—handshape, location, and movement. Statistically significantly differences in the direction of greater sign to preprimer and primer Dolch Word chi-square frequency comparisons for American Sign Language and Idioma de Señas de Nicaragua were found. Furthermore, based on the moderate to substantial Pearson product-moment correlations and coefficient of determination areas of shared variance observed between Dolch preprimer American Sign Language signed phonemes for handshapes, locations, and movements and Dolch preprimer Idioma de Señas de Nicaragua signed phonemes for handshapes, locations, and movements it may be assumed that children using Idioma de Señas de Nicaragua have between 59% to 97% of the phonemic means of expressing themselves as do children using American Sign Language. For Dolch primer American Sign Language signed phonemes for handshapes, locations, and movements and Dolch primer Idioma de Señas de Nicaragua signed phonemes for handshapes, locations, and movements it may be assumed that children using Idioma de Señas de Nicaragua have between 60% to 94% of the phonemic means of expressing themselves as do children using American Sign Language

Communication Resources for Deaf or Hard of Hearing Children in Mississippi: Parents’ Perspectives

A parent’s ability to communicate with their child through formative years may often be taken for granted, as the options for such communication seem intuitive and apparent. However, hearing parents of children with hearing loss must not only make a choice between several communication methods, but they must also navigate an environment where the methodologies are not clearly delineated. Blaiser and colleague provide succinct descriptions of the most common methods which can be chosen. These methods include listening and spoken language systems, manual-visual systems, and systems combining these two modalities (Blaiser & Bargen, 2018). This choice is often challenging because many factors impact the accessibility to and availability of each. Availability of communication resources can vary across geographic locations, and absence of access to certain services render some options moot. It has been reported that rural areas are especially lacking in such resources (Furno et al., 2020; Meadow-Orlans et al., 2003). Consequently, the purpose of this study is twofold: Primarily, it explores variables that may affect the communication choices of hearing parents for their deaf or hard of hearing child. Secondarily, it seeks to gain a better understanding of these choices, investigating why parents chose their communication method and exploring the choices they felt they had available. A mixed methods research design was employed to address the question: What factors contribute to the communication choices made by hearing parents of deaf and hard of hearing children in the state of Mississippi? Quantitative and qualitative analyses were performed on the data to reveal correlations between variables and themes in the decision-making processes of parents. The results indicated correlations between (a) parent age and child age, (b) parent proficiency in American Sign Language (ASL) and child proficiency in ASL, and (c) parent ratings of communicative support in recreational environments and community environments. Themes identified in the qualitative data were (a) general knowledge on hearing loss prior to the child’s diagnosis, (b) support systems, and (c) methods of communication used