Inclusive Communication with Augmented Reality for deaf and hard of hearing

UIDB/05021/2020 UIDP/05021/2020Deafness is an often undervalued but increasing problem amongst world’s population. Besides the difficulty in hearing sounds, it involves many cognitive and emotional issues, like learning difficulties, isolation and disempowerment. In this paper, designing artefacts for deaf people is analyzed from two different perspectives: technology and communication science. Regarding technical issues, research shows how Augmented Reality can be applied using screen interfaces or smart glasses translating sounds into visual stimuli; from the communication side, the focus is on storytelling and how it can be combined with the technology referred to engage people, enhance learning activity and create a community. By looking into these two aspects, the suggested approach is to merge them in a conceptual creative project that can be appealing and useful to the public, through the use of interactive storytelling, while also using the visual benefits of an immersive Augmented Reality experience.publishersversionpublishe

Inclusive AR-games for Education of Deaf Children: Challenges and Opportunities

Game-based learning has had a rapid development in the 21st century, attracting an increasing audience. However, inclusion of all is still not a reality in society, with accessibility for deaf and hard of hearing children as a remaining challenge. To be excluded from learning due to communication barriers can have severe consequences for further studies and work. Based on previous research Augmented Reality (AR) games can be joyful learning tools that include activities with different sign languages, but AR based learning games for deaf and hard of hearing lack research. This paper aims to present opportunities and challenges of designing inclusive AR games for education of deaf children. Methods involved conducting a scoping review of previous studies about AR for deaf people. Experts were involved as co-authors for in-depth understanding of sign languages and challenges for deaf people. A set of AR input and output techniques were analysed for appropriateness, and various AR based game mechanics were compared. Results indicate that inclusive AR gameplay for deaf people could be built on AR based image and object tracking, complemented with sign recognition. These technologies provide input from the user and the real-world environment typically via the camera to the app. Scene tracking and GPS can be used for location-based game mechanics. Output to the user can be done via local signed videos ideally, but also with images and animations. Moreover, a civic intelligence approach can be applied to overcome many of the challenges that have been identified in five dimensions for inclusion of deaf people i.e., cultural, educational, psycho-social, semantic, and multimodal. The input from trusted, educated signers and teachers can enable the connection between real world objects and signed videos to provide explanations of concepts. The conclusion is that the development of an inclusive, multi-language AR game for deaf people needs to be carried out as an international collaboration, addressing all five dimensions

DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

There is an undeniable communication barrier between deaf people and people with normal hearing ability. Although innovations in sign language translation technology aim to tear down this communication barrier, the majority of existing sign language translation systems are either intrusive or constrained by resolution or ambient lighting conditions. Moreover, these existing systems can only perform single-sign ASL translation rather than sentence-level translation, making them much less useful in daily-life communication scenarios. In this work, we fill this critical gap by presenting DeepASL, a transformative deep learning-based sign language translation technology that enables ubiquitous and non-intrusive American Sign Language (ASL) translation at both word and sentence levels. DeepASL uses infrared light as its sensing mechanism to non-intrusively capture the ASL signs. It incorporates a novel hierarchical bidirectional deep recurrent neural network (HB-RNN) and a probabilistic framework based on Connectionist Temporal Classification (CTC) for word-level and sentence-level ASL translation respectively. To evaluate its performance, we have collected 7,306 samples from 11 participants, covering 56 commonly used ASL words and 100 ASL sentences. DeepASL achieves an average 94.5% word-level translation accuracy and an average 8.2% word error rate on translating unseen ASL sentences. Given its promising performance, we believe DeepASL represents a significant step towards breaking the communication barrier between deaf people and hearing majority, and thus has the significant potential to fundamentally change deaf people's lives

An Assistive Technology Framework for Communication with Hearing Impaired Persons

This paper presents a novel assistive technology framework which provides an interface to support communication between hearing impaired person and ordinary person over the mobile phone. It converts the ordinary person's voice to text and afterward text to tactile feedback at the hearing impaired person’s end. The Morse Code tactile feedback have been identified as the most appropriate method for providing the tactile feedback at the hearing impaired person’s end, since it is a standard code which helps persons with impairments. The work addresses the challenge of using a set of Morse Code shorthand vibration patterns to translate the whole text message to tactile feedback to provide a simple, efficient and synchronous communication, rather than vibrating each and every character in text usingMorse Code characters. The user evaluation found that, most hearing impaired persons’ preferred method of conversation is the Morse Code shorthand forms with two or three character length rather than reading the entire text message. Due to less perspicuity of a hearing impaired person’s voice, the study comes up with the conversion of the hearing impaired persons’ voice to text and sends it to the ordinary person synchronously as a voice reply. The results of the evaluation experiment shows that the assistive technology framework facilitates by improving the quality of communication of hearing impaired persons over a mobile device

Augmented reality to memorize Al-Quran for hearing impaired students: a preliminary analysis

Quran learning among the hearing impaired has not been given much attention and exposure. This is due to the difference in the learning method among the hearing impaired compared to the non-hearing impaired. In lieu with this, a preliminary study has been conducted by using a qualitative approach. An interview was conducted with five experienced teachers to identify hearing impaired students’ behaviours and the method used to learn Al-Quran. Meanwhile, an observation with the hearing impaired students was also conducted. The findings show that the Tahfiz Akhyar method is used to memorize the surahs of Al-Quran by orderly arranging the verses using numbers. The memorization technique is based on the teaching of Prophet Muhammad when He taught the companions to learn Al-Quran. However, the result shows that, there are still errors in arranging the verses of the surahs in the correct order using the existing teaching aid. Therefore, these findings will be used as a guideline to develop the teaching aid application integrated with the Augmented Reality technology to help hearing impaired students to memorize the surahs of Al-Quran.Keywords: Augmented Reality, Quran learning, Hearing Impaired, and Memorizatio

Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing People

Co-located collaborative shared augmented reality (CS-AR) environments have gained considerable research attention, mainly focusing on design, implementation, accuracy, and usability. Yet, a gap persists in our understanding regarding the accessibility and inclusivity of such environments for diverse user groups, such as deaf and Hard of Hearing (DHH) people. To investigate this domain, we used Urban Legends, a multiplayer game in a co-located CS-AR setting. We conducted a user study followed by one-on-one interviews with 17 DHH participants. Our findings revealed the usage of multimodal communication (verbal and non-verbal) before and during the game, impacting the amount of collaboration among participants and how their coordination with AR components, their surroundings, and other participants improved throughout the rounds. We utilize our data to propose design enhancements, including onscreen visuals and speech-to-text transcription, centered on participant perspectives and our analysis

Mobile app communication aid for Cypriot deaf people

© 2019, Emerald Publishing Limited. Purpose: People with severe or profound hearing loss face daily communication problems mainly due to the language barrier between themselves and the hearing community. Their hearing deficiency, as well as their use of sign language, often makes it difficult for them to use and understand spoken language. Cyprus is amongst the top 5 European countries with a relatively high proportion of registered deaf people (0.12 per cent of the population: GUL, 2010). However, lack of technological and financial support to the Deaf Community of Cyprus leaves the Cypriot deaf people unsupported and marginalised. The paper aims to discuss this issue. Design/methodology/approach: This study implemented user-centred design methods to explore the communication needs and requirements of Cypriot deaf people and develop a functional prototype of a mobile app to help them to communicate more effectively with hearing people. A total of 76 deaf adults were involved in various stages of the research. This paper presents the participatory design activities (N=8) and results of usability testing (N=8). Findings: The study found that users were completely satisfied with the mobile app and, in particular, they liked the use of Cypriot Sign Language (CSL) videos of a real person interpreting hearing people’s speech in real time and the custom onscreen keyboard to allow faster selection of text input. Originality/value: Despite advances in communication aid technologies, there is currently no technology available that supports CSL or real-time speech to sign language conversion for the deaf people of Cyprus

Quiet Interaction: Designing an Accessible Home Environment for Deaf and Hard of Hearing (DHH) Individuals through AR, AI, and IoT Technologies

As technology rapidly evolves, voice-command-based smart assistants are becoming integral to our daily lives. However, this advancement overlooks the needs of the Deaf and Hard of Hearing (DHH) community, creating a technological gap in current systems. To address this technological oversight, this study develops a Mixed-Reality (MR) application that integrates Augmented Reality (AR), Artificial Intelligence (AI), and the Internet of Things (IoT) technologies to fill the gaps in safety, communication, and accessibility for DHH individuals at home. By employing the User-Centric design methodology, this study begins with a needs assessment through a literature review and online survey to understand the unique challenges and preferences of the DHH community. The key contribution of this study lies in its innovative integration of technologies within a Mixed-Reality (MR) framework, with the goal of creating a more inclusive and accessible home environment for the DHH community

On Equal Footing? A study on how adult members of the Deaf and hard of hearing community (in the United States) perceive that the procurement, use, or absence of assistive technology has influenced their lives

Technology has changed each of our lives, that is certain. Findings from previous studies about assistive technology plus personal interviews and a survey were used for this interpretive phenomenological study. Interviews were conducted with people associated with the researcher’s work. Existing literature regarding assistive technology produced themes revolving around systemic policies & practices, various types of assistive technology, the users of assistive technology themselves (situation, age, circumstances, etc.), and the impact of assistive technology. The last theme is the focus of this qualitative study: how adult members of the Deaf and hard of hearing community’s lives have been affected by assistive technology. The devices and services mentioned by the participants are explained and/or illustrated. Comments made by the interviewees are divided into two arenas: the effects of using assistive technology and the effects of not having it. Assistive technology has helped to create a much better life for the interviewees overall, in their opinion. Each expressed that they feel a part of the larger society, can be successful at work, can know and be known, and look forward to what the future holds, albeit with some concerns. This study shows that although technology has improved their lives, equal footing for them (& others) in a hearing world may not have arrived yet