Inclusive AR-games for Education of Deaf Children: Challenges and Opportunities

Game-based learning has had a rapid development in the 21st century, attracting an increasing audience. However, inclusion of all is still not a reality in society, with accessibility for deaf and hard of hearing children as a remaining challenge. To be excluded from learning due to communication barriers can have severe consequences for further studies and work. Based on previous research Augmented Reality (AR) games can be joyful learning tools that include activities with different sign languages, but AR based learning games for deaf and hard of hearing lack research. This paper aims to present opportunities and challenges of designing inclusive AR games for education of deaf children. Methods involved conducting a scoping review of previous studies about AR for deaf people. Experts were involved as co-authors for in-depth understanding of sign languages and challenges for deaf people. A set of AR input and output techniques were analysed for appropriateness, and various AR based game mechanics were compared. Results indicate that inclusive AR gameplay for deaf people could be built on AR based image and object tracking, complemented with sign recognition. These technologies provide input from the user and the real-world environment typically via the camera to the app. Scene tracking and GPS can be used for location-based game mechanics. Output to the user can be done via local signed videos ideally, but also with images and animations. Moreover, a civic intelligence approach can be applied to overcome many of the challenges that have been identified in five dimensions for inclusion of deaf people i.e., cultural, educational, psycho-social, semantic, and multimodal. The input from trusted, educated signers and teachers can enable the connection between real world objects and signed videos to provide explanations of concepts. The conclusion is that the development of an inclusive, multi-language AR game for deaf people needs to be carried out as an international collaboration, addressing all five dimensions

Inclusive Communication with Augmented Reality for deaf and hard of hearing

UIDB/05021/2020 UIDP/05021/2020Deafness is an often undervalued but increasing problem amongst world’s population. Besides the difficulty in hearing sounds, it involves many cognitive and emotional issues, like learning difficulties, isolation and disempowerment. In this paper, designing artefacts for deaf people is analyzed from two different perspectives: technology and communication science. Regarding technical issues, research shows how Augmented Reality can be applied using screen interfaces or smart glasses translating sounds into visual stimuli; from the communication side, the focus is on storytelling and how it can be combined with the technology referred to engage people, enhance learning activity and create a community. By looking into these two aspects, the suggested approach is to merge them in a conceptual creative project that can be appealing and useful to the public, through the use of interactive storytelling, while also using the visual benefits of an immersive Augmented Reality experience.publishersversionpublishe

Assistive technologies for severe and profound hearing loss: beyond hearing aids and implants

Assistive technologies offer capabilities that were previously inaccessible to individuals with severe and profound hearing loss who have no or limited access to hearing aids and implants. This literature review aims to explore existing assistive technologies and identify what still needs to be done. It is found that there is a lack of focus on the overall objectives of assistive technologies. In addition, several other issues are identified i.e. only a very small number of assistive technologies developed within a research context have led to commercial devices, there is a predisposition to use the latest expensive technologies and a tendency to avoid designing products universally. Finally, the further development of plug-ins that translate the text content of a website to various sign languages is needed to make information on the internet more accessible

Keskusteluavustimen kehittäminen kuulovammaisia varten automaattista puheentunnistusta käyttäen

Understanding and participating in conversations has been reported as one of the biggest challenges hearing impaired people face in their daily lives. These communication problems have been shown to have wide-ranging negative consequences, affecting their quality of life and the opportunities available to them in education and employment. A conversational assistance application was investigated to alleviate these problems. The application uses automatic speech recognition technology to provide real-time speech-to-text transcriptions to the user, with the goal of helping deaf and hard of hearing persons in conversational situations. To validate the method and investigate its usefulness, a prototype application was developed for testing purposes using open-source software. A user test was designed and performed with test participants representing the target user group. The results indicate that the Conversation Assistant method is valid, meaning it can help the hearing impaired to follow and participate in conversational situations. Speech recognition accuracy, especially in noisy environments, was identified as the primary target for further development for increased usefulness of the application. Conversely, recognition speed was deemed to be sufficient and already surpass the transcription speed of human transcribers.Keskustelupuheen ymmärtäminen ja keskusteluihin osallistuminen on raportoitu yhdeksi suurimmista haasteista, joita kuulovammaiset kohtaavat jokapäiväisessä elämässään. Näillä viestintäongelmilla on osoitettu olevan laaja-alaisia negatiivisia vaikutuksia, jotka heijastuvat elämänlaatuun ja heikentävät kuulovammaisten yhdenvertaisia osallistumismahdollisuuksia opiskeluun ja työelämään. Työssä kehitettiin ja arvioitiin apusovellusta keskustelupuheen ymmärtämisen ja keskusteluihin osallistumisen helpottamiseksi. Sovellus käyttää automaattista puheentunnistusta reaaliaikaiseen puheen tekstittämiseen kuuroja ja huonokuuloisia varten. Menetelmän toimivuuden vahvistamiseksi ja sen hyödyllisyyden tutkimiseksi siitä kehitettiin prototyyppisovellus käyttäjätestausta varten avointa lähdekoodia hyödyntäen. Testaamista varten suunniteltiin ja toteutettiin käyttäjäkoe sovelluksen kohderyhmää edustavilla koekäyttäjillä. Saadut tulokset viittaavat siihen, että työssä esitetty Keskusteluavustin on toimiva ja hyödyllinen apuväline huonokuuloisille ja kuuroille. Puheentunnistustarkkuus erityisesti meluisissa olosuhteissa osoittautui ensisijaiseksi kehityskohteeksi apusovelluksen hyödyllisyyden lisäämiseksi. Puheentunnistuksen nopeus arvioitiin puolestaan jo riittävän nopeaksi, ylittäen selkeästi kirjoitustulkkien kirjoitusnopeuden

An Assistive Technology Framework for Communication with Hearing Impaired Persons

This paper presents a novel assistive technology framework which provides an interface to support communication between hearing impaired person and ordinary person over the mobile phone. It converts the ordinary person's voice to text and afterward text to tactile feedback at the hearing impaired person’s end. The Morse Code tactile feedback have been identified as the most appropriate method for providing the tactile feedback at the hearing impaired person’s end, since it is a standard code which helps persons with impairments. The work addresses the challenge of using a set of Morse Code shorthand vibration patterns to translate the whole text message to tactile feedback to provide a simple, efficient and synchronous communication, rather than vibrating each and every character in text usingMorse Code characters. The user evaluation found that, most hearing impaired persons’ preferred method of conversation is the Morse Code shorthand forms with two or three character length rather than reading the entire text message. Due to less perspicuity of a hearing impaired person’s voice, the study comes up with the conversion of the hearing impaired persons’ voice to text and sends it to the ordinary person synchronously as a voice reply. The results of the evaluation experiment shows that the assistive technology framework facilitates by improving the quality of communication of hearing impaired persons over a mobile device

AR Comic Chat

Live speech transcription and captioning are important for the accessibility of deaf and hard of hearing individuals, especially in situations with no visible ASL translators. If live captioning is available at all, it is typically rendered in the style of closed captions on a display such as a phone screen or TV and away from the real conversation. This can potentially divide the focus of the viewer and detract from the experience. This paper proposes an investigation into an alternative, Augmented Reality driven approach to the display of these captions, using deep neural networks to compute, track and associate deep visual and speech descriptors in order to maintain captions as speech bubbles above the speaker

Landscape of sign language research based on smartphone apps: coherent literature analysis, motivations, open challenges, recommendations and future directions for app assessment

Numerous nations have prioritised the inclusion of citizens with disabilities, such as hearing loss, in all aspects of social life. Sign language is used by this population, yet they still have trouble communicating with others. Many sign language apps are being created to help bridge the communication gap as a result of technology advances enabled by the widespread use of smartphones. These apps are widely used because they are accessible and inexpensive. The services and capabilities they offer and the quality of their content, however, differ greatly. Evaluation of the quality of the content provided by these applications is necessary if they are to have any kind of real effect. A thorough evaluation like this will inspire developers to work hard on new apps, which will lead to improved software development and experience overall. This research used a systematic literature review (SLR) method, which is recognised in gaining a broad understanding of the study whilst offer- ing additional information for future investigations. SLR was adopted in this research for smartphone-based sign language apps to understand the area and main discussion aspects utilised in the assessment. These studies were reviewed on the basis of related work analysis, main issues, discussions and methodological aspects. Results revealed that the evaluation of sign language mobile apps is scarce. Thus, we proposed a future direction for the quality assessment of these apps. The findings will benefit normal-hearing and hearing-impaired users and open up a new area where researchers and developers could work together on sign language mobile apps. The results will help hearing and non-hearing users and will pave the way for future collaboration between academicians and app developers in the field of sign language technology