COMPUTER ASSISTED COMMUNICATION FOR THE HEARING IMPAIRED FOR AN EMERGENCY ROOM SCENARIO

While there has been research on computerized communication facilities for those with hearing impairment, issues still remain. Current approaches utilize an avatar based approach which lacks the ability to adequately use facial expressions which are an integral aspect to the communication process in American Sign Language (ASL). Additionally, there is a lack of research into integrating a system to facilitate communication with the hearing impaired into a clinical environment, namely an emergency room admission scenario. This research aims to determine if an alternate approach of using videos created by humans in ASL can overcome the understandability barrier and still be usable in the communication process

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages.  This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems

Emotional engineering of artificial representations of sign languages

The fascination and challenge of making an appropriate digital representation of sign language for a highly specialised and culturally rich community such as the Deaf, has brought about the development and production of several digital representations of sign language (DRSL). These range from pictorial depictions of sign language, filmed video recordings to animated avatars (virtual humans). However, issues relating to translating and representing sign language in the digital-domain and the effectiveness of various approaches, has divided the opinion of the target audience. As a result there is still no universally accepted digital representation of sign language. For systems to reach their full potential, researchers have postulated that further investigation is needed into the interaction and representational issues associated with the mapping of sign language into the digital domain. This dissertation contributes a novel approach that investigates the comparative effectiveness of digital representations of sign language within different information delivery contexts. The empirical studies presented have supported the characterisation of the prescribed properties of DRSL's that make it an effective communication system, which when defined by the Deaf community, was often referred to as "emotion". This has led to and supported the developed of the proposed design methodology for the "Emotional Engineering of Artificial Sign Languages", which forms the main contribution of this thesis

Improving Accessibility of e-Learning Templates for Students with Disabilities

Because of the effective development of contemporary technologies, also during the recent COVID-19 pandemic, students with disabilities have been increasingly integrated into the society, work, and academic environment. Accordingly, the question of how to provide accessible graphical design of e-learning materials for various categories of students with disabilities and how to comply with international accessibility standards is becoming more and more prominent. The purpose of the study was to propose recommendations for the design of e-learning materials to achieve the preparation of effective study materials for students with hearing and visual loss, as well as autism spectrum disorders. According to their needs and requirements, recommendations for the graphic design of accessible e-learning materials have been designed, developed, implemented, and evaluated. The findings revealed a higher level of acceptance of adapted material for all experimental groups and can be helpful for teachers and other professionals educating students with disabilities

Guidelines for and evaluation of the design of technology-supported lessons to teach basic programming principles to deaf and hard of hearing learners: a case study of a school for the deaf

Deaf and Hard of Hearing (DHH) learners are part of a diverse population with unique learning challenges, strengths and needs. Learning material should be developed specifically for them to provide for their needs and capitalise on their strengths. These materials should include visual material and strategies as well as sign language. Furthermore, DHH learners have the same capacity for learning as hearing learners. However, in South Africa, DHH learners do not have adequate access to training in computer-related subjects, and therefore no material exists that has been developed specifically for DHH learners who want to learn a programming language. This research provides guidelines on the way technology-supported lessons can be designed to teach basic programming principles using the programming language Scratch, to DHH learners. Provision was made for the South African context where limited technology is available at most schools for DHH learners, but where most educators have access to Microsoft Office applications – specifically MS PowerPoint. Two goals were pursued. The primary goal of this research project was to determine the user experience (UX) of the participants (both learners and educators) during and after using and attending the technology-supported lessons. This was achieved through a case study. Four UX evaluation elements were evaluated in this project. They were: usability, accessibility, emotional user reaction, and hedonic aspects. Questionnaires, semi-structured interviews as well as participant-observation were used to determine the UX of participants. The UX evaluation provided sufficient evidence to claim that UX of participants was satisfactory, and therefore the guidelines that were developed to create technology-supported lessons to teach basic programming principles to DHH learners were appropriate. The secondary goal was to develop guidelines for the design of technology-supported lessons to teach programming to DHH learners, and to apply these guidelines to develop a high-fidelity, fully functional prototype – a set of technology-supported lessons. This was achieved through a prototype construction research strategy. The lessons consisted of two vocabulary lessons and one programming lesson. The words that were taught in the vocabulary lesson were either terms appearing in the interface of Scratch, or words needed in the explanation of programming principles and Scratch context. The programming lesson (a PowerPoint slide show) was a guide for the educator to present the content in a logical way, and not to leave out important information. It used multimedia techniques (colour, pictures, animation) to explain programming concepts, and to display the tasks to be completed to the learners, so that they could remember the sequence of the steps. Practical strategies have been included in the guidelines to address the learning challenges DHH experience in the following areas: Comprehension skills, application of knowledge and knowledge organisation, relational and individual-item orientations, metacognition, memory, distractibility. The guidelines referred to techniques and principles that can be followed to design the interface and navigation tools of a technology-supported lesson; enhance communication with DHH learners, and provide support for them to work independently; specify the educator’s role and attitude when facilitating or presenting programming lessons and to structure a programming lesson

A Mobile Deaf-to-hearing communication aid for medical diagnosis

>Magister Scientiae - MScMany South African Deaf people use their mobile phones for communication with SMSs yet they would prefer to converse in South African Sign Language. Deaf people with a capital `D' are different from deaf or hard of hearing as they primarily use sign language to communicate. This study explores how to design and evaluate a prototype that will allow a Deaf person using SASL to tell a hearing doctor how s/he is feeling and provide a way for the doctor to respond. A computer{based prototype was designed and evaluated with the Deaf people in a previous study. Results from the user trial of the computer{based mock{up indicated that Deaf users would like to see the prototype on a cell phone. Those user trial results, combined with our own user survey results conducted with Deaf people, are used as requirements. We built a prototype for a mobile phone browser by embedding SASL videos inside XHTML pages using Adobe Flash. The prototype asks medical questions using SASL videos. These questions are arranged in an organized way that helps in identifying a medical problem. The answers to the questions are then displayed in English and shown to the doctor on the phone. A content authoring tool was also designed and implemented. The content authoring tool is used for populating the prototype in a context free manner allowing for plug and play scenarios such as a doctor's office, Department of Home A airs or police station. A focus group consisting of Deaf people was conducted to help in the design and pilot trial of the system. A final user trial was conducted with more than thirty Deaf people and the results are presented and analyzed. Data is collected with questionnaires, semi-structured interviews and video recordings. The results indicate that most of the Deaf people found the system easy to learn, easy to navigate through, did not get lost and understood the sign language in the videos on the mobile phone. The hand gestures and facial expressions on the sign language videos were clear. Most of them indicated they would like to use the system for free, and that the system did not ask too many questions. Most of them were happy with the quality of the sign language videos on the mobile phone and would consider using the system in real life. Finally they felt their private information was safe while using the system. Many South African Deaf people use their mobile phones for communication with SMSs yet they would prefer to converse in South African Sign Language. Deaf people with a capital `D' are different from deaf or hard of hearing as they primarily use sign language to communicate. This study explores how to design and evaluate a prototype that will allow a Deaf person using SASL to tell a hearing doctor how s/he is feeling and provide a way for the doctor to respond. A computer{based prototype was designed and evaluated with the Deaf people in a previous study. Results from the user trial of the computer{based mock{up indicated that Deaf users would like to see the prototype on a cell phone. Those user trial results, combined with our own user survey results conducted with Deaf people, are used as requirements. We built a prototype for a mobile phone browser by embedding SASL videos inside XHTML pages using Adobe Flash. The prototype asks medical questions using SASL videos. These questions are arranged in an organized way that helps in identifying a medical problem. The answers to the questions are then displayed in English and shown to the doctor on the phone. A content authoring tool was also designed and implemented. The content authoring tool is used for populating the prototype in a context free manner allowing for plug and play scenarios such as a doctor's office, Department of Home A airs or police station. A focus group consisting of Deaf people was conducted to help in the design and pilot trial of the system. A final user trial was conducted with more than thirty Deaf people and the results are presented and analyzed. Data is collected with questionnaires, semi-structured interviews and video recordings. The results indicate that most of the Deaf people found the system easy to learn, easy to navigate through, did not get lost and understood the sign language in the videos on the mobile phone. The hand gestures and facial expressions on the sign language videos were clear. Most of them indicated they would like to use the system for free, and that the system did not ask too many questions. Most of them were happy with the quality of the sign language videos on the mobile phone and would consider using the system in real life. Finally they felt their private information was safe while using the system.South Afric