Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages.  This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems

Tech for Understanding: An Introduction to Assistive and Instructional Technology in the Classroom

This paper examines the different types of assistive and instructional technology available to students who are classified with one or more of the thirteen disabilities outlined in the Individuals with Disabilities Education Act (referred to as, IDEA). While the roles of assistive and instructional technology are different, there are many instances where their uses may overlap. Thus, while these two categories will be discussed separately, it should be noted that some information may be applied to each category and more than one piece of technology. The purpose of this paper is to provide an introduction to the world of assistive and instructional technology for those who may be new to its concepts, particularly parents who have recently learned that their child may benefit from extra assistance and future educators who are interested in learning more about the devices they will be using to reach their students. Each of the thirteen disabilities will be discussed briefly, and then each disability will be assigned several types of assistive and instructional technology that serve it well. This will by no means be an exhaustive list of all types of technology available to teachers, parents, and students. However, it will attempt to provide a varied glimpse at some of the options that are available and how they may help children who are struggling to access the curriculum

Ambient Assisted Living Technology

Ambient assisted living technology, known as ambient welfare technology in Denmark, promises to play a prominent role in the future of home health care. This project, sponsored by the Danish Association of the Blind, investigated the social and political context of ambient welfare technology and the primary concerns of stakeholder’s in its development. The results from our research were used to develop criteria for evaluating assistive devices and make recommendations for future development of these technologies

Augmented Reality

Augmented Reality (AR) is a natural development from virtual reality (VR), which was developed several decades earlier. AR complements VR in many ways. Due to the advantages of the user being able to see both the real and virtual objects simultaneously, AR is far more intuitive, but it's not completely detached from human factors and other restrictions. AR doesn't consume as much time and effort in the applications because it's not required to construct the entire virtual scene and the environment. In this book, several new and emerging application areas of AR are presented and divided into three sections. The first section contains applications in outdoor and mobile AR, such as construction, restoration, security and surveillance. The second section deals with AR in medical, biological, and human bodies. The third and final section contains a number of new and useful applications in daily living and learning

Ambient awareness on a sidewalk for visually impaired

Safe navigation by avoiding obstacles is vital for visually impaired while walking on a sidewalk. There are both static and dynamic obstacles to avoid. Detection, monitoring, and estimating the threat posed by obstacles remain challenging. Also, it is imperative that the design of the system must be energy efficient and low cost. An additional challenge in designing an interactive system capable of providing useful feedback is to minimize users\u27 cognitive load. We started the development of the prototype system through classifying obstacles and providing feedback. To overcome the limitations of the classification-based system, we adopted the image annotation framework in describing the scene, which may or may not include the obstacles. Both solutions partially solved the safe navigation but were found to be ineffective in providing meaningful feedback and issues with the diurnal cycle. To address such limitations, we introduce the notion of free-path and threat level imposed by the static or dynamic obstacles. This solution reduced the overhead of obstacle detection and helped in designing meaningful feedback. Affording users a natural conversation through an interactive dialog enabled interface was found to promote safer navigation. In this dissertation, we modeled the free-path and threat level using a reinforcement learning (RL) framework.We built the RL model in the Gazebo robot simulation environment and implanted that in a handheld device. A natural conversation model was created using data collected through a Wizard of OZ approach. The RL model and conversational agent model together resulted in the handheld assistive device called Augmented Guiding Torch (AGT). The AGT provides improved mobility over white cane by providing ambient awareness through natural conversation. It can inform the visually impaired about the obstacles which are helpful to be warned about ahead of time, e.g., construction site, scooter, crowd, car, bike, or big hole. Using the RL framework, the robot avoided over 95% obstacles. The visually impaired avoided over 85% obstacles with the help of AGT on a 500 feet U-shape sidewalk. Findings of this dissertation support the effectiveness of augmented guiding through RL for navigation and obstacle avoidance of visually impaired users

Examining Activities Concerning the Use of Words Encountered in Content Area Courses in Writing by Children with Hearing Loss

Improvement of vocabulary constitutes an important part of the literacy practices. Children with hearing loss experience difficulties in using newly encountered words in verbal and written languages due to the delay they experience in linguistic skills. The objective of the present study is to investigate the course of applying a balanced literacy program, that allows the words encountered by the children with hearing loss during informative lessons, to be used in their written expressions. The study has been designed as case study.  The participants of the study consisted of 6 students with hearing loss educating at the 4th class of elementary school, a class teacher, and the researcher. The data source of the study included the field observations, records of the validity meetings, lesson program, and the materials obtained during the course of practice; additionally the diary of the researcher was reported, and the students’ yields and the audiological documents were investigated. The study results have been discussed in light of the activities which improve vocabulary by using picture, by using text and by using words, sentences, and the techniques used during these activities. Keywords: child with hearing loss, vocabulary, writing skills, balanced literacy, literacy activitie

A study of applications of microcomputer technology in special education in western Massachusetts schools.

The purpose of this study is to survey microcomputer applications in special education in Western Massachusetts Schools and, in particular, to assess the extent to which special education is moving beyond drill and practice software with special needs students. Data were collected from 185 special education teachers by a questionnaire and follow-up interviews from eleven special education teachers in Western Massachusetts. Results showed that computers and software are generally integrated in special education teachers\u27 curricula. They used the microcomputer as a compensatory tool to sharpen students\u27 mathematics skills, language arts and reading comprehension. Some special education teachers also used computers for language assessment, speech training, eye-hand coordination and communication. Apple computers were the most popular brand used in this study. Adaptive devices such as firmware cards, switches, and speech synthesizers were used to help special needs students access computers. Computer-assisted instruction, word processing and games were the most popular software used. Students worked on computers generally alone, or in a small group, or in combination; the amount of supervision required depended upon students\u27 functioning level and physical limitations. Most special education teachers did not teach and computer language; only a few teachers explored Logo or BASIC with their students. Special education teachers realized that the computer is a good tool to motivate students and to increase self-esteem and attention; they received some inservice training on computer uses, but complained that it was not enough to help their students. Factors making it difficult for special education teachers to use computers were: lack of appropriate software, teachers being behind the trend, not enough class time to use computers, and perceptions of computers as dehumanizing. The study concludes with recommendations for increasing special education teachers\u27 computer training via input from hardware and software experts, and for requiring special education teachers to take introductory computer courses such as Logo, BASIC programming, authoring language systems and software evaluation. Also, it recommends that school administrations give financial and technical support for such training in order to use microcomputers and related devices more effectively

How a Diverse Research Ecosystem Has Generated New Rehabilitation Technologies: Review of NIDILRR’s Rehabilitation Engineering Research Centers

Over 50 million United States citizens (1 in 6 people in the US) have a developmental, acquired, or degenerative disability. The average US citizen can expect to live 20% of his or her life with a disability. Rehabilitation technologies play a major role in improving the quality of life for people with a disability, yet widespread and highly challenging needs remain. Within the US, a major effort aimed at the creation and evaluation of rehabilitation technology has been the Rehabilitation Engineering Research Centers (RERCs) sponsored by the National Institute on Disability, Independent Living, and Rehabilitation Research. As envisioned at their conception by a panel of the National Academy of Science in 1970, these centers were intended to take a “total approach to rehabilitation”, combining medicine, engineering, and related science, to improve the quality of life of individuals with a disability. Here, we review the scope, achievements, and ongoing projects of an unbiased sample of 19 currently active or recently terminated RERCs. Specifically, for each center, we briefly explain the needs it targets, summarize key historical advances, identify emerging innovations, and consider future directions. Our assessment from this review is that the RERC program indeed involves a multidisciplinary approach, with 36 professional fields involved, although 70% of research and development staff are in engineering fields, 23% in clinical fields, and only 7% in basic science fields; significantly, 11% of the professional staff have a disability related to their research. We observe that the RERC program has substantially diversified the scope of its work since the 1970’s, addressing more types of disabilities using more technologies, and, in particular, often now focusing on information technologies. RERC work also now often views users as integrated into an interdependent society through technologies that both people with and without disabilities co-use (such as the internet, wireless communication, and architecture). In addition, RERC research has evolved to view users as able at improving outcomes through learning, exercise, and plasticity (rather than being static), which can be optimally timed. We provide examples of rehabilitation technology innovation produced by the RERCs that illustrate this increasingly diversifying scope and evolving perspective. We conclude by discussing growth opportunities and possible future directions of the RERC program