An exploration of the potential of Automatic Speech Recognition to assist and enable receptive communication in higher education

The potential use of Automatic Speech Recognition to assist receptive communication is explored. The opportunities and challenges that this technology presents students and staff to provide captioning of speech online or in classrooms for deaf or hard of hearing students and assist blind, visually impaired or dyslexic learners to read and search learning material more readily by augmenting synthetic speech with natural recorded real speech is also discussed and evaluated. The automatic provision of online lecture notes, synchronised with speech, enables staff and students to focus on learning and teaching issues, while also benefiting learners unable to attend the lecture or who find it difficult or impossible to take notes at the same time as listening, watching and thinking

Word Importance Modeling to Enhance Captions Generated by Automatic Speech Recognition for Deaf and Hard of Hearing Users

People who are deaf or hard-of-hearing (DHH) benefit from sign-language interpreting or live-captioning (with a human transcriptionist), to access spoken information. However, such services are not legally required, affordable, nor available in many settings, e.g., impromptu small-group meetings in the workplace or online video content that has not been professionally captioned. As Automatic Speech Recognition (ASR) systems improve in accuracy and speed, it is natural to investigate the use of these systems to assist DHH users in a variety of tasks. But, ASR systems are still not perfect, especially in realistic conversational settings, leading to the issue of trust and acceptance of these systems from the DHH community. To overcome these challenges, our work focuses on: (1) building metrics for accurately evaluating the quality of automatic captioning systems, and (2) designing interventions for improving the usability of captions for DHH users. The first part of this dissertation describes our research on methods for identifying words that are important for understanding the meaning of a conversational turn within transcripts of spoken dialogue. Such knowledge about the relative importance of words in spoken messages can be used in evaluating ASR systems (in part 2 of this dissertation) or creating new applications for DHH users of captioned video (in part 3 of this dissertation). We found that models which consider both the acoustic properties of spoken words as well as text-based features (e.g., pre-trained word embeddings) are more effective at predicting the semantic importance of a word than models that utilize only one of these types of features. The second part of this dissertation describes studies to understand DHH users\u27 perception of the quality of ASR-generated captions; the goal of this work was to validate the design of automatic metrics for evaluating captions in real-time applications for these users. Such a metric could facilitate comparison of various ASR systems, for determining the suitability of specific ASR systems for supporting communication for DHH users. We designed experimental studies to elicit feedback on the quality of captions from DHH users, and we developed and evaluated automatic metrics for predicting the usability of automatically generated captions for these users. We found that metrics that consider the importance of each word in a text are more effective at predicting the usability of imperfect text captions than the traditional Word Error Rate (WER) metric. The final part of this dissertation describes research on importance-based highlighting of words in captions, as a way to enhance the usability of captions for DHH users. Similar to highlighting in static texts (e.g., textbooks or electronic documents), highlighting in captions involves changing the appearance of some texts in caption to enable readers to attend to the most important bits of information quickly. Despite the known benefits of highlighting in static texts, research on the usefulness of highlighting in captions for DHH users is largely unexplored. For this reason, we conducted experimental studies with DHH participants to understand the benefits of importance-based highlighting in captions, and their preference on different design configurations for highlighting in captions. We found that DHH users subjectively preferred highlighting in captions, and they reported higher readability and understandability scores and lower task-load scores when viewing videos with captions containing highlighting compared to the videos without highlighting. Further, in partial contrast to recommendations in prior research on highlighting in static texts (which had not been based on experimental studies with DHH users), we found that DHH participants preferred boldface, word-level, non-repeating highlighting in captions

Improving Deaf Accessibility to Web-based Multimedia

Internet technologies have expanded rapidly over the past two decades, making information of all sorts more readily available. Not only are they more cost-effective than traditional media, these new media have contributed to quality and convenience. However, proliferation of video and audio media on the internet creates an inadvertent disadvantage for deaf Internet users. Despite technological and legislative milestones in recent decades in making television and movies more accessible, there has been little progress with online access. A major obstacle to providing captions for internet media is the high cost of captioning and transcribing services. To respond to this problem, a possible solution lies in automatic speech recognition (ASR). This research investigates possible solutions to Web accessibility through utilization of ASR technologies. It surveys previous studies that employ visualization and ASR to determine their effectiveness in the context of deaf accessibility. Since there was no existing literature indicating the area of greatest need, a preliminary study identified an application that would serve as a case study for applying and evaluating speech visualization technology. A total of 20 deaf and hard-of-hearing participants were interviewed via video phone and their responses in American Sign Language were transcribed to English. The most common theme was concern over a lack of accessibility for online news. The second study evaluated different presentation strategies for making online news videos more accessible. A total of 95 participants viewed four different caption styles. Each style was presented on different news stories with control for content level and delivery. In addition to pre-test and post-test questionnaires, both performance and preference measures were conducted. Results from the study offer emphatic support for the hypothesis that captioning the online videos makes the Internet more accessible to the deaf users. Furthermore, the findings lend strong evidence to the idea of utilizing automatic captions to make videos comprehensible to the deaf viewers at a fraction of the cost. The color-coded captions that used highlighting to reflect the accuracy ratings were found neither to be beneficial nor detrimental; however, when asked directly about the benefit of color-coding there was support for the concept. Further development and research will be necessary to find the appropriate solution

Accessibility at Film Festivals: Guidelines for Inclusive Subtitling

In today's media-dominated world, the imperative for accessibility has never been greater, and ensuring that audiovisual experiences cater to individuals with sensory disabilities has become a pressing concern. One of the key initiatives in this endeavour is inclusive subtitling (IS), a practice rooted in the broader contexts of subtitling for the deaf and hard of hearing (SDH/CC), audiovisual translation studies (AVTS), media accessibility studies (MAS), and the evolving field of Deaf studies (DS). This study aims to offer a comprehensive exploration of how inclusive subtitling contributes to fostering accessible and inclusive audiovisual experiences, with a particular focus on its implications within the unique environment of film festivals. To gain a holistic perspective of inclusive subtitling, it is essential to examine its lineage in relation to analogous practices, which is the focus of the first chapter. Inclusive subtitling is an extension of SDH/CC, designed for individuals with hearing impairments, and SDH/CC, in turn, is a nuanced variation of traditional subtitling extensively explored within the realm of AVTS. To encapsulate the diverse techniques and modalities aimed at making audiovisual content universally accessible, the study recognises the term "Audiovisual Accessibility" (AVA). The second chapter explores the interconnection of accessibility studies (AS), AVTS, and MAS, highlighting their symbiotic relationship and their role in framing inclusive subtitles within these fields. These interconnections are pivotal in shaping a framework for the practice of inclusive subtitling, enabling a comprehensive examination of its applicability and research implications. The third chapter delves into Deaf studies and the evolution of Deafhood, which hinges on the history and culture of Deaf individuals. This chapter elucidates the distinction between ‘deafness’ as a medical construct and ‘Deafhood’ as a cultural identity, crucial to the understanding of audiovisual accessibility and its intersection with the Deaf community's perspectives. In the fourth chapter, the focus turns to the exploration of film festivals, with a specific emphasis on the crucial role of subtitles in enhancing accessibility, particularly when films are presented in their original languages. The chapter marks a critical point, highlighting the inherent connection between subtitles and the immersive nature of film festivals that aspire to promote inclusivity in the cinematic experience. The emphasis on inclusivity extends to the evolution of film festivals, giving rise to more advanced forms, including accessible film festivals and Deaf film festivals. At the core of the chapter is a thorough examination of the corpus, specifically, the SDH/CC of films spanning the editions from 2020 to 2023 of two highly significant film festivals, namely BFI Flare and the London Film Festival. The corpus serves as the foundation upon which my research unfolds, providing a nuanced understanding of the role subtitles play in film festival contexts. The main chapter, chapter five, thoroughly analyses the technical and linguistic aspects of inclusive subtitling, drawing insights from the Inclusive Subtitling Guidelines - a two version document devised by myself - and offering real-world applications supported by a case study at an Italian film festival and another case study of the short film Pure, with the relevant inclusive subtitles file annexed. In conclusion, the research sets the stage for a comprehensive exploration of inclusive subtitling's role in ensuring accessible and inclusive audiovisual experiences, particularly within film festivals. It underscores the importance of accessibility in the world of audiovisual media and highlights the need for inclusive practices to cater to diverse audiences

Using Information Communications Technologies to Implement Universal Design for Learning

The purpose of this paper is to assist Ministries of Education, their donors and partners, Disabled Persons Organizations (DPOs), and the practitioner community funded by and working with USAID to select, pilot, and (as appropriate) scale up ICT4E solutions to facilitate the implementation of Universal Design for Learning (UDL), with a particular emphasis on supporting students with disabilities to acquire literacy and numeracy skills. The paper focuses primarily on how technology can support foundational skills acquisition for students with disabilities, while also explaining when, why, and how technologies that assist students with disabilities can, in some applications, have positive impacts on all students’ basic skills development. In 2018, USAID released the Toolkit for Universal Design for Learning to Help All Children Read, section 3.1 of which provides basic information on the role of technologies to support UDL principles and classroom learning. This paper expands upon that work and offers more extensive advice on using ICT4E1 to advance equitable access to high quality learning. Like the UDL toolkit, the audience for this guide is mainly Ministries of Education and development agencies working in the area of education, but this resource can also be helpful for DPOs and non-governmental organizations (NGOs) wishing to pilot or spearhead ICT initiatives. Content for this paper was informed by expert interviews and reviews of field reports during 2018. These included programs associated with United Nations, Zero Project, World Innovation Summit, UNESCO Mobile Learning Awards, and USAID’s All Children Reading: A Grand Challenge for Development. Relevant case studies of select education programs integrating technology to improve learning outcomes for students with disabilities were summarized for this document

Communication devices for the hearing impaired

None provided

Subtitling for the Deaf and the Hard-of-hearing: A Reception Study in the Turkish Context

This study aims to contribute to a better understanding of subtitling for people with hearing impairments and to improve the accessibility to audiovisual material for hearing-impaired viewers in Turkey. It starts by providing a detailed general overview of the current state of accessibility and includes a detailed discussion on existing legislation, an outline of the limited practice of subtitling for the deaf and the hard-of-hearing (SDH) in Turkish and a profile of the assumed target audience. The ultimate goal of this research is to create a set of guidelines that can be used in the production of quality SDH in Turkey. In order to achieve these aims, the study adopts a product-oriented descriptive approach and first investigates the guidelines applied in countries where SDH has long been established as a professional practice in an attempt to reveal some of the shared values of good practice as well as potential divergences. Following this descriptive analysis, some of the key contradicting practices in the guidelines – speaker identification, reading speed, indication of sound and paralinguistic information – are tested on an audience of (37) Turkish hearing-impaired viewers so as to unveil their needs and preferences within the framework of Audience Reception Theory. Quantitative data on the preferences of Turkish viewers was collected by means of questionnaires filled in by the participants after they had watched different sets of subtitles, each of them testing a different feature. Further qualitative data was obtained through interviews conducted with four participants who took part in the experiment so as to generate more in-depth information regarding their preferences. The results yielded by the statistical analysis of the quantitative data and the interpretive phenomenological analysis of the qualitative data culminated in the drafting of a set of guidelines that can be used in the production of SDH in Turkey