Inclusive Augmented and Virtual Reality: A Research Agenda

Augmented and virtual reality experiences present significant barriers for disabled people, making it challenging to fully engage with immersive platforms. Whilst researchers have started to explore potential solutions addressing these accessibility issues, we currently lack a comprehensive understanding of research areas requiring further investigation to support the development of inclusive AR/VR systems. To address current gaps in knowledge, we led a series of multidisciplinary sandpits with relevant stakeholders (i.e., academic researchers, industry specialists, people with lived experience of disability, assistive technologists, and representatives from disability organisations, charities, and special needs educational institutions) to collaboratively explore research challenges, opportunities, and solutions. Based on insights shared by participants, we present a research agenda identifying key areas where further work is required in relation to specific forms of disability (i.e., across the spectrum of physical, visual, cognitive, and hearing impairments), including wider considerations associated with the development of more accessible immersive platforms

Inclusive Intelligent Learning Management System Framework - Application of Data Science in Inclusive Education

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Data ScienceBeing a disabled student the author faced higher education with a handicap which as experience studying during COVID 19 confinement periods matched the findings in recent research about the importance of digital accessibility through more e-learning intensive academic experiences. Narrative and systematic literature reviews enabled providing context in World Health Organization’s International Classification of Functioning, Disability and Health, legal and standards framework and information technology and communication state-of-the art. Assessing Portuguese higher education institutions’ web sites alerted to the fact that only outlying institutions implemented near perfect, accessibility-wise, websites. Therefore a gap was identified in how accessible the Portuguese higher education websites are, the needs of all students, including those with disabilities, and even the accessibility minimum legal requirements for digital products and the services provided by public or publicly funded organizations. Having identified a problem in society and exploring the scientific base of knowledge for context and state of the art was a first stage in the Design Science Research methodology, to which followed development and validation cycles of an Inclusive Intelligent Learning Management System Framework. The framework blends various Data Science study fields contributions with accessibility guidelines compliant interface design and content upload accessibility compliance assessment. Validation was provided by a focus group whose inputs were considered for the version presented in this dissertation. Not being the purpose of the research to deliver a complete implementation of the framework and lacking consistent data to put all the modules interacting with each other, the most relevant modules were tested with open data as proof of concept. The rigor cycle of DSR started with the inclusion of the previous thesis on Atlântica University Institute Scientific Repository and is to be completed with the publication of this thesis and the already started PhD’s findings in relevant journals and conferences

Library Trends 41 (1) 1992: Libraries Serving an Underserved Population: Deaf and Hearing-Impaired Patrons

published or submitted for publicatio

Adapting Hands-On Science Programs for Students with Disabilities

The CSIRO, Australia\u27s National Science Agency, has recognized their current hands-on science outreach programs do not accommodate students with disabilities and seeks to make adjustments to existing programs. Present methods for teaching students with disabilities were investigated through literature review and interviews with teachers and disability practitioners. Through observations of CSIRO programs, barriers for students with disabilities were identified. This information was used to develop a framework for adapting hands-on science programs for students with disabilities. Implementation of the framework achieves program accessibility and increases awareness of the capabilities of students with disabilities, affording them the same opportunities as their peers

Word Importance Modeling to Enhance Captions Generated by Automatic Speech Recognition for Deaf and Hard of Hearing Users

People who are deaf or hard-of-hearing (DHH) benefit from sign-language interpreting or live-captioning (with a human transcriptionist), to access spoken information. However, such services are not legally required, affordable, nor available in many settings, e.g., impromptu small-group meetings in the workplace or online video content that has not been professionally captioned. As Automatic Speech Recognition (ASR) systems improve in accuracy and speed, it is natural to investigate the use of these systems to assist DHH users in a variety of tasks. But, ASR systems are still not perfect, especially in realistic conversational settings, leading to the issue of trust and acceptance of these systems from the DHH community. To overcome these challenges, our work focuses on: (1) building metrics for accurately evaluating the quality of automatic captioning systems, and (2) designing interventions for improving the usability of captions for DHH users. The first part of this dissertation describes our research on methods for identifying words that are important for understanding the meaning of a conversational turn within transcripts of spoken dialogue. Such knowledge about the relative importance of words in spoken messages can be used in evaluating ASR systems (in part 2 of this dissertation) or creating new applications for DHH users of captioned video (in part 3 of this dissertation). We found that models which consider both the acoustic properties of spoken words as well as text-based features (e.g., pre-trained word embeddings) are more effective at predicting the semantic importance of a word than models that utilize only one of these types of features. The second part of this dissertation describes studies to understand DHH users\u27 perception of the quality of ASR-generated captions; the goal of this work was to validate the design of automatic metrics for evaluating captions in real-time applications for these users. Such a metric could facilitate comparison of various ASR systems, for determining the suitability of specific ASR systems for supporting communication for DHH users. We designed experimental studies to elicit feedback on the quality of captions from DHH users, and we developed and evaluated automatic metrics for predicting the usability of automatically generated captions for these users. We found that metrics that consider the importance of each word in a text are more effective at predicting the usability of imperfect text captions than the traditional Word Error Rate (WER) metric. The final part of this dissertation describes research on importance-based highlighting of words in captions, as a way to enhance the usability of captions for DHH users. Similar to highlighting in static texts (e.g., textbooks or electronic documents), highlighting in captions involves changing the appearance of some texts in caption to enable readers to attend to the most important bits of information quickly. Despite the known benefits of highlighting in static texts, research on the usefulness of highlighting in captions for DHH users is largely unexplored. For this reason, we conducted experimental studies with DHH participants to understand the benefits of importance-based highlighting in captions, and their preference on different design configurations for highlighting in captions. We found that DHH users subjectively preferred highlighting in captions, and they reported higher readability and understandability scores and lower task-load scores when viewing videos with captions containing highlighting compared to the videos without highlighting. Further, in partial contrast to recommendations in prior research on highlighting in static texts (which had not been based on experimental studies with DHH users), we found that DHH participants preferred boldface, word-level, non-repeating highlighting in captions

A sound idea: An investigation into accessible video game design for the deaf and hard of hearing

A widely accepted, and incorrect, assumption towards hearing accessibility in video games is that deaf and hard of hearing (DHH) users are those who encounter the least barriers and are generally well catered for. Rapid advancement in video game technology has seen video game sound evolve from simple blips generated by internal circuitry to fully realised digital audio used to convey critical information. To accommodate the DHH, this information needs to be conveyed in an alternative manner. However, evidence suggests existing accessible design solutions for the DHH lack specificity and are insufficient. Thus, the inability to hear, or hear well, has historically resulted in DHH users left with impeded experience and gameplay. This thesis describes an investigation to address the primary research question: How might accessible video game design practices be facilitated to better accommodate the deaf and hard of hearing? To examine this question, an action research method as part of a transformative mixed methods methodology was used. Data collection procedures included critical analysis of literature, observations, and a cross-sectional self-administered survey for triangulation. The critical analysis of literature exposed issues relating to accessible video game design, particularly in relation to the identification of solutions and technical implementation. Further, issues related to the classification of video game software were identified. This posed potential problems with identification of game design methods and led to the development of a new video game classification model. The new model informed an analysis on the methods used for the design of video games, and results were visually represented and mapped to the different approaches to accessible design. Subsequent analysis determined that a game assessment framework is a suitable approach to facilitating accessible design. Further investigation identified visual feedback as the most suitable form of complementary feedback to game audio. This led to the development of a new model to classify visual feedback elements used in video games, and identification of audio feedback categories based on diegetic film theory. Through triangulation of results, a new game feedback model (GFM) was developed. The GFM was used for observational experimentation to identify and classify individual visual feedback elements used in video games. Each element was analysed and mapped to categories of game sound. The resulting model, with populated data, was used to determine what visual feedback elements may be used to complement specific categories of critical game audio. A survey was subsequently used for triangulation, and resulted in amendments to the final model. Through iterative development, and interpretation of findings, the research culminated in the development of a game assessment framework. The three-step framework aids in the classification of game sounds; assesses the impact of those game sounds; and provides recommendations for complementary visual feedback elements for sounds identified as having an adverse impact on user experience and gameplay if they were to be removed. The framework is innovative and has the potential to provide practical guidance for developers of video games. In addition, this research provides the foundation for future research, with the potential to influence accessible game design for the DHH

Crazy Japanese subtitle? Shedding light on the impact of impact captions

This paper addresses intralingual captions called “impact captions” (Park 2009) that have become an integral part of entertainment TV programmes in parts of Asia. These captions are different from the mainstream intralingual captions designed for accessibility for deaf and hard-of-hearing viewers. Aimed at enhancing the entertainment value of a programme for hearing viewers, impact captions are designed to draw the viewer’s attention to particular elements according to the TV producer’s perspective. Despite the prevalent and increasing use of such captions, however, they are created without formal guidelines at the discretion of TV producers. Focusing on these novel captions which fall outside the norms of TV captions elsewhere, this paper discusses their impact on viewers while exploring methodological issues in eye-tracking research. The initial experiment results show few fixations in the caption area; despite the participants declaring that they read the captions, viewers fixate far more on the middle region of the screen where faces are shown. The paper discusses the limitations and advantages of reception studies based on eye-tracking while contributing towards further refinement of empirically-oriented reception studies in audiovisual translation (AVT) research

Conceptual model of mobile augmented reality for engaging hearing-impaired museum visitors

Mobile Augmented Reality (MAR) has matured significantly over the past decades since the birth of multimedia system. It has evolved from the conceptual idea of augmented reality experience to its actual practical applications in use on smartphones. Researchers in MAR have resolved to employ the concept of engagement in designing MAR applications to attract museum visitors’ interest and ensure a more effective learning environment. However, most of these MAR applications are tailored to normal hearing visitors while the hearing-impaired (HI) visitors are less supported. This makes HI visitors to go through unpalatable experiences and eventually become dissatisfied with their visit to the museum. Also, there is lack of studies on the conceptual model of MAR for engaging the HI museum visitors. Therefore, this study proposes a conceptual model of MAR for the HI museum engagement (MARHIME) and eventually enhances their engagement during their museum visits. In achieving the aim of this study, design science research methodology was adapted. This study has determined engagement elements through expert review which were used to design the conceptual model of MARHIME. In addition, an MAR prototype was developed based on the MARHIME conceptual model and its architecture. The MARHIME prototype includes three-dimensional models, video, text, and images to deliver salient information of important artefacts to HI museum visitors. Moreover, the MARHIME application may only function in the museum by scanning the museum environment because the HI can use MAR as a visual signal guide in order to catch missing aural signals during their visit to the museums. The study involved 73 HI museum visitors as participants in order to evaluate the MARHIME prototype on their engagement experience. From the results of the evaluation, it was found that the MARHIME prototype was able to engage the HI visitors during their visit to the museum. Therefore, this study has validated a conceptual model on MAR for engaging the HI museum visitors. This conceptual model of MARHIME can be used as guidelines for researchers in understanding the elements of MAR in engaging the HI museum visitors and for developers in assisting the process of designing and developing MAR application for the HI museum visitors. This study contributes to the engagement of HI people during their museum visits to ensure the inclusiveness of disabled people in the MAR design

Design for Accessible Collaborative Engagement: Making online synchronous collaborative learning more accessible for students with sensory impairments.

This thesis looks at the accessibility of collaborative learning and the barriers to engagement experienced by blind/visually impaired (BVI) students and deaf/hard of hearing (DHH) students. It focuses specifically on online synchronous collaborative learning after establishing that this format presented the greatest barriers, and that these student groups were not engaging. Taking a design-based research (DBR) approach, five studies were undertaken to identify these barriers and determine potential interventions. The product of the research, a result of collaborative design by the participants in the study, is a framework for accessible collaborative engagement represented in the form of an interactive website model, the Model for Accessible Collaborative Engagement (MACE). The studies involved representatives of all stakeholders in the collaborative learning process at the institution (the Open University): students, tutors, modules teams, academics, support staff, and the student union Disabled Students Group. These studies took the form of an online survey of 327 students, 10 interviews with staff and students, 6 staff workshops and a collaborative design focus group. With significant representation of the target groups (BVI and DHH) in all studies, and taking an iterative approach to the design, evaluation and construction of the framework model, the studies established that barriers existed in four main categories covering different themes: 1. Communications: aural, visual, screen reading and navigation, text and captioning, lip reading and non-verbal communications, interpretation and third-party communications, mode control, and synchronisation. 2. Emotional and Social Factors: familiarisation, support networks, self-advocacy, opting out, cognitive load, and stress and anxiety. 3. Provisioning and Technical Factors: dissemination, speed and pacing of sessions, staff training, participation control, group size, technical provisioning, and recordings. 4. Activity and Session Design: Volume of materials, advance materials, accessible materials, accessible activities, and session formats. Interventions were designed that could reduce the barriers in each of these categories and themes by adjustments and changes from both the student and institutional standpoints. MACE is designed to be utilised by both students and staff to provide guidance and suggestions on how to identify and acknowledge these barriers and implement interventions to reduce them. This research represents an original and essential contribution to the field of investigation. As well as informing future research inquiry, the model can be used by all participants and stakeholders in online collaborative learning to help reduce barriers for BVI and DHH students and improve inclusivity in synchronous online events