Word Importance Modeling to Enhance Captions Generated by Automatic Speech Recognition for Deaf and Hard of Hearing Users

People who are deaf or hard-of-hearing (DHH) benefit from sign-language interpreting or live-captioning (with a human transcriptionist), to access spoken information. However, such services are not legally required, affordable, nor available in many settings, e.g., impromptu small-group meetings in the workplace or online video content that has not been professionally captioned. As Automatic Speech Recognition (ASR) systems improve in accuracy and speed, it is natural to investigate the use of these systems to assist DHH users in a variety of tasks. But, ASR systems are still not perfect, especially in realistic conversational settings, leading to the issue of trust and acceptance of these systems from the DHH community. To overcome these challenges, our work focuses on: (1) building metrics for accurately evaluating the quality of automatic captioning systems, and (2) designing interventions for improving the usability of captions for DHH users. The first part of this dissertation describes our research on methods for identifying words that are important for understanding the meaning of a conversational turn within transcripts of spoken dialogue. Such knowledge about the relative importance of words in spoken messages can be used in evaluating ASR systems (in part 2 of this dissertation) or creating new applications for DHH users of captioned video (in part 3 of this dissertation). We found that models which consider both the acoustic properties of spoken words as well as text-based features (e.g., pre-trained word embeddings) are more effective at predicting the semantic importance of a word than models that utilize only one of these types of features. The second part of this dissertation describes studies to understand DHH users\u27 perception of the quality of ASR-generated captions; the goal of this work was to validate the design of automatic metrics for evaluating captions in real-time applications for these users. Such a metric could facilitate comparison of various ASR systems, for determining the suitability of specific ASR systems for supporting communication for DHH users. We designed experimental studies to elicit feedback on the quality of captions from DHH users, and we developed and evaluated automatic metrics for predicting the usability of automatically generated captions for these users. We found that metrics that consider the importance of each word in a text are more effective at predicting the usability of imperfect text captions than the traditional Word Error Rate (WER) metric. The final part of this dissertation describes research on importance-based highlighting of words in captions, as a way to enhance the usability of captions for DHH users. Similar to highlighting in static texts (e.g., textbooks or electronic documents), highlighting in captions involves changing the appearance of some texts in caption to enable readers to attend to the most important bits of information quickly. Despite the known benefits of highlighting in static texts, research on the usefulness of highlighting in captions for DHH users is largely unexplored. For this reason, we conducted experimental studies with DHH participants to understand the benefits of importance-based highlighting in captions, and their preference on different design configurations for highlighting in captions. We found that DHH users subjectively preferred highlighting in captions, and they reported higher readability and understandability scores and lower task-load scores when viewing videos with captions containing highlighting compared to the videos without highlighting. Further, in partial contrast to recommendations in prior research on highlighting in static texts (which had not been based on experimental studies with DHH users), we found that DHH participants preferred boldface, word-level, non-repeating highlighting in captions

A competencies framework of visual impairments for enabling shared understanding in design

Existing work in Human Computer Interaction and accessibility research has long sought to investigate the experiences of people with visual impairments in order to address their needs through technology design and integrate their participation into different stages of the design process. Yet challenges remain regarding how disabilities are framed in technology design and the extent of involvement of disabled people within it. Furthermore, accessibility is often considered a specialised job and misunderstandings or assumptions about visually impaired people’s experiences and needs occur outside dedicated fields. This thesis presents an ethnomethodology-informed design critique for supporting awareness and shared understanding of visual impairments and accessibility that centres on their experiences, abilities, and participation in early-stage design. This work is rooted in an in-depth empirical investigation of the interactional competencies that people with visual impairments exhibit through their use of technology, which informs and shapes the concept of a Competencies Framework of Visual Impairments. Although past research has established stances for considering the individual abilities of disabled people and other social and relational factors in technology design, by drawing on ethnomethodology and its interest in situated competence this thesis employs an interactional perspective to investigate the practical accomplishments of visually impaired people. Thus, this thesis frames visual impairments in terms of competencies to be considered in the design process, rather than a deficiency or problem to be fixed through technology. Accordingly, this work favours supporting awareness and reflection rather than the design of particular solutions, which are also strongly needed for advancing accessible design at large. This PhD thesis comprises two main empirical studies branched into three different investigations. The first and second investigations are based on a four-month ethnographic study with visually impaired participants examining their everyday technology practices. The third investigation comprises the design and implementation of a workshop study developed to include people with and without visual impairments in collaborative reflections about technology and accessibility. As such, each investigation informed the ones that followed, revisiting and refining concepts and design materials throughout the thesis. Although ethnomethodology is the overarching approach running through this PhD project, each investigation has a different focus of enquiry: • The first is focused on analysing participants’ technology practices and unearthing the interactional competencies enabling them. • The second is focused on analysing technology demonstrations, which were a pervasive phenomenon recorded during fieldwork, and the work of demonstrating as exhibited by visually impaired participants. • Lastly, the third investigation defines a workshop approach employing video demonstrations and a deck of reflective design cards as building blocks for enabling shared understanding among people with and without visual impairments from different technology backgrounds; that is, users, technologists, designers, and researchers. Overall, this thesis makes several contributions to audiences within and outside academia, such as the detailed accounts of some of the main technology practices of people with visual impairments and the methodological analysis of demonstrations in empirical Human Computer Interaction and accessibility research. Moreover, the main contribution lies in the conceptualisation of a Competencies Framework of Visual Impairments from the empirical analysis of interactional competencies and their practical exhibition through demonstrations, as well as the creation and use of a deck of cards that encapsulates the competencies and external elements involved in the everyday interactional accomplishments of people with visual impairments. All these contributions are lastly brought together in the implementation of the workshop approach that enabled participants to interact with and learn from each other. Thus, this thesis builds upon and advances contemporary strands of work in Human Computer Interaction that call for re-orienting how visual impairments and, overall, disabilities are framed in technology design, and ultimately for re-shaping the design practice itself

A competencies framework of visual impairments for enabling shared understanding in design

Existing work in Human Computer Interaction and accessibility research has long sought to investigate the experiences of people with visual impairments in order to address their needs through technology design and integrate their participation into different stages of the design process. Yet challenges remain regarding how disabilities are framed in technology design and the extent of involvement of disabled people within it. Furthermore, accessibility is often considered a specialised job and misunderstandings or assumptions about visually impaired people’s experiences and needs occur outside dedicated fields. This thesis presents an ethnomethodology-informed design critique for supporting awareness and shared understanding of visual impairments and accessibility that centres on their experiences, abilities, and participation in early-stage design. This work is rooted in an in-depth empirical investigation of the interactional competencies that people with visual impairments exhibit through their use of technology, which informs and shapes the concept of a Competencies Framework of Visual Impairments. Although past research has established stances for considering the individual abilities of disabled people and other social and relational factors in technology design, by drawing on ethnomethodology and its interest in situated competence this thesis employs an interactional perspective to investigate the practical accomplishments of visually impaired people. Thus, this thesis frames visual impairments in terms of competencies to be considered in the design process, rather than a deficiency or problem to be fixed through technology. Accordingly, this work favours supporting awareness and reflection rather than the design of particular solutions, which are also strongly needed for advancing accessible design at large. This PhD thesis comprises two main empirical studies branched into three different investigations. The first and second investigations are based on a four-month ethnographic study with visually impaired participants examining their everyday technology practices. The third investigation comprises the design and implementation of a workshop study developed to include people with and without visual impairments in collaborative reflections about technology and accessibility. As such, each investigation informed the ones that followed, revisiting and refining concepts and design materials throughout the thesis. Although ethnomethodology is the overarching approach running through this PhD project, each investigation has a different focus of enquiry: • The first is focused on analysing participants’ technology practices and unearthing the interactional competencies enabling them. • The second is focused on analysing technology demonstrations, which were a pervasive phenomenon recorded during fieldwork, and the work of demonstrating as exhibited by visually impaired participants. • Lastly, the third investigation defines a workshop approach employing video demonstrations and a deck of reflective design cards as building blocks for enabling shared understanding among people with and without visual impairments from different technology backgrounds; that is, users, technologists, designers, and researchers. Overall, this thesis makes several contributions to audiences within and outside academia, such as the detailed accounts of some of the main technology practices of people with visual impairments and the methodological analysis of demonstrations in empirical Human Computer Interaction and accessibility research. Moreover, the main contribution lies in the conceptualisation of a Competencies Framework of Visual Impairments from the empirical analysis of interactional competencies and their practical exhibition through demonstrations, as well as the creation and use of a deck of cards that encapsulates the competencies and external elements involved in the everyday interactional accomplishments of people with visual impairments. All these contributions are lastly brought together in the implementation of the workshop approach that enabled participants to interact with and learn from each other. Thus, this thesis builds upon and advances contemporary strands of work in Human Computer Interaction that call for re-orienting how visual impairments and, overall, disabilities are framed in technology design, and ultimately for re-shaping the design practice itself

Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

How a Diverse Research Ecosystem Has Generated New Rehabilitation Technologies: Review of NIDILRR’s Rehabilitation Engineering Research Centers

Over 50 million United States citizens (1 in 6 people in the US) have a developmental, acquired, or degenerative disability. The average US citizen can expect to live 20% of his or her life with a disability. Rehabilitation technologies play a major role in improving the quality of life for people with a disability, yet widespread and highly challenging needs remain. Within the US, a major effort aimed at the creation and evaluation of rehabilitation technology has been the Rehabilitation Engineering Research Centers (RERCs) sponsored by the National Institute on Disability, Independent Living, and Rehabilitation Research. As envisioned at their conception by a panel of the National Academy of Science in 1970, these centers were intended to take a “total approach to rehabilitation”, combining medicine, engineering, and related science, to improve the quality of life of individuals with a disability. Here, we review the scope, achievements, and ongoing projects of an unbiased sample of 19 currently active or recently terminated RERCs. Specifically, for each center, we briefly explain the needs it targets, summarize key historical advances, identify emerging innovations, and consider future directions. Our assessment from this review is that the RERC program indeed involves a multidisciplinary approach, with 36 professional fields involved, although 70% of research and development staff are in engineering fields, 23% in clinical fields, and only 7% in basic science fields; significantly, 11% of the professional staff have a disability related to their research. We observe that the RERC program has substantially diversified the scope of its work since the 1970’s, addressing more types of disabilities using more technologies, and, in particular, often now focusing on information technologies. RERC work also now often views users as integrated into an interdependent society through technologies that both people with and without disabilities co-use (such as the internet, wireless communication, and architecture). In addition, RERC research has evolved to view users as able at improving outcomes through learning, exercise, and plasticity (rather than being static), which can be optimally timed. We provide examples of rehabilitation technology innovation produced by the RERCs that illustrate this increasingly diversifying scope and evolving perspective. We conclude by discussing growth opportunities and possible future directions of the RERC program

The effectiveness of the web accessibility audit as a motivational and educational tool in inclusive web design

The importance of Web sites that can be accessed and used regardless of an individual?s disability is critical. One barrier to improved accessibility of Web sites relates to the gap between Web authors? technical knowledge of Web accessibility guidelines and a broader understanding of the challenges facing disabled people when interacting with Web sites.This thesis describes the development and evaluation of a Web accessibility auditing methodology with the dual aims of accurately identifying accessibility barriers present in a Web site, and presenting the audit findings and recommended actions in a way that informs, educates and engenders an improved understanding of accessibility amongst the audience.The methodology was piloted amongst a sample of Web sites, validated against other published accessibility evaluation methodologies, and adopted for subsequent audits carried out on a commercial basis. The impact on recipient organisations and individuals of a sample of 14 commercially commissioned audits was then evaluated. Audit recipients were surveyed, and each Web site audited evaluated to identify any changes to accessibility, and the presence of evidence of changes or improvements to accessibility strategy.Strong indications were found that the audits had a positive impact both on individuals and on the commissioning organisations. The audits were identified as having a particularly positive educational and motivational impact on recipients who did not identify themselves as having expertise in Web accessibility. There was also evidence that the design approach promoted by the audits had been adopted and applied by some of the commissioning organisations. The majority of respondents cited the recommendations for improvement as the most valuable feature of the audit. This illustrates a tension between the importance of presenting specific recommendations for actions and providing richer narrative accounts of evaluation stages to encourage a more holistic appreciation of accessibility.The particular benefits of the study are found in the identification of evidence of impact of commercially-commissioned Web accessibility audits over a period of time to recipients of varying characteristics. A number of areas for further investigation have been identified, focusing on investigating the potential value of the accessibility audit in providing more ?experiential? evaluation stages.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Modern Socio-Technical Perspectives on Privacy

This open access book provides researchers and professionals with a foundational understanding of online privacy as well as insight into the socio-technical privacy issues that are most pertinent to modern information systems, covering several modern topics (e.g., privacy in social media, IoT) and underexplored areas (e.g., privacy accessibility, privacy for vulnerable populations, cross-cultural privacy). The book is structured in four parts, which follow after an introduction to privacy on both a technical and social level: Privacy Theory and Methods covers a range of theoretical lenses through which one can view the concept of privacy. The chapters in this part relate to modern privacy phenomena, thus emphasizing its relevance to our digital, networked lives. Next, Domains covers a number of areas in which privacy concerns and implications are particularly salient, including among others social media, healthcare, smart cities, wearable IT, and trackers. The Audiences section then highlights audiences that have traditionally been ignored when creating privacy-preserving experiences: people from other (non-Western) cultures, people with accessibility needs, adolescents, and people who are underrepresented in terms of their race, class, gender or sexual identity, religion or some combination. Finally, the chapters in Moving Forward outline approaches to privacy that move beyond one-size-fits-all solutions, explore ethical considerations, and describe the regulatory landscape that governs privacy through laws and policies. Perhaps even more so than the other chapters in this book, these chapters are forward-looking by using current personalized, ethical and legal approaches as a starting point for re-conceptualizations of privacy to serve the modern technological landscape. The book’s primary goal is to inform IT students, researchers, and professionals about both the fundamentals of online privacy and the issues that are most pertinent to modern information systems. Lecturers or teacherscan assign (parts of) the book for a “professional issues” course. IT professionals may select chapters covering domains and audiences relevant to their field of work, as well as the Moving Forward chapters that cover ethical and legal aspects. Academicswho are interested in studying privacy or privacy-related topics will find a broad introduction in both technical and social aspects

Modern Socio-Technical Perspectives on Privacy

This open access book provides researchers and professionals with a foundational understanding of online privacy as well as insight into the socio-technical privacy issues that are most pertinent to modern information systems, covering several modern topics (e.g., privacy in social media, IoT) and underexplored areas (e.g., privacy accessibility, privacy for vulnerable populations, cross-cultural privacy). The book is structured in four parts, which follow after an introduction to privacy on both a technical and social level: Privacy Theory and Methods covers a range of theoretical lenses through which one can view the concept of privacy. The chapters in this part relate to modern privacy phenomena, thus emphasizing its relevance to our digital, networked lives. Next, Domains covers a number of areas in which privacy concerns and implications are particularly salient, including among others social media, healthcare, smart cities, wearable IT, and trackers. The Audiences section then highlights audiences that have traditionally been ignored when creating privacy-preserving experiences: people from other (non-Western) cultures, people with accessibility needs, adolescents, and people who are underrepresented in terms of their race, class, gender or sexual identity, religion or some combination. Finally, the chapters in Moving Forward outline approaches to privacy that move beyond one-size-fits-all solutions, explore ethical considerations, and describe the regulatory landscape that governs privacy through laws and policies. Perhaps even more so than the other chapters in this book, these chapters are forward-looking by using current personalized, ethical and legal approaches as a starting point for re-conceptualizations of privacy to serve the modern technological landscape. The book’s primary goal is to inform IT students, researchers, and professionals about both the fundamentals of online privacy and the issues that are most pertinent to modern information systems. Lecturers or teacherscan assign (parts of) the book for a “professional issues” course. IT professionals may select chapters covering domains and audiences relevant to their field of work, as well as the Moving Forward chapters that cover ethical and legal aspects. Academicswho are interested in studying privacy or privacy-related topics will find a broad introduction in both technical and social aspects

Digital support for self-management and meaningful activities of people with mild dementia:Development, implementation and feasibility of a person-centred touch-screen intervention

This thesis reports on the research that was conducted to develop and pilot test a person-centred touchscreen-based program (FindMyApps intervention) that supports people with mild dementia and their informal carers in how to use a tablet and apps for self-management and meaningful activities. The program consists of the FindMyApps training for informal carers in supporting people with mild dementia in using a tablet and the FindMyApps selection tool designed to help people with mild dementia find apps for self-management and meaningful activities that fit their needs, wishes and abilities. The program was developed following the Medical Research Council (MRC) Framework for the design and evaluation of complex interventions and in co-creation with end users. The chapters of this thesis describe the first three phases of this framework i.e.: the preclinical or theoretical phase (0); the modelling phase (I) and the exploratory trial (II). The execution of an RCT (III), and further implementation (IV) in case the intervention proves effective, are beyond the scope of this thesis. In the first phase user needs studies were performed to identify user requirements for desired self-management and meaningful activities and to identify user-requirements related to the use of apps. Eight focus groups were performed in which people with mild cognitive impairment (MCI) or mild dementia (n=13) and informal carers (n=15) participated. We also conducted a literature review to explore which training interventions are most effective for people with mild dementia in (re)learning how to use technologies, including handheld touchscreen devices. An electronic search was conducted in the following databases: PubMed, APA PsycInfo (EBSCO) and CINAHL (EBSCO). In total 16 studies met the inclusion criteria. Our review contributed to the growing amount of promising evidence on the potential impact of Errorless learning training interventions for people with mild to moderate dementia in (re)learning how to use technology. Based on these results we developed the FindMyApps training. In the second phase of the framework we developed a first concept of the FindMyApps selection tool based on identified user requirements. To ensure its usability, the web-based tool was developed using a ‘user-participatory design’ involving the close collaboration of potential users, a development team (researchers, developers and designers) and an expert team (experts in person-centred dementia care and ICT). In three short iterative rounds – so called ‘sprints’ – the users (people with dementia (n=8) and (in)formal carers (n=10)) were invited to test whether the prototypes were in line with their needs, wishes and abilities. This generated important insights into user-interface aspects relating to (i) useful content and (ii) a user-friendly tool design. In the third phase we pilot tested the FindMyApps intervention by means of individual semi-structured interviews and a pilot randomized controlled exploratory trial. Twenty people with mild dementia and informal carer dyads were randomly assigned to the FindMyApps group (n=10), receiving either the FindMyApps training and selection tool, or a control condition (n=10), receiving only a short tablet training. Pre- and post-test measurements at a three month follow-up, consisted of questionnaires and post-test semi-structured interviews. Based on the qualitative results and the effect sizes on the outcomes measured in this study, we consider that the FindMyApps intervention has the potential to positively influence the self-management and engagement in meaningful activities in people with dementia. Future studies with a larger sample should better indicate whether this expectation can be confirmed. The intervention will be further improved and tested in a larger pilot-RCT study and its effectiveness subsequently evaluated in a definite RCT