Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing

The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants' subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems.Comment: 10 pages, 8 figures, published in ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17

Availability of specialized healthcare facilities for deaf and hard of hearing individuals.

Context: To allow a medical consultation to proceed successfully, it is essential that physicians are aware of the linguistic and cultural backgrounds of deaf and hard of hearing individuals (DHH) and related communication aspects. Some specialised healthcare facilities have emerged to respond to the specific needs of people who are DHH. Objective: This study aims to provide insight into the various types of general healthcare facilities available for DHH individuals. By sharing and comparing experiences and results improvements can be made. Design, Data Sources and Study selection: A systematic review of the literature on specialised healthcare for DHH people was performed. The following databases were searched: PubMed, Web of Science, PsycINFO, Academic Search Premier, CINAHL and Embase. After independent extraction per article by two readers, fifteen articles were included in the systematic review. As it appeared that not all existing locations of facilities of which we were aware were described in the literature, we expanded the data collection with internet searches, specific literature searches and unstructured interviews. Results: Some countries have developed facilities to meet the needs DHH people Experts and patients’ groups report that the perceived quality of healthcare and health education in specialised healthcare settings is higher compared to regular healthcare settings. Two projects undertaken to improve the health related knowledge level of DHH people, proved to be effective. Conclusion: Some facilities or combinations of facilities are used in different countries to attempt to meet the needs of DHH patients. These facilities are rarely described in the scientific literature. Further development of specialised healthcare facilities for DHH patients, which should include high quality studies on their effectiveness, is imperative to comply with medical ethical standards and respect the human rights of DHH people

Video Captions for Online Courses: Do YouTube’s Auto-generated Captions Meet Deaf Students’ Needs?

  Providing captions for videos used in online courses is an area of interest for institutions of higher education. There are legal and ethical ramifications as well as time constraints to consider. Captioning tools are available, but some universities rely on the auto-generated YouTube captions. This study looked at a particular type of video—the weekly informal news update created by individual professors for their online classes—to see if automatic captions (also known as subtitles) are sufficiently accurate to meet the needs of deaf students. A total of 68 minutes of video captions were analysed and 525 phrase-level errors were found. On average, therefore, there were 7.7 phrase errors per minute. Findings indicate that auto-generated captions are too inaccurate to be used exclusively. Additional studies are needed to determine whether they can provide a starting point for a process of captioning that reduces the preparation time