Diegetic Sonification for Low Vision Gamers

There are not many games designed for all players that provide accommodations for low vision users. This means that low vision users may not get to engage with the gaming community in the same way as their sighted peers. In this thesis, I explore how diegetic sonification can be used as a tool to support these low vision gamers in the typical gaming environment. I asked low vision players to engage with a prototype game level with two diegetic sonification techniques applied, without the use of their corrective lenses. I found that participants had more enjoyment and experienced less difficulty and confusion when the sonification techniques were applied. This shows that diegetic sonification can be used to support low vision players in playing games without their corrective lenses and improved their overall enjoyment.

Adventures in software engineering : plugging HCI & acessibility gaps with open source solutions

There has been a great deal of research undertaken in the field of Human-Computer Interfaces (HCI), input devices, and output modalities in recent years. From touch-based and voice control input mechanisms such as those found on modern smart-devices to the use of touch-free input through video-stream/image analysis (including depth streams and skeletal mapping) and the inclusion of gaze tracking, head tracking, virtual reality and beyond - the availability and variety of these I/O (Input/Output) mechanisms has increased tremendously and progressed both into our living rooms and into our lives in general. With regard to modern desktop computers and videogame consoles, at present many of these technologies are at a relatively immature stage of development - their use often limited to simple adjuncts to the staple input mechanisms of mouse, keyboard, or joystick / joypad inputs. In effect, we have these new input devices - but we're not quite sure how best to use them yet; that is, where their various strengths and weaknesses lie, and how or if they can be used to conveniently and reliably drive or augment applications in our everyday lives. In addition, much of this technology is provided by proprietary hardware and software, providing limited options for customisation or adaptation to better meet the needs of specific users. Therefore, this project investigated the development of open source software solutions to address various aspects of innovative user I/O in a flexible manner. Towards this end, a number of original software applications have been developed which incorporate functionality aimed at enhancing the current state of the art in these areas and making that software freely available for use by any who may find it beneficial.Doctor of Philosoph

Computer Entertainment Technologies for the Visually Impaired: An Overview

Over the last years, works related to accessible technologies have increased both in number and in quality. This work presents a series of articles which explore different trends in the field of accessible video games for the blind or visually impaired. Reviewed articles are distributed in four categories covering the following subjects: (1) video game design and architecture, (2) video game adaptations, (3) accessible games as learning tools or treatments and (4) navigation and interaction in virtual environments. Current trends in accessible game design are also analysed, and data is presented regarding keyword use and thematic evolution over time. As a conclusion, a relative stagnation in the field of human-computer interaction for the blind is detected. However, as the video game industry is becoming increasingly interested in accessibility, new research opportunities are starting to appear

Evaluation of a non-visual auditory choropleth and travel map viewer

Presented at the 27th International Conference on Auditory Display (ICAD 2022) 24-27 June 2022, Virtual conference.The auditory virtual reality interface of Audiom, a web-based map viewer, was evaluated by thirteen blind participants. In Audiom, the user is an avatar that navigates, using the arrow keys, through geographic data, as if they are playing a first-person, egocentric game. The research questions were: What will make blind users want to use Audiom maps? And Can participants demonstrate basic acquisition of spatial knowledge after viewing an auditory map? A dynamic choropleth map of state-level US COVID-19 data, and a detailed OpenStreetMap powered travel map, were evaluated. All participants agreed they wanted more maps of all kinds, in particular county-level COVID data, and they would use Audiom once some bugs were fixed and their few recommended features were added. Everyone wanted to see Audiom embedded in their existing travel and mapping applications. All participants were able to answer a question evaluating spatial knowledge. Participants also agreed this spatial information was not available in existing applications

Using Sound to Represent Uncertainty in Spatial Data

There is a limit to the amount of spatial data that can be shown visually in an effective manner, particularly when the data sets are extensive or complex. Using sound to represent some of these data (sonification) is a way of avoiding visual overload. This thesis creates a conceptual model showing how sonification can be used to represent spatial data and evaluates a number of elements within the conceptual model. These are examined in three different case studies to assess the effectiveness of the sonifications. Current methods of using sonification to represent spatial data have been restricted by the technology available and have had very limited user testing. While existing research shows that sonification can be done, it does not show whether it is an effective and useful method of representing spatial data to the end user. A number of prototypes show how spatial data can be sonified, but only a small handful of these have performed any user testing beyond the authors’ immediate colleagues (where n > 4). This thesis creates and evaluates sonification prototypes, which represent uncertainty using three different case studies of spatial data. Each case study is evaluated by a significant user group (between 45 and 71 individuals) who completed a task based evaluation with the sonification tool, as well as reporting qualitatively their views on the effectiveness and usefulness of the sonification method. For all three case studies, using sound to reinforce information shown visually results in more effective performance from the majority of the participants than traditional visual methods. Participants who were familiar with the dataset were much more effective at using the sonification than those who were not and an interactive sonification which requires significant involvement from the user was much more effective than a static sonification, which did not provide significant user engagement. Using sounds with a clear and easily understood scale (such as piano notes) was important to achieve an effective sonification. These findings are used to improve the conceptual model developed earlier in this thesis and highlight areas for future research

Auditory Displays for People with Visual Impairments during Travel

Menschen mit Blindheit oder Sehbehinderungen begegnen beim Reisen zahlreichen Barrieren, was sich auf die Lebensqualität auswirkt. Obwohl spezielle elektronische Reisehilfen schon seit vielen Jahren im Mittelpunkt der Forschung stehen, werden sie von der Zielgruppe nach wie vor kaum genutzt. Dies liegt unter anderem daran, dass die von den Nutzern benötigten Informationen von der Technologie nur unzureichend bereitgestellt werden. Außerdem entsprechen die Schnittstellen selten den Bedürfnissen der Nutzer. In der vorliegender Arbeit gehen wir auf diese Defizite ein und definieren die Anforderungen für barrierefreies Reisen in Bezug auf den Informationsbedarf (Was muss vermittelt werden?) und die nichtfunktionalen Anforderungen (Wie muss es vermittelt werden?). Außerdem schlagen wir verschiedene auditive Displays vor, die die Bedürfnisse von Menschen mit Sehbeeinträchtigungen während einer Reise berücksichtigen. Wir entwerfen, implementieren und evaluieren unsere Schnittstellen nach einem nutzerzentriertem Ansatz, wobei wir während des gesamten Prozesses Nutzer und Experten aus diesem Bereich einbeziehen. In einem ersten Schritt erheben wir den Informationsbedarf von Menschen mit Behinderungen im Allgemeinen und von Menschen mit Sehbeeinträchtigungen im Besonderen, wenn sie sich in Gebäuden bewegen. Außerdem vergleichen wir die gesammelten Informationen mit dem, was derzeit in OpenStreetMap (OSM), einer freien geografischen Datenbank, kartiert werden kann, und machen Vorschläge zur Schließung der Lücke. Unser Ziel ist es, die Kartierung aller benötigten Informationen zu ermöglichen, um sie in Lösungen zur Unterstützung des unabhängigen Reisens zu verwenden. Nachdem wir die Frage beantwortet haben, welche Informationen benötigt werden, gehen wir weiter und beantworten die Frage, wie diese den Nutzern vermittelt werden können. Wir definieren eine Sammlung nicht-funktionaler Anforderungen, die wir in einer Befragung mit 22 Mobilitätstrainern verfeinern und bewerten. Anschließend schlagen wir eine Grammatik - oder anders ausgedrückt, eine strukturierte Art der Informationsvermittlung - für Navigationsanweisungen bei Reisen im Freien vor, die Straßenränder, das Vorhandensein von Gehwegen und Kreuzungen berücksichtigt - alles wichtige Informationen für Menschen mit Sehbeeinträchtigungen. Darüber hinaus können mit unserer Grammatik auch Orientierungspunkte, Sehenswürdigkeiten und Hindernisse vermittelt werden, was die Reise zu einem ganzheitlichen und sichereren Erlebnis macht. Wir implementieren unsere Grammatik in einen bestehenden Prototyp und evaluieren sie mit der Zielgruppe. Es hat sich gezeigt, dass in Gebäuden Beschreibungen der Umgebung die Erstellung von mentalen Karten unterstützen und damit die Erkundung und spontane Entscheidungsfindung besser fördern als Navigationsanweisungen. Wir definieren daher eine Grammatik für die Vermittlung von Informationen über die Umgebung in Innenräumen für Menschen mit Sehbeeinträchtigungen. Wir bewerten die Grammatik in einer Online-Studie mit 8 Nutzern aus der Zielgruppe. Wir zeigen, dass die Nutzer strukturierte Sätze mit fester Wortreihenfolge benötigen. Schließlich implementieren wir die Grammatik als Proof-of-Concept in eine bestehende prototypische App. Sprachausgabe ist zwar Stand der Technik im Bereich der Ausgabeschnittstellen für Menschen mit Sehbeeinträchtigungen, hat aber auch Nachteile: es ist für Menschen mit Leseschwäche unzugänglich und kann für manche Nutzer zu langsam sein. Wir nehmen uns dieses Problems an und untersuchen den Einsatz von Sonifikation in Form von auditiven Symbolen in Kombination mit Parameter-Mapping zur Vermittlung von Informationen über Objekte und deren Verortung in der Umgebung. Da eine erste Evaluierung positive Ergebnisse lieferte, erstellten wir in einem nutzerzentrierten Entwicklungsansatz einen Datensatz mit kurzen auditiven Symbolen für 40 Alltagsgegenstände. Wir evaluieren den Datensatz mit 16 blinden Menschen und zeigen, dass die Töne intuitiv sind. Schließlich vergleichen wir in einer Nutzerstudie mit 5 Teilnehmern Sprachausgabe mit nicht-sprachlicher Sonifikation. Wir zeigen, dass Sonifikation für die Vermittlung von groben Informationen über Objekte in der Umgebung genau so gut geeignet ist wie Sprache, was die Benutzerfreundlichkeit angeht. Abschließend listen wir einige Vorteile von Sprache und Sonifikation auf, die zum Vergleich und als Entscheidungshilfe dienen sollen. Diese Arbeit befasst sich mit den Bedürfnissen von Menschen mit Sehbeeinträchtigungen während der Reise in Bezug auf die benötigten Informationen und Schnittstellen. In einem nutzerzentrierten Ansatz schlagen wir verschiedene akustische Schnittstellen vor, die auf sprachlicher und nicht-sprachlicher Sonifikation basieren. Anhand mehrerer Nutzerstudien, an denen sowohl Nutzer als auch Experten beteiligt sind, entwerfen, implementieren und evaluieren wir unsere Schnittstellen. Wir zeigen, dass elektronische Reisehilfen in der Lage sein müssen, große Mengen an Informationen auf strukturierte Weise zu vermitteln, jedoch angepasst an den Nutzungskontext und die Präferenzen und Fähigkeiten der Nutzer

Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

Technical Document Accessibility

Electrical and Electronic Engineerin

ASSISTIVE TECHNOLOGIES ON MOBILE DEVICES FOR PEOPLE WITH VISUAL IMPAIRMENTS

Spatial understanding and cognitive mapping are challenging tasks for people with visual impairments. The goal of this work is to leverage computer vision and spatial understanding techniques along with audio-haptic proprioceptive interaction paradigms for assisting people with visual impairments in spatial comprehension and memorization. Abstract space exploration in the field of assistive didactics is tackled through tactile exploration and audio feedback resulting in two solutions. The first one focuses on math learning in primary education while the second one focuses on function graph tactile exploration and sonification. In the field of spatial comprehension during way-finding for people with visual impairments, computer vision and spatial reasoning techniques are used for detecting visual cues such as zebra pedestrian crossings and for safely guiding the user with respect to the detected elements. Suitable interaction paradigms based on sonification and haptic feedback are designed to assist the user efficiently and quickly during the navigation

Design and Evaluation of a Virtual Reality Game to Improve Physical and Cognitive Acuity

Physical and mental health are both integral to healthy living and ageing, and a causal-cum-symbiotic relationship has been observed between the two. Physical and cognitive activities such as exercise and board games are known to promote healthy ageing. In this regard, highly engaging lightboard games are known to improve hand-eye coordination, reflexes, and motor skills for individuals. Immersivity of virtual reality games can transform mundane and repetitive exercise routines into stimulating experiences, and they can be utilized by users to improve physical and cognitive performance from the comfort of their homes. In this study, we adopt design science framework to design, develop and evaluate a VR BATAK lightboard game to improve physical reaction, hand-eye coordination, visual memory and cognitive processing. Based on the findings from evaluation over three phases, we propose three design principles related to accessibility, sensory cueing and cognitive loading, as theoretical and practical contributions of this study