Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

視空間支援のためのデバイスアート:人間の反響定位能力の拡張

この博士論文は全文公表に適さないやむを得ない事由があり要約のみを公表していましたが、解消したため、令和３(2021)年1月18日に全文を公表しました。筑波大学 (University of Tsukuba)201

Creative Haptic Interface Design for the Aging Population

Audiovisual human-computer-interfaces still make up the majority of content to the public; however, haptic interfaces offer unique advantage over the dominant information infrastructure, particularly for users with a disability or diminishing cognitive and physical skills like the elderly. The tactile sense allows users to integrate new, unobstructive channels for digital information into their sensorium, one that is less likely to be overwhelmed compared to vision and audition. Haptics research focus on the development of hardware, improving resolution, modality, and fidelity of the actuators. Despite the technological limitations, haptic interfaces are shown to reinforce physical skill acquisition, therapy, and communication. This chapter will present key characteristics intuitive tactile interfaces should capture for elderly end-users; sample projects will showcase unique applications and designs that identify the limitations of the UI

Multisensory Integration Design in Music for Cochlear Implant Users

Cochlear implant (CI) users experience several challenges when listening to music. However, their hearing abilities are greatly diverse and their musical experiences may significantly vary from each other. In this research, we investigate this diversity in CI users' musical experience, preferences, and practices. We integrate multisensory feedback into their listening experiences to support the perception of specific musical features and elements. Four installations are implemented, each exploring a different sensory modality assisting or supporting CI users' listening experience. We study these installations throughout semi-structured and exploratory workshops with participants. We report the results of our process-oriented assessment of CI users' experience with music. Because the CI community is a minority participant group in music, musical instrument design frameworks and practices vary from those of hearing cultures. We share guidelines for designing multisensory integration that derived from our studies with individual CI users and specifically aimed to enrich their experiences

Haptic Wave

We present the Haptic Wave, a device that allows cross-modal mapping of digital audio to the haptic domain, intended for use by audio producers/engineers with visual impairments. We describe a series of participatory design activities adapted to non-sighted users where the act of prototyping facilitates dialog. A series of workshops scoping user needs, and testing a technology mock up and lo-fidelity prototype fed into the design of a final high-spec prototype. The Haptic Wave was tested in the laboratory, then deployed in real world settings in recording studios and audio production facilities. The cross-modal mapping is kinesthetic and allows the direct manipulation of sound without the translation of an existing visual interface. The research gleans insight into working with users with visual impairments, and transforms perspective to think of them as experts in non-visual interfaces for all users. This received the Best Paper Award at CHI 2016, the most prestigious human-computer interaction conference and one of the top-ranked conferences in computer science

Computer Entertainment Technologies for the Visually Impaired: An Overview

Over the last years, works related to accessible technologies have increased both in number and in quality. This work presents a series of articles which explore different trends in the field of accessible video games for the blind or visually impaired. Reviewed articles are distributed in four categories covering the following subjects: (1) video game design and architecture, (2) video game adaptations, (3) accessible games as learning tools or treatments and (4) navigation and interaction in virtual environments. Current trends in accessible game design are also analysed, and data is presented regarding keyword use and thematic evolution over time. As a conclusion, a relative stagnation in the field of human-computer interaction for the blind is detected. However, as the video game industry is becoming increasingly interested in accessibility, new research opportunities are starting to appear

THE CROSS-SENSORY GLOBE: Co-Designing a 3D Audio-Tactile Globe Prototype for Blind and Low- Vision Users to Learn Geography

This MRP presents a co-operatively and iteratively designed 3D audio-tactile globe that enables blind and low-vision users to perceive geo-spatial information. Blind and low-vision users rely on learning aids such as 2D-tactile graphics, braille maps and 3D models to learn about geography. I employed co-design as an approach to prototype and evaluate four different iterations of a cross-sensory globe that uses 3D detachable continents to provide geo-spatial haptic information in combination with audio labels. Informed by my co-design and evaluation, I discuss cross-sensory educational aids as an alternative to visually-oriented globes. My findings reveal affordances of 3D-tactile models for conveying concrete features of the Earth (such as varying elevations of landforms) and audio labels for conveying abstract categories about the Earth (such as continent names). I highlight the advantages of longitudinal participatory design that includes the lived experiences and DIY innovations of blind and low-vision users and makers

Multisensory interactive technologies for primary education: From science to technology

While technology is increasingly used in the classroom, we observe at the same time that making teachers and students accept it is more difficult than expected. In this work, we focus on multisensory technologies and we argue that the intersection between current challenges in pedagogical practices and recent scientific evidence opens novel opportunities for these technologies to bring a significant benefit to the learning process. In our view, multisensory technologies are ideal for effectively supporting an embodied and enactive pedagogical approach exploiting the best-suited sensory modality to teach a concept at school. This represents a great opportunity for designing technologies, which are both grounded on robust scientific evidence and tailored to the actual needs of teachers and students. Based on our experience in technology-enhanced learning projects, we propose six golden rules we deem important for catching this opportunity and fully exploiting it