AudioViewer: Learning to Visualize Sounds

A long-standing goal in the field of sensory substitution is to enable sound perception for deaf and hard of hearing (DHH) people by visualizing audio content. Different from existing models that translate to hand sign language, between speech and text, or text and images, we target immediate and low-level audio to video translation that applies to generic environment sounds as well as human speech. Since such a substitution is artificial, without labels for supervised learning, our core contribution is to build a mapping from audio to video that learns from unpaired examples via high-level constraints. For speech, we additionally disentangle content from style, such as gender and dialect. Qualitative and quantitative results, including a human study, demonstrate that our unpaired translation approach maintains important audio features in the generated video and that videos of faces and numbers are well suited for visualizing high-dimensional audio features that can be parsed by humans to match and distinguish between sounds and words. Code and models are available at https://chunjinsong.github.io/audioviewe

Information transmission with a multi-finger tactual display

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1996.Includes bibliographical references.by Hong Zhang Tan.Ph.D

Mood Glove Enhancing mood in film music through haptic sensations for an enriched film experience

PhDThis research explores a new way of enhancing audience experience in film entertainment, presenting the design and implementation of a wearable prototype system that uses haptic sensations to intensify moods in lm music. The aim of this work is to enrich the musical experience of lm audiences and might also have implication on the hearing-impaired, providing them with a new enhanced emotional experience while watching a movie. Although there has been previous work into music displays of a visual and haptic nature, and on the importance of music in film, there is no documented research on musical enhancement experience in film entertainment. This work focuses on the mood conveyed by lm music in order to understand what role it plays in creating the film experience, and also explores the possibility of enhancing those feelings through haptic sensations. Drawing on HCI and interaction design principles, the design of a piece of haptic wearable technology is proposed and used as the tool for user studies. This research contributes to the fields of: HCI, interaction design, user experience design, multimodal interaction, creative technology, wearable technology, haptics, entertainment technology and lm music. This work also provides a set of design suggestions to aid future research and designers of haptic sensations for media enhancement. Proposed guidelines are based on a number of empirical findings that describe and explain aspects of audience emotional response to haptics, providing some first evidence that there is a correlation between vibrotactile stimuli (such as frequency and intensity) and perceived feelings.EPSRC Doctoral Training Centre in Digital Music and Media for the Creative Economy [EP/G03723X/1]

Context-aware gestural interaction in the smart environments of the ubiquitous computing era

A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements for the degree of Doctor of PhilosophyTechnology is becoming pervasive and the current interfaces are not adequate for the interaction with the smart environments of the ubiquitous computing era. Recently, researchers have started to address this issue introducing the concept of natural user interface, which is mainly based on gestural interactions. Many issues are still open in this emerging domain and, in particular, there is a lack of common guidelines for coherent implementation of gestural interfaces. This research investigates gestural interactions between humans and smart environments. It proposes a novel framework for the high-level organization of the context information. The framework is conceived to provide the support for a novel approach using functional gestures to reduce the gesture ambiguity and the number of gestures in taxonomies and improve the usability. In order to validate this framework, a proof-of-concept has been developed. A prototype has been developed by implementing a novel method for the view-invariant recognition of deictic and dynamic gestures. Tests have been conducted to assess the gesture recognition accuracy and the usability of the interfaces developed following the proposed framework. The results show that the method provides optimal gesture recognition from very different view-points whilst the usability tests have yielded high scores. Further investigation on the context information has been performed tackling the problem of user status. It is intended as human activity and a technique based on an innovative application of electromyography is proposed. The tests show that the proposed technique has achieved good activity recognition accuracy. The context is treated also as system status. In ubiquitous computing, the system can adopt different paradigms: wearable, environmental and pervasive. A novel paradigm, called synergistic paradigm, is presented combining the advantages of the wearable and environmental paradigms. Moreover, it augments the interaction possibilities of the user and ensures better gesture recognition accuracy than with the other paradigms