Somatic ABC's: A Theoretical Framework for Designing, Developing and Evaluating the Building Blocks of Touch-Based Information Delivery

abstract: Situations of sensory overload are steadily becoming more frequent as the ubiquity of technology approaches reality--particularly with the advent of socio-communicative smartphone applications, and pervasive, high speed wireless networks. Although the ease of accessing information has improved our communication effectiveness and efficiency, our visual and auditory modalities--those modalities that today's computerized devices and displays largely engage--have become overloaded, creating possibilities for distractions, delays and high cognitive load; which in turn can lead to a loss of situational awareness, increasing chances for life threatening situations such as texting while driving. Surprisingly, alternative modalities for information delivery have seen little exploration. Touch, in particular, is a promising candidate given that it is our largest sensory organ with impressive spatial and temporal acuity. Although some approaches have been proposed for touch-based information delivery, they are not without limitations including high learning curves, limited applicability and/or limited expression. This is largely due to the lack of a versatile, comprehensive design theory--specifically, a theory that addresses the design of touch-based building blocks for expandable, efficient, rich and robust touch languages that are easy to learn and use. Moreover, beyond design, there is a lack of implementation and evaluation theories for such languages. To overcome these limitations, a unified, theoretical framework, inspired by natural, spoken language, is proposed called Somatic ABC's for Articulating (designing), Building (developing) and Confirming (evaluating) touch-based languages. To evaluate the usefulness of Somatic ABC's, its design, implementation and evaluation theories were applied to create communication languages for two very unique application areas: audio described movies and motor learning. These applications were chosen as they presented opportunities for complementing communication by offloading information, typically conveyed visually and/or aurally, to the skin. For both studies, it was found that Somatic ABC's aided the design, development and evaluation of rich somatic languages with distinct and natural communication units.Dissertation/ThesisPh.D. Computer Science 201

Accelerated and improved motor learning and rehabilitation using kinesthetic feedback

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2006.Includes bibliographical references (p. 69-71).About 21 million people in the United States [roughly 8%] have a basic motor skill inability [13], many stemming not from atrophy, but an improper mapping from the brain to the motor system. Devices exist today to aid people in rebuilding their motor system mappings, but do so in bulky, and inconvenient ways, since many of the users have adequate muscle strength, but the inability to control it properly. Hundreds of millions of people in the world participate in the arts, most of which involve motion of some sort. Typically, to become able to properly perform/paint/dance/etc, training is necessary. We learn from visual and auditory feedback, and sometimes, from the touch of a teacher. This research aims to improve the efficacy of such training with robotic touch, to enable people to become better, faster. This research proposes an augmented sensory feedback system - a lightweight comfortable wearable device that utilizes the communication channel of direct touch on the body, to give real-time feedback to the wearer about their performance in motor skill tasks. Using vibrotactile signals to indicate joint error in a user's motion, we enable a user to wear a full-body suit that provides subtle cues for the brain, as they perform a variety of motor skill tasks.(cont.) The hope is that utilizing tactile real-time feedback will act as a dance teacher or physical therapist does: by giving muscle aid through informational touch cues, not only through force or torque. This will enable people to undergo constant therapy/training, over all joints of the body simultaneously. with higher accuracy than a therapist/teacher provides. The device will enable more rapid motor rehabilitation and postural retraining to combat repetitive strain injuries (RSIs). It will also allow allow communication between a motion expert and a student in real-time [by comparing the student's performance to an expert's]. to aid in higher level motor learning skills such as sports and dance. It will function as a tool to accelerate and deepen peoples motor learning capabilities. This thesis focuses on actuator selection and feedback mechanisms for such a suit, in a low-joint-number test, comprising elements of the upper arm. Initial tests on a 5 degree-of-freedom suit show a decrease in motion errors of roughly 21% (p = 0.015), with 15% lower steady-state error (p = 0.007) and a 7% accelerated rate of learning (p = 0.007).by Jeff Lieberman.S.M

ASSESSING THE EMOTIONAL DIMENSIONS OF THE CUTANEOUS-RABBIT EFFECT USING FACES

This research investigated how emotional images affects tactile saltation. More specifically, we combined visual faces with three different emotions: angry, sad and happy with a tactile illusion known as the cutaneous rabbit effect (CRE) that was delivered on participants’ shoulder at different durations. Because some of the faces were political figures, we assessed participant’s political affiliation using Political Typology Quiz. Participants rated their emotional response using the self-assessment manikin (SAM), a three dimensional scale that measures valence, arousal and dominance. For the valence dimension, our results indicate that participants found it less pleasant to see likeable faces sad than any other figure. Data also showed that it was the least pleasant for participants to see a hateful public face when they were angry. Political affiliation and the duration of the CRE both affected the arousal dimension suggesting that liberals seems more concerned by fictional and public figures than political ones while conservatives seems to be more affected by angry faces of political figures when the tactile duration is the longest. Finally, for the dominance dimension, it was found that likeable figures made participants feel more in control of the situation and that dislikeable figures made participants feel controlled

Effects of Visual Locomotion and Tactile Stimuli Duration on the Emotional Dimensions of the Cutaneous Rabbit Illusion

In this study, we assessed the emotional dimensions (valence, arousal, and dominance) of the multimodal visual-cutaneous rabbit effect. Simultaneously to the tactile bursts on the forearm, visual silhouettes of saltatorial animals (rabbit, kangaroo, spider, grasshopper, frog, and flea) were projected on the left arm. Additionally, there were two locomotion conditions: taking-off and landing. The results showed that the valence dimension (happy-unhappy) was only affected by the visual stimuli with no effect of the tactile conditions nor the locomotion phases. Arousal (excited-calm) showed a significant difference for the three tactile conditions with an interaction effect with the locomotion condition. Arousal scores were higher when the taking-off condition was associated with the intermediate duration (24 ms) and when the landing condition was associated with either the shortest duration (12 ms) or the longest duration (48 ms). There was no effect for the dominance dimension. Similar to our previous results, the valence dimension seems to be highly affected by visual information reducing any effect of tactile information, while touch can modulate the arousal dimension. This can be beneficial for designing multimodal interfaces for virtual or augmented reality.acceptedVersionPeer reviewe

Head-mounted Sensory Augmentation System for Navigation in Low Visibility Environments

Sensory augmentation can be used to assist in some tasks where sensory information is limited or sparse. This thesis focuses on the design and investigation of a head-mounted vibrotactile sensory augmentation interface to assist navigation in low visibility environments such as firefighters’ navigation or travel aids for visually impaired people. A novel head-mounted vibrotactile interface comprising a 1-by-7 vibrotactile display worn on the forehead is developed. A series of psychophysical studies is carried out with this display to (1) determine the vibrotactile absolute threshold, (2) investigate the accuracy of vibrotactile localization, and (3) evaluate the funneling illusion and apparent motion as sensory phenomena that could be used to communicate navigation signals. The results of these studies provide guidelines for the design of head-mounted interfaces. A 2nd generation head-mounted sensory augmentation interface called the Mark-II Tactile Helmet is developed for the application of firefighters’ navigation. It consists of a ring of ultrasound sensors mounted to the outside of a helmet, a microcontroller, two batteries and a refined vibrotactile display composed of seven vibration motors based on the results of the aforementioned psychophysical studies. A ‘tactile language’, that is, a set of distinguishable vibrotactile patterns, is developed for communicating navigation commands to the Mark-II Tactile Helmet. Four possible combinations of two command presentation modes (continuous, discrete) and two command types (recurring, single) are evaluated for their effectiveness in guiding users along a virtual wall in a structured environment. Continuous and discrete presentation modes use spatiotemporal patterns that induce the experience of apparent movement and discrete movement on the forehead, respectively. The recurring command type presents the tactile command repeatedly with an interval between patterns of 500 ms while the single command type presents the tactile command just once when there is a change in the command. The effectiveness of this tactile language is evaluated according to the objective measures of the users’ walking speed and the smoothness of their trajectory parallel to the virtual wall and subjective measures of utility and comfort employing Likert-type rating scales. The Recurring Continuous (RC) commands that exploit the phenomena of apparent motion are most effective in generating efficient routes and fast travel, and are most preferred. Finally, the optimal tactile language (RC) is compared with audio guidance using verbal instructions to investigate effectiveness in delivering navigation commands. The results show that haptic guidance leads to better performance as well as lower cognitive workload compared to auditory feedback. This research demonstrates that a head-mounted sensory augmentation interface can enhance spatial awareness in low visibility environments and could help firefighters’ navigation by providing them with supplementary sensory information

Investigating Social Haptic Illusions for Tactile Stroking (SHIFTS)

A common and effective form of social touch is stroking on the forearm. We seek to replicate this stroking sensation using haptic illusions. This work compares two methods that provide sequential discrete stimulation: sequential normal indentation and sequential lateral skin-slip using discrete actuators. Our goals are to understand which form of stimulation more effectively creates a continuous stroking sensation, and how many discrete contact points are needed. We performed a study with 20 participants in which they rated sensations from the haptic devices on continuity and pleasantness. We found that lateral skin-slip created a more continuous sensation, and decreasing the number of contact points decreased the continuity. These results inform the design of future wearable haptic devices and the creation of haptic signals for effective social communication.Comment: To be published in IEEE Haptics Symposium 202

HapticHead - Augmenting Reality via Tactile Cues

Information overload is increasingly becoming a challenge in today's world. Humans have only a limited amount of attention to allocate between sensory channels and tend to miss or misjudge critical sensory information when multiple activities are going on at the same time. For example, people may miss the sound of an approaching car when walking across the street while looking at their smartphones. Some sensory channels may also be impaired due to congenital or acquired conditions. Among sensory channels, touch is often experienced as obtrusive, especially when it occurs unexpectedly. Since tactile actuators can simulate touch, targeted tactile stimuli can provide users of virtual reality and augmented reality environments with important information for navigation, guidance, alerts, and notifications. In this dissertation, a tactile user interface around the head is presented to relieve or replace a potentially impaired visual channel, called \emph{HapticHead}. It is a high-resolution, omnidirectional, vibrotactile display that presents general, 3D directional, and distance information through dynamic tactile patterns. The head is well suited for tactile feedback because it is sensitive to mechanical stimuli and provides a large spherical surface area that enables the display of precise 3D information and allows the user to intuitively rotate the head in the direction of a stimulus based on natural mapping. Basic research on tactile perception on the head and studies on various use cases of head-based tactile feedback are presented in this thesis. Several investigations and user studies have been conducted on (a) the funneling illusion and localization accuracy of tactile stimuli around the head, (b) the ability of people to discriminate between different tactile patterns on the head, (c) approaches to designing tactile patterns for complex arrays of actuators, (d) increasing the immersion and presence level of virtual reality applications, and (e) assisting people with visual impairments in guidance and micro-navigation. In summary, tactile feedback around the head was found to be highly valuable as an additional information channel in various application scenarios. Most notable is the navigation of visually impaired individuals through a micro-navigation obstacle course, which is an order of magnitude more accurate than the previous state-of-the-art, which used a tactile belt as a feedback modality. The HapticHead tactile user interface's ability to safely navigate people with visual impairments around obstacles and on stairs with a mean deviation from the optimal path of less than 6~cm may ultimately improve the quality of life for many people with visual impairments.Die Informationsüberlastung wird in der heutigen Welt zunehmend zu einer Herausforderung. Der Mensch hat nur eine begrenzte Menge an Aufmerksamkeit, die er zwischen den Sinneskanälen aufteilen kann, und neigt dazu, kritische Sinnesinformationen zu verpassen oder falsch einzuschätzen, wenn mehrere Aktivitäten gleichzeitig ablaufen. Zum Beispiel können Menschen das Geräusch eines herannahenden Autos überhören, wenn sie über die Straße gehen und dabei auf ihr Smartphone schauen. Einige Sinneskanäle können auch aufgrund von angeborenen oder erworbenen Erkrankungen beeinträchtigt sein. Unter den Sinneskanälen wird Berührung oft als aufdringlich empfunden, besonders wenn sie unerwartet auftritt. Da taktile Aktoren Berührungen simulieren können, können gezielte taktile Reize den Benutzern von Virtual- und Augmented Reality Anwendungen wichtige Informationen für die Navigation, Führung, Warnungen und Benachrichtigungen liefern. In dieser Dissertation wird eine taktile Benutzeroberfläche um den Kopf herum präsentiert, um einen möglicherweise beeinträchtigten visuellen Kanal zu entlasten oder zu ersetzen, genannt \emph{HapticHead}. Es handelt sich um ein hochauflösendes, omnidirektionales, vibrotaktiles Display, das allgemeine, 3D-Richtungs- und Entfernungsinformationen durch dynamische taktile Muster darstellt. Der Kopf eignet sich gut für taktiles Feedback, da er empfindlich auf mechanische Reize reagiert und eine große sphärische Oberfläche bietet, die die Darstellung präziser 3D-Informationen ermöglicht und es dem Benutzer erlaubt, den Kopf aufgrund der natürlichen Zuordnung intuitiv in die Richtung eines Reizes zu drehen. Grundlagenforschung zur taktilen Wahrnehmung am Kopf und Studien zu verschiedenen Anwendungsfällen von kopfbasiertem taktilem Feedback werden in dieser Arbeit vorgestellt. Mehrere Untersuchungen und Nutzerstudien wurden durchgeführt zu (a) der Funneling Illusion und der Lokalisierungsgenauigkeit von taktilen Reizen am Kopf, (b) der Fähigkeit von Menschen, zwischen verschiedenen taktilen Mustern am Kopf zu unterscheiden, (c) Ansätzen zur Gestaltung taktiler Muster für komplexe Arrays von Aktoren, (d) der Erhöhung des Immersions- und Präsenzgrades von Virtual-Reality-Anwendungen und (e) der Unterstützung von Menschen mit Sehbehinderungen bei der Führung und Mikronavigation. Zusammenfassend wurde festgestellt, dass taktiles Feedback um den Kopf herum als zusätzlicher Informationskanal in verschiedenen Anwendungsszenarien sehr wertvoll ist. Am interessantesten ist die Navigation von sehbehinderten Personen durch einen Mikronavigations-Hindernisparcours, welche um eine Größenordnung präziser ist als der bisherige Stand der Technik, der einen taktilen Gürtel als Feedback-Modalität verwendete. Die Fähigkeit der taktilen Benutzerschnittstelle HapticHead, Menschen mit Sehbehinderungen mit einer mittleren Abweichung vom optimalen Pfad von weniger als 6~cm sicher um Hindernisse und auf Treppen zu navigieren, kann letztendlich die Lebensqualität vieler Menschen mit Sehbehinderungen verbessern

Assisting Navigation and Object Selection with Vibrotactile Cues

Our lives have been drastically altered by information technology in the last decades, leading to evolutionary mismatches between human traits and the modern environment. One particular mismatch occurs when visually demanding information technology overloads the perceptual, cognitive or motor capabilities of the human nervous system. This information overload could be partly alleviated by complementing visual interaction with haptics. The primary aim of this thesis was to investigate how to assist movement control with vibrotactile cues. Vibrotactile cues refer to technologymediated vibrotactile signals that notify users of perceptual events, propose users to make decisions, and give users feedback from actions. To explore vibrotactile cues, we carried out five experiments in two contexts of movement control: navigation and object selection. The goal was to find ways to reduce information load in these tasks, thus helping users to accomplish the tasks more effectively. We employed measurements such as reaction times, error rates, and task completion times. We also used subjective rating scales, short interviews, and free-form participant comments to assess the vibrotactile assisted interactive systems. The findings of this thesis can be summarized as follows. First, if the context of movement control allows the use of both feedback and feedforward cues, feedback cues are a reasonable first option. Second, when using vibrotactile feedforward cues, using low-level abstractions and supporting the interaction with other modalities can keep the information load as low as possible. Third, the temple area is a feasible actuation location for vibrotactile cues in movement control, including navigation cues and object selection cues with head turns. However, the usability of the area depends on contextual factors such as spatial congruency, the actuation device, and the pace of the interaction task