Somatic ABC's: A Theoretical Framework for Designing, Developing and Evaluating the Building Blocks of Touch-Based Information Delivery

abstract: Situations of sensory overload are steadily becoming more frequent as the ubiquity of technology approaches reality--particularly with the advent of socio-communicative smartphone applications, and pervasive, high speed wireless networks. Although the ease of accessing information has improved our communication effectiveness and efficiency, our visual and auditory modalities--those modalities that today's computerized devices and displays largely engage--have become overloaded, creating possibilities for distractions, delays and high cognitive load; which in turn can lead to a loss of situational awareness, increasing chances for life threatening situations such as texting while driving. Surprisingly, alternative modalities for information delivery have seen little exploration. Touch, in particular, is a promising candidate given that it is our largest sensory organ with impressive spatial and temporal acuity. Although some approaches have been proposed for touch-based information delivery, they are not without limitations including high learning curves, limited applicability and/or limited expression. This is largely due to the lack of a versatile, comprehensive design theory--specifically, a theory that addresses the design of touch-based building blocks for expandable, efficient, rich and robust touch languages that are easy to learn and use. Moreover, beyond design, there is a lack of implementation and evaluation theories for such languages. To overcome these limitations, a unified, theoretical framework, inspired by natural, spoken language, is proposed called Somatic ABC's for Articulating (designing), Building (developing) and Confirming (evaluating) touch-based languages. To evaluate the usefulness of Somatic ABC's, its design, implementation and evaluation theories were applied to create communication languages for two very unique application areas: audio described movies and motor learning. These applications were chosen as they presented opportunities for complementing communication by offloading information, typically conveyed visually and/or aurally, to the skin. For both studies, it was found that Somatic ABC's aided the design, development and evaluation of rich somatic languages with distinct and natural communication units.Dissertation/ThesisPh.D. Computer Science 201

Neuromorphic vibrotactile stimulation of fingertips for encoding object stiffness in telepresence sensory substitution and augmentation applications

We present a tactile telepresence system for real-time transmission of information about object stiffness to the human fingertips. Experimental tests were performed across two laboratories (Italy and Ireland). In the Italian laboratory, a mechatronic sensing platform indented different rubber samples. Information about rubber stiffness was converted into on-off events using a neuronal spiking model and sent to a vibrotactile glove in the Irish laboratory. Participants discriminated the variation of the stiffness of stimuli according to a two-alternative forced choice protocol. Stiffness discrimination was based on the variation of the temporal pattern of spikes generated during the indentation of the rubber samples. The results suggest that vibrotactile stimulation can effectively simulate surface stiffness when using neuronal spiking models to trigger vibrations in the haptic interface. Specifically, fractional variations of stiffness down to 0.67 were significantly discriminated with the developed neuromorphic haptic interface. This is a performance comparable, though slightly worse, to the threshold obtained in a benchmark experiment evaluating the same set of stimuli naturally with the own hand. Our paper presents a bioinspired method for delivering sensory feedback about object properties to human skin based on contingency-mimetic neuronal models, and can be useful for the design of high performance haptic devices

Principles and Guidelines for Advancement of Touchscreen-Based Non-visual Access to 2D Spatial Information

Graphical materials such as graphs and maps are often inaccessible to millions of blind and visually-impaired (BVI) people, which negatively impacts their educational prospects, ability to travel, and vocational opportunities. To address this longstanding issue, a three-phase research program was conducted that builds on and extends previous work establishing touchscreen-based haptic cuing as a viable alternative for conveying digital graphics to BVI users. Although promising, this approach poses unique challenges that can only be addressed by schematizing the underlying graphical information based on perceptual and spatio-cognitive characteristics pertinent to touchscreen-based haptic access. Towards this end, this dissertation empirically identified a set of design parameters and guidelines through a logical progression of seven experiments. Phase I investigated perceptual characteristics related to touchscreen-based graphical access using vibrotactile stimuli, with results establishing three core perceptual guidelines: (1) a minimum line width of 1mm should be maintained for accurate line-detection (Exp-1), (2) a minimum interline gap of 4mm should be used for accurate discrimination of parallel vibrotactile lines (Exp-2), and (3) a minimum angular separation of 4mm should be used for accurate discrimination of oriented vibrotactile lines (Exp-3). Building on these parameters, Phase II studied the core spatio-cognitive characteristics pertinent to touchscreen-based non-visual learning of graphical information, with results leading to the specification of three design guidelines: (1) a minimum width of 4mm should be used for supporting tasks that require tracing of vibrotactile lines and judging their orientation (Exp-4), (2) a minimum width of 4mm should be maintained for accurate line tracing and learning of complex spatial path patterns (Exp-5), and (3) vibrotactile feedback should be used as a guiding cue to support the most accurate line tracing performance (Exp-6). Finally, Phase III demonstrated that schematizing line-based maps based on these design guidelines leads to development of an accurate cognitive map. Results from Experiment-7 provide theoretical evidence in support of learning from vision and touch as leading to the development of functionally equivalent amodal spatial representations in memory. Findings from all seven experiments contribute to new theories of haptic information processing that can guide the development of new touchscreen-based non-visual graphical access solutions

Investigating the effect of sensory concurrency on learning haptic spatiotemporal signals

A new generation of multimodal interfaces and interactions is emerging. Drawing on the principles of Sensory Substitution and Augmentation Devices (SSADs), these new interfaces offer the potential for rich, immersive human-computer interactions, but are difficult to design well, and take time to master, creating significant barriers towards wider adoption. Following a review of the literature surrounding existing SSADs, their metrics for success and their growing influence on interface design in Human Computer Interaction, we present a medium term (4-day) study comparing the effectiveness of various combinations of visual and haptic feedback (sensory concurrencies) in preparing users to perform a virtual maze navigation task using haptic feedback alone. Participants navigated 12 mazes in each of 3 separate sessions under a specific combination of visual and haptic feedback, before performing the same task using the haptic feedback alone. Visual sensory deprivation was shown to be inferior to visual & haptic concurrency in enabling haptic signal comprehension, while a new hybridized condition combining reduced visual feedback with the haptic signal was shown to be superior. Potential explanations for the effectiveness of the hybrid mechanism are explored, and the scope and implications of its generalization to new sensory interfaces is presented.PostprintPeer reviewe

The Graphical Access Challenge for People with Visual Impairments: Positions and Pathways Forward

Graphical access is one of the most pressing challenges for individuals who are blind or visually impaired. This chapter discusses some of the factors underlying the graphics access challenge, reviews prior approaches to addressing this long-standing information access barrier, and describes some promising new solutions. We specifically focus on touchscreen-based smart devices, a relatively new class of information access technologies, which our group believes represent an exemplary model of user-centered, needs-based design. We highlight both the challenges and the vast potential of these technologies for alleviating the graphics accessibility gap and share the latest results in this line of research. We close with recommendations on ideological shifts in mindset about how we approach solving this vexing access problem, which will complement both technological and perceptual advancements that are rapidly being uncovered through a growing research community in this domain

Testing a Shape-Changing Haptic Navigation Device With Vision-Impaired and Sighted Audiences in an Immersive Theater Setting

Flatland was an immersive “in-the-wild” experimental theater and technology project, undertaken with the goal of developing systems that could assist “real-world” pedestrian navigation for both vision-impaired (VI) and sighted individuals, while also exploring inclusive and equivalent cultural experiences for VI and sighted audiences. A novel shape-changing handheld haptic navigation device, the “Animotus,” was developed. The device has the ability to modify its form in the user's grasp to communicate heading and proximity to navigational targets. Flatland provided a unique opportunity to comparatively study the use of novel navigation devices with a large group of individuals (79 sighted, 15 VI) who were primarily attending a theater production rather than an experimental study. In this paper, we present our findings on comparing the navigation performance (measured in terms of efficiency, average pace, and time facing targets) and opinions of VI and sighted users of the Animotus as they negotiated the 112 m2 production environment. Differences in navigation performance were nonsignificant across VI and sighted individuals and a similar range of opinions on device function and engagement spanned both groups. We believe more structured device familiarization, particularly for VI users, could improve performance and incorrect technology expectations (such as obstacle avoidance capability), which influenced overall opinion. This paper is intended to aid the development of future inclusive technologies and cultural experiences

Head-mounted Sensory Augmentation Device: Designing a Tactile Language

Abstract—Sensory augmentation operates by synthesizing new information then displaying it through an existing sensory channel and can be used to help people with impaired sensing or to assist in tasks where sensory information is limited or sparse, for example, when navigating in a low visibility environment. This paper presents the design of a 2nd generation head-mounted vibrotactile interface as a sensory augmentation prototype designed to present navigation commands that are intuitive, informative and minimize information overload. We describe an experiment in a structured environment in which the user navigates along a virtual wall whilst the position and orientation of the user’s head is tracked in real time by a motion capture system. Navigation commands in the form of vibrotactile feedback are presented according to the user’s distance from the virtual wall and their head orientation. We test the four possible combinations of two command presentation modes (continuous, discrete) and two command types (recurring, single). We evaluated the effectiveness of this ‘tactile language’ according to the users’ walking speed and the smoothness of their trajectory parallel to the virtual wall. Results showed that recurring continuous commands allowed users to navigate with lowest route deviation and highest walking speed. In addition, subjects preferred recurring continuous commands over other commands

Sensorimotor interfaces : towards enactivity in HCI

This thesis explores the application of enactive techniques to human computer interaction, focusing on how devices following ‘sensorimotor’ principles can be blended with interface goals to lead to new perceptual experiences. Building sensorimotor interfaces is an exciting, emerging field of research facing challenges surrounding application, design, training and uptake. To tackle these challenges, this thesis cuts a line of investigation from a review of enactivity in the related field of sensory substitution and augmentation devices, to a schematic taxonomy, model and design guide of ‘the sensorimotor interface’; developed from a theoretically-grounded, enactive approach to cognition. Device, interaction and training guidelines are drawn from this model, formalising the application of the enactive approach to HCI. A readily-available consumer device is then characterised and calibrated in preparation for testing the model validity and associated insights. The process highlights the effects of accessible, easily-implemented calibrations, and the importance of mixed-method approaches in assessing sensorimotor interface potential. The calibrated device is utilised to conduct a detailed, methodological investigation into how concurrently available sensory information affects and contributes to uptake of novel sensorimotor skills. Robust statistical modelling concludes that sensory concurrency has a profound effect on the comprehension and integration of enactive haptic signals, and that efforts to carefully control the nature and degree of sensory concurrency improve user comprehension and enjoyability when engaging with novel sensorimotor tasks, while reducing confusion and stress. The work is concluded by speculation on how the presented derivations, methods and observations can be used to directly influence future sensorimotor interface design in HCI. This thesis therefore constitutes a primer to the principles and history of sensory substitution and augmentation, details the requirements and limitations of the enactive approach in academia and industry, and brings enactivity forward as an accessible, viable and exciting methodology in interaction design