Mobile navigation: a multimodal approach

The functionality and processing power of mobile devices has increased dramatically over the last few years. Location based services and rich interactions are feasible with the majority of smart phones available today. However, whilst the capabilities of current devices afford rich interaction tailored to the user in mobile situations, they are still linked with desktop style interactions. Spatially situated virtual objects are used to represent multiple forms of information. Ranging from nav- igation beacons to places of interest and gaming objects. This thesis gives an review of the current literature of the use of virtual objects and examines the role of vibrotactile feedback for egocentric heading detection for virtual objects. Experiment results are also reported showing users can utilise vibrotactile feedback for heading acquisition. Possible future steps include combining directions and distance information for mobile navigation systems

A hybrid haptic stimulation prosthetic wearable device to recover the missing sensation of the upper limb amputees

A hybrid haptic feedback stimulation system that is capable in sensing the contact pressure, the surface texture, and the temperature, simultaneously, was designed for a prosthetic hand to provide a tactile sensation to amputation patients. In addition, the haptic system was developed to enable the prosthetic’s users to implement withdrawal reflexes due to the thermal noxious stimulus in a quick manner. The re-sensation is achieved by non-invasively stimulating the skin of the patients’ residual limbs, based on the type and the level of tactile signals provided by the sensory system of the prostheses. Accordingly, three stages of design and development were performed to satisfy the research methodology. A vibrotactile prosthetic device, which is designed for the detection of contact pressure and surface texture in upper extremity, represents. While, the design of a novel wearable hybrid pressure-vibration haptic feedback stimulation device for conveying the tactile information regarding the contact pressure between the prosthetic hand and the grasped objects represents the second methodology stage. Lastly, the third stage was achieved by designing a novel hybrid pressure-vibration-temperature feedback stimulation system to provide a huge information regarding the prostheses environment to the users without brain confusing or requiring long pre-training. The main contribution of this work is the development and evaluation of the first step of a novel approach for a lightweight, 7 Degrees-Of-Freedom (DOF) tactile prosthetic arm to perform an effective as well as fast object manipulation and grasping. Furthermore, this study investigates the ability to convey the tactile information about the contact pressure, surface texture, and object temperature to the amputees with high identification accuracy by mean of using the designed hybrid pressure-vibration-temperature feedback wearable device. An evaluation of sensation and response has been conducted on forty healthy volunteers to evaluate the ability of the haptic system to stimulate the human nervous system. The results in term of Stimulus Identification Rate (SIR) show that all the volunteers were correctly able to discriminate the sensation of touch, start of touch, end of touch, and grasping objects. While 94%, 96%, 97%, and 95.24% of the entire stimuli were successfully identified by the volunteers during the experiments of slippage, pressure level, surface texture, and temperature, respectively. The position tracking controller system was designed to synchronize the movements of the volunteers’ elbow joints and the prosthetic’s elbow joint to record the withdrawal reflexes. The results verified the ability of the haptic system to excite the human brain at the abnormal noxious stimulus and enable the volunteers to perform a quick withdrawal reflex within 0.32 sec. The test results and the volunteers' response established evidence that amputees are able to recover their sense of the contact pressure, the surface texture, and the object temperature as well as to perform thermal withdrawal reflexes using the solution developed in this work

HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People with Visual Impairments

Many activities of daily living such as getting dressed, preparing food, wayfinding, or shopping rely heavily on visual information, and the inability to access that information can negatively impact the quality of life for people with vision impairments. While numerous researchers have explored solutions for assisting with visual tasks that can be performed at a distance, such as identifying landmarks for navigation or recognizing people and objects, few have attempted to provide access to nearby visual information through touch. Touch is a highly attuned means of acquiring tactile and spatial information, especially for people with vision impairments. By supporting touch-based access to information, we may help users to better understand how a surface appears (e.g., document layout, clothing patterns), thereby improving the quality of life. To address this gap in research, this dissertation explores methods to augment a visually impaired user’s sense of touch with interactive, real-time computer vision to access information about the physical world. These explorations span three application areas: reading and exploring printed documents, controlling mobile devices, and identifying colors and visual textures. At the core of each application is a system called HandSight that uses wearable cameras and other sensors to detect touch events and identify surface content beneath the user’s finger. To create HandSight, we designed and implemented the physical hardware, developed signal processing and computer vision algorithms, and designed real-time feedback that enables users to interpret visual or digital content. We involve visually impaired users throughout the design and development process, conducting several user studies to assess usability and robustness and to improve our prototype designs. The contributions of this dissertation include: (i) developing and iteratively refining HandSight, a novel wearable system to assist visually impaired users in their daily lives; (ii) evaluating HandSight across a diverse set of tasks, and identifying tradeoffs of a finger-worn approach in terms of physical design, algorithmic complexity and robustness, and usability; and (iii) identifying broader design implications for future wearable systems and for the fields of accessibility, computer vision, augmented and virtual reality, and human-computer interaction

Automatic Posture Correction Utilizing Electrical Muscle Stimulation

Habitually poor posture can lead to repetitive strain injuries that lower an individual\u27s quality of life and productivity. Slouching over computer screens and smart phones, asymmetric weight distribution due to uneven leg loading, and improper loading posture are some of the common examples that lead to postural problems and health ramifications. To help cultivate good postural habits, researchers have proposed slouching, balance, and improper loading posture detection systems that alert users through traditional visual, auditory or vibro-tactile feedbacks when posture requires attention. However, such notifications are disruptive and can be easily ignored. We address these issues with a new physiological feedback system that uses sensors to detect these poor postures, and electrical muscle stimulation to automatically correct the poor posture. We compare our automatic approach against other alternative feedback systems and through different unique contexts. We find that our approach outperformed alternative traditional feedback systems by being faster and more accurate while delivering an equally comfortable user experience

Head-mounted Sensory Augmentation System for Navigation in Low Visibility Environments

Sensory augmentation can be used to assist in some tasks where sensory information is limited or sparse. This thesis focuses on the design and investigation of a head-mounted vibrotactile sensory augmentation interface to assist navigation in low visibility environments such as firefighters’ navigation or travel aids for visually impaired people. A novel head-mounted vibrotactile interface comprising a 1-by-7 vibrotactile display worn on the forehead is developed. A series of psychophysical studies is carried out with this display to (1) determine the vibrotactile absolute threshold, (2) investigate the accuracy of vibrotactile localization, and (3) evaluate the funneling illusion and apparent motion as sensory phenomena that could be used to communicate navigation signals. The results of these studies provide guidelines for the design of head-mounted interfaces. A 2nd generation head-mounted sensory augmentation interface called the Mark-II Tactile Helmet is developed for the application of firefighters’ navigation. It consists of a ring of ultrasound sensors mounted to the outside of a helmet, a microcontroller, two batteries and a refined vibrotactile display composed of seven vibration motors based on the results of the aforementioned psychophysical studies. A ‘tactile language’, that is, a set of distinguishable vibrotactile patterns, is developed for communicating navigation commands to the Mark-II Tactile Helmet. Four possible combinations of two command presentation modes (continuous, discrete) and two command types (recurring, single) are evaluated for their effectiveness in guiding users along a virtual wall in a structured environment. Continuous and discrete presentation modes use spatiotemporal patterns that induce the experience of apparent movement and discrete movement on the forehead, respectively. The recurring command type presents the tactile command repeatedly with an interval between patterns of 500 ms while the single command type presents the tactile command just once when there is a change in the command. The effectiveness of this tactile language is evaluated according to the objective measures of the users’ walking speed and the smoothness of their trajectory parallel to the virtual wall and subjective measures of utility and comfort employing Likert-type rating scales. The Recurring Continuous (RC) commands that exploit the phenomena of apparent motion are most effective in generating efficient routes and fast travel, and are most preferred. Finally, the optimal tactile language (RC) is compared with audio guidance using verbal instructions to investigate effectiveness in delivering navigation commands. The results show that haptic guidance leads to better performance as well as lower cognitive workload compared to auditory feedback. This research demonstrates that a head-mounted sensory augmentation interface can enhance spatial awareness in low visibility environments and could help firefighters’ navigation by providing them with supplementary sensory information

Augmenting communication technologies with non-primary sensory modalities

Humans combine their senses to enhance the world around them. While computers have evolved to reflect these sensory demands, only the primary senses of vision and audition (and to an extent, touch) are used in modern communication. This thesis investigated how additional information, such as emotion and navigational assistance, might be communicated using technology-based implementations of sensory displays that output the non-primary modalities of smell, vibrotactile touch, and thermo-touch. This thesis explored using a portable atomiser sprayer to deliver emotional information via smell to mobile phone users, a ring-shaped device worn on the finger to display emotional information using vibration and colours, and an array of thermoelectric coolers worn on the arm to create temperature sensations. Additionally, this thesis explored two methods of signalling temperature using the thermal implementation, and finally, used it in a controlled study to augment the perceived emotion of text messages using temperature. There were challenges with using some of these implementations to display information. Smells produced with the scent technology were ambiguous and highly cognitive, and poor delivery to the user produced undesirable cross-adaption effects when smells lingered and mixed in the environment. The device used to communicate vibrotactile and colour lighting cues neutralized emotions in text messages. Furthermore, temperature pattern discrimination using the thermal implementation was difficult due to non-linear interaction effects that occurred on the skin’s surface, as well as latency resulting from the thermal neurological pathway and the technology used to heat and cool the skin. However, the thermal implementation enabled more accurate user discrimination between thermal signals than what a single stimulator design provided. Furthermore, the utility of continuous thermal feedback, in the context of spatial navigation, was demonstrated, which improved user performance compared to when the user was not presented with any thermal information. Finally, temperature was demonstrated to elicit arousal reactions across subjects using the thermal implementation, and could augment the arousal of text messages, especially when the content of the message was strongly neutral. However, no similar statistical significance was observed with valence, demonstrating the complex implications of using thermal cues to convey emotional information

Touching is believing: creating illusions and feeling of embodiment with mid-air haptic technology

Over the last two decades, the sense of touch has received new attention from the scientific community.Several haptic devices have been developed to address the complexity of the sense of touch, the latest addition being mid-air (contactless) haptic technology. An interesting series of previous research has suggested an easier way to tackle the complexity of designing convincing tactile sensations by exploiting tactile illusions. Tactile illusions rely on perceptual shortcuts based on the psychophysics of the tactile receptors. Currently, studies exploring the perceptual space of mid-air haptics and its applicability in the tactile illusions field are still limited in number. This thesis aims to contribute to the field of Human-Computer Interaction (HCI) by investigating the perceptual design space of ultrasonic mid-air haptics technology. Specifically, in a first set of three studies, we investigate the absolute thresholds (minimal amount of a property of astimulus that a user can detect) for control points (CP) at different frequencies on the hand and arm (Study 1). Then we investigate the optimal sampling rate needed to drive the device in an optimal fashion and its relationship with shape size (Study 2). Next, we apply a new technique to increase users’ performance in a shape discrimination task (Study 3). In Study 4, we start the exploration of a tactile illusion of movement using contact touch and later, we apply a similar procedure to investigate the feasibility of creating a tactile illusion of movement between the two non-interconnected hands by using mid-air touch (Study 5). Finally, in Study 6, we explore our sense of touch in VR, while providing an illusion of rain drops through mid-air haptics, to recreate a virtual hand illusion (VHI) to explore the boundaries of our sense of embodiment. Therefore, the contribution of this work is threefold: a) we contribute by adding new knowledge on the psychophysical space for mid-air haptics, b) we test the potential to create realistic tactile sensations by exploiting tactile illusions with mid-air haptic technology, and c) we demonstrate how tactile illusions mediated by mid-air haptics can convey a sense of embodiment in VR environments

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 13th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2022, held in Hamburg, Germany, in May 2022. The 36 regular papers included in this book were carefully reviewed and selected from 129 submissions. They were organized in topical sections as follows: haptic science; haptic technology; and haptic applications

Blending the Material and Digital World for Hybrid Interfaces

The development of digital technologies in the 21st century is progressing continuously and new device classes such as tablets, smartphones or smartwatches are finding their way into our everyday lives. However, this development also poses problems, as these prevailing touch and gestural interfaces often lack tangibility, take little account of haptic qualities and therefore require full attention from their users. Compared to traditional tools and analog interfaces, the human skills to experience and manipulate material in its natural environment and context remain unexploited. To combine the best of both, a key question is how it is possible to blend the material world and digital world to design and realize novel hybrid interfaces in a meaningful way. Research on Tangible User Interfaces (TUIs) investigates the coupling between physical objects and virtual data. In contrast, hybrid interfaces, which specifically aim to digitally enrich analog artifacts of everyday work, have not yet been sufficiently researched and systematically discussed. Therefore, this doctoral thesis rethinks how user interfaces can provide useful digital functionality while maintaining their physical properties and familiar patterns of use in the real world. However, the development of such hybrid interfaces raises overarching research questions about the design: Which kind of physical interfaces are worth exploring? What type of digital enhancement will improve existing interfaces? How can hybrid interfaces retain their physical properties while enabling new digital functions? What are suitable methods to explore different design? And how to support technology-enthusiast users in prototyping? For a systematic investigation, the thesis builds on a design-oriented, exploratory and iterative development process using digital fabrication methods and novel materials. As a main contribution, four specific research projects are presented that apply and discuss different visual and interactive augmentation principles along real-world applications. The applications range from digitally-enhanced paper, interactive cords over visual watch strap extensions to novel prototyping tools for smart garments. While almost all of them integrate visual feedback and haptic input, none of them are built on rigid, rectangular pixel screens or use standard input modalities, as they all aim to reveal new design approaches. The dissertation shows how valuable it can be to rethink familiar, analog applications while thoughtfully extending them digitally. Finally, this thesis’ extensive work of engineering versatile research platforms is accompanied by overarching conceptual work, user evaluations and technical experiments, as well as literature reviews.Die Durchdringung digitaler Technologien im 21. Jahrhundert schreitet stetig voran und neue Geräteklassen wie Tablets, Smartphones oder Smartwatches erobern unseren Alltag. Diese Entwicklung birgt aber auch Probleme, denn die vorherrschenden berührungsempfindlichen Oberflächen berücksichtigen kaum haptische Qualitäten und erfordern daher die volle Aufmerksamkeit ihrer Nutzer:innen. Im Vergleich zu traditionellen Werkzeugen und analogen Schnittstellen bleiben die menschlichen Fähigkeiten ungenutzt, die Umwelt mit allen Sinnen zu begreifen und wahrzunehmen. Um das Beste aus beiden Welten zu vereinen, stellt sich daher die Frage, wie neuartige hybride Schnittstellen sinnvoll gestaltet und realisiert werden können, um die materielle und die digitale Welt zu verschmelzen. In der Forschung zu Tangible User Interfaces (TUIs) wird die Verbindung zwischen physischen Objekten und virtuellen Daten untersucht. Noch nicht ausreichend erforscht wurden hingegen hybride Schnittstellen, die speziell darauf abzielen, physische Gegenstände des Alltags digital zu erweitern und anhand geeigneter Designparameter und Entwurfsräume systematisch zu untersuchen. In dieser Dissertation wird daher untersucht, wie Materialität und Digitalität nahtlos ineinander übergehen können. Es soll erforscht werden, wie künftige Benutzungsschnittstellen nützliche digitale Funktionen bereitstellen können, ohne ihre physischen Eigenschaften und vertrauten Nutzungsmuster in der realen Welt zu verlieren. Die Entwicklung solcher hybriden Ansätze wirft jedoch übergreifende Forschungsfragen zum Design auf: Welche Arten von physischen Schnittstellen sind es wert, betrachtet zu werden? Welche Art von digitaler Erweiterung verbessert das Bestehende? Wie können hybride Konzepte ihre physischen Eigenschaften beibehalten und gleichzeitig neue digitale Funktionen ermöglichen? Was sind geeignete Methoden, um verschiedene Designs zu erforschen? Wie kann man Technologiebegeisterte bei der Erstellung von Prototypen unterstützen? Für eine systematische Untersuchung stützt sich die Arbeit auf einen designorientierten, explorativen und iterativen Entwicklungsprozess unter Verwendung digitaler Fabrikationsmethoden und neuartiger Materialien. Im Hauptteil werden vier Forschungsprojekte vorgestellt, die verschiedene visuelle und interaktive Prinzipien entlang realer Anwendungen diskutieren. Die Szenarien reichen von digital angereichertem Papier, interaktiven Kordeln über visuelle Erweiterungen von Uhrarmbändern bis hin zu neuartigen Prototyping-Tools für intelligente Kleidungsstücke. Um neue Designansätze aufzuzeigen, integrieren nahezu alle visuelles Feedback und haptische Eingaben, um Alternativen zu Standard-Eingabemodalitäten auf starren Pixelbildschirmen zu schaffen. Die Dissertation hat gezeigt, wie wertvoll es sein kann, bekannte, analoge Anwendungen zu überdenken und sie dabei gleichzeitig mit Bedacht digital zu erweitern. Dabei umfasst die vorliegende Arbeit sowohl realisierte technische Forschungsplattformen als auch übergreifende konzeptionelle Arbeiten, Nutzerstudien und technische Experimente sowie die Analyse existierender Forschungsarbeiten

Haptic Media Scenes

The aim of this thesis is to apply new media phenomenological and enactive embodied cognition approaches to explain the role of haptic sensitivity and communication in personal computer environments for productivity. Prior theory has given little attention to the role of haptic senses in influencing cognitive processes, and do not frame the richness of haptic communication in interaction design—as haptic interactivity in HCI has historically tended to be designed and analyzed from a perspective on communication as transmissions, sending and receiving haptic signals. The haptic sense may not only mediate contact confirmation and affirmation, but also rich semiotic and affective messages—yet this is a strong contrast between this inherent ability of haptic perception, and current day support for such haptic communication interfaces. I therefore ask: How do the haptic senses (touch and proprioception) impact our cognitive faculty when mediated through digital and sensor technologies? How may these insights be employed in interface design to facilitate rich haptic communication? To answer these questions, I use theoretical close readings that embrace two research fields, new media phenomenology and enactive embodied cognition. The theoretical discussion is supported by neuroscientific evidence, and tested empirically through case studies centered on digital art. I use these insights to develop the concept of the haptic figura, an analytical tool to frame the communicative qualities of haptic media. The concept gauges rich machine- mediated haptic interactivity and communication in systems with a material solution supporting active haptic perception, and the mediation of semiotic and affective messages that are understood and felt. As such the concept may function as a design tool for developers, but also for media critics evaluating haptic media. The tool is used to frame a discussion on opportunities and shortcomings of haptic interfaces for productivity, differentiating between media systems for the hand and the full body. The significance of this investigation is demonstrating that haptic communication is an underutilized element in personal computer environments for productivity and providing an analytical framework for a more nuanced understanding of haptic communication as enabling the mediation of a range of semiotic and affective messages, beyond notification and confirmation interactivity