BRAILLESHAPES : efficient text input on smartwatches for blind people

Tese de Mestrado, Engenharia Informática, 2023, Universidade de Lisboa, Faculdade de CiênciasMobile touchscreen devices like smartphones or smartwatches are a predominant part of our lives. They have evolved, and so have their applications. Due to the constant growth and advancements in technology, using such devices as a means to accomplish a vast amount of tasks has become common practice. Nonetheless, relying on touch-based interactions, requiring good spatial ability and memorization inherent to mobile devices, and lacking sufficient tactile cues, makes these devices visually demanding, thus providing a strenuous interaction modality for visually impaired people. In scenarios occurring in movement-based contexts or where onehanded use is required, it is even more apparent. We believe devices like smartwatches can provide numerous advantages when addressing such topics. However, they lack accessible solutions for several tasks, with most of the existing ones for mobile touchscreen devices targeting smartphones. With communication being of the utmost importance and intrinsic to humankind, one task, in particular, for which it is imperative to provide solutions addressing its surrounding accessibility concerns is text entry. Since Braille is a reading standard for blind people and provided positive results in prior work regarding accessible text entry approaches, we believe using it as the basis for an accessible text entry solution can help solidify a standardization for this type of interaction modality. It can also allow users to leverage previous knowledge, reducing possible extra cognitive load. Yet, even though Braille-based chording solutions achieved good results, due to the reduced space of the smartwatch’s touchscreen, a tapping approach is not the most feasible. Hence, we found the best option to be a gesture-based solution. Therefore, with this thesis, we explored and validated the concept and feasibility of Braille-based shapes as the foundation for an accessible gesture-based smartwatch text entry method for visually impaired people

Exploring the Use of Wearables to develop Assistive Technology for Visually Impaired People

This thesis explores the usage of two prominent wearable devices to develop assistive technology for users who are visually impaired. Specifically, the work in this thesis aims at improving the quality of life of users who are visually impaired by improving their mobility and ability to socially interact with others. We explore the use of a smart watch for creating low-cost spatial haptic applications. This app explores the use of haptic feedback provided using a smartwatch and smartphone to provide navigation instructions that let visually impaired people safely traverse a large open space. This spatial feedback guides them to walk on a straight path from source to destination by avoiding veering. Exploring the paired interaction between a Smartphone and a Smartwatch, helped to overcome the limitation that smart devices have only single haptic actuator.We explore the use of a head-mounted display to enhance social interaction by helping people with visual impairments align their head towards a conversation partner as well as maintain personal space during a conversation. Audio feedback is provided to the users guiding them to achieve effective face-to-face communication. A qualitative study of this method shows the effectiveness of the application and explains how it helps visually impaired people to perceive non-verbal cues and feel more engaged and assertive in social interactions

Design and Effect of Continuous Wearable Tactile Displays

Our sense of touch is one of our core senses and while not as information rich as sight and hearing, it tethers us to reality. Our skin is the largest sensory organ in our body and we rely on it so much that we don\u27t think about it most of the time. Tactile displays - with the exception of actuators for notifications on smartphones and smartwatches - are currently understudied and underused. Currently tactile cues are mostly used in smartphones and smartwatches to notify the user of an incoming call or text message. Specifically continuous displays - displays that do not just send one notification but stay active for an extended period of time and continuously communicate information - are rarely studied. This thesis aims at exploring the utilization of our vibration perception to create continuous tactile displays. Transmitting a continuous stream of tactile information to a user in a wearable format can help elevate tactile displays from being mostly used for notifications to becoming more like additional senses enabling us to perceive our environment in new ways. This work provides a serious step forward in design, effect and use of continuous tactile displays and their use in human-computer interaction. The main contributions include: Exploration of Continuous Wearable Tactile Interfaces This thesis explores continuous tactile displays in different contexts and with different types of tactile information systems. The use-cases were explored in various domains for tactile displays - Sports, Gaming and Business applications. The different types of continuous tactile displays feature one- or multidimensional tactile patterns, temporal patterns and discrete tactile patterns. Automatic Generation of Personalized Vibration Patterns In this thesis a novel approach of designing vibrotactile patterns without expert knowledge by leveraging evolutionary algorithms to create personalized vibration patterns - is described. This thesis presents the design of an evolutionary algorithm with a human centered design generating abstract vibration patterns. The evolutionary algorithm was tested in a user study which offered evidence that interactive generation of abstract vibration patterns is possible and generates diverse sets of vibration patterns that can be recognized with high accuracy. Passive Haptic Learning for Vibration Patterns Previous studies in passive haptic learning have shown surprisingly strong results for learning Morse Code. If these findings could be confirmed and generalized, it would mean that learning a new tactile alphabet could be made easier and learned in passing. Therefore this claim was investigated in this thesis and needed to be corrected and contextualized. A user study was conducted to study the effects of the interaction design and distraction tasks on the capability to learn stimulus-stimulus-associations with Passive Haptic Learning. This thesis presents evidence that Passive Haptic Learning of vibration patterns induces only a marginal learning effect and is not a feasible and efficient way to learn vibration patterns that include more than two vibrations. Influence of Reference Frames for Spatial Tactile Stimuli Designing wearable tactile stimuli that contain spatial information can be a challenge due to the natural body movement of the wearer. An important consideration therefore is what reference frame to use for spatial cues. This thesis investigated allocentric versus egocentric reference frames on the wrist and compared them for induced cognitive load, reaction time and accuracy in a user study. This thesis presents evidence that using an allocentric reference frame drastically lowers cognitive load and slightly lowers reaction time while keeping the same accuracy as an egocentric reference frame, making a strong case for the utilization of allocentric reference frames in tactile bracelets with several tactile actuators

Haptic feedback to gaze events

Eyes are the window to the world, and most of the input from the surrounding environment is captured through the eyes. In Human-Computer Interaction too, gaze based interactions are gaining prominence, where the user’s gaze acts as an input to the system. Of late portable and inexpensive eye-tracking devices have made inroads in the market, opening up wider possibilities for interacting with a gaze. However, research on feedback to the gaze-based events is limited. This thesis proposes to study vibrotactile feedback to gaze-based interactions. This thesis presents a study conducted to evaluate different types of vibrotactile feedback and their role in response to a gaze-based event. For this study, an experimental setup was designed wherein when the user fixated the gaze on a functional object, vibrotactile feedback was provided either on the wrist or on the glasses. The study seeks to answer questions such as the helpfulness of vibrotactile feedback in identifying functional objects, user preference for the type of vibrotactile feedback, and user preference of the location of the feedback. The results of this study indicate that vibrotactile feedback was an important factor in identifying the functional object. The preference for the type of vibrotactile feedback was somewhat inconclusive as there were wide variations among the users over the type of vibrotactile feedback. The personal preference largely influenced the choice of location for receiving the feedback

Using wrist vibrations to guide hand movement and whole body navigation

International audienceIn the absence of vision, mobility and orientation are challenging. Audio and tactile feedback can be used to guide visually impaired people. In this paper, we present two complementary studies on the use of vibrational cues for hand guidance during the exploration of itineraries on a map, and whole body-guidance in a virtual environment. Concretely, we designed wearable Arduino bracelets integrating a vibratory motor producing multiple patterns of pulses. In a first study, this bracelet was used for guiding the hand along unknown routes on an interactive tactile map. A wizard-of-Oz study with six blindfolded participants showed that tactons, vibrational patterns, may be more efficient than audio cues for indicating directions. In a second study, this bracelet was used by blindfolded participants to navigate in a virtual environment. The results presented here show that it is possible to significantly decrease travel distance with vibrational cues. To sum up, these preliminary but complementary studies suggest the interest of vibrational feedback in assistive technology for mobility and orientation for blind people

Design and Evaluation of Biofeedback:Interfaces for Awareness and Regulation of Affect

Biofeedback interfaces enable dynamic representations of bodily data using sensors and actuators to actively control complex physiological activities. These provide individuals with access to their psychophysiological processes, help regulate bodily responses, and have been shown to have positive effects on affective health and wellbeing. Traditionally biofeedback has been provided using audiovisual modality whose understanding usually required technical input from physicians. There are still a limited number of biofeedback interfaces that have been deployed from the lab settings to everyday lives. Specifically, there is a limited focus on low-cost, non-screen based, emerging alternative technologies that could support biosensory information in different ways so that users themselves can understand it. To address these challenges, this thesis engages in the design and evaluation of low-cost, wearable smart materials and actuators to support awareness and regulation of affect. The thesis presents six studies describing them. The first exploration of smart materials and actuators helped in unpacking their material qualities. These include responsiveness, duration, rhythm, aliveness, and range, which led to the design of six wearable visual and haptic interfaces representing physiological arousal. By evaluating the six interfaces in daily life settings, the thesis' findings have shown how the material-driven qualities of the interfaces shape people’s awareness of emotions in different ways starting with reflexivity, emotion identification, and finally, its attribution. This thesis then presents the design of the ThermoPixels toolkit containing digital and physical materials. The toolkit is evaluated by involving users in the design of affective displays for arousal. Findings reveal two distinct motivations for designing physiological arousal interfaces, i.e., awareness and regulation. Analysis of both types of representations helped study their qualities and the role of colors and shapes for personalizing interfaces for awareness and regulation of arousal, i.e., awareness of increased arousal can be supported by angular shapes, warm colors, and rich patterns and regulation of high arousal can be supported by round shapes, cool colors, and light patterns. Moving forward, the thesis engages in the exploration of heart rate variability to regulate affect. It introduces a mixed-methods approach to compare and evaluate wearable heart rate variability sensors in terms of data quality and user acceptance. Following heart rate variability exploration, the thesis involves users in the design of vibrotactile and temperature patterns for affect regulation and demonstrates the value of personalized haptic patterns in regulating affect as measured by self-reported forms and heart rate variability. Interviews with the haptic group help study haptic patterns' experiential qualities and participants' experiences. Between subjects analysis indicates that subjective and objective measures of anxiety and stress decreased under haptic patterns than without and that low frequency vibration was the most effective pattern for stress regulation. The contribution of this work includes unpacking experiential qualities of high - low frequency vibration and warm - cool thermal patterns for affect regulation by engaging users in their design and guidelines for designing these patterns. Finally, two visual and haptic wearable smartwatch apps i.e., Breathe and Heart are designed for affect regulation. These utilize slow bodily rhythms of breathing and heartbeat and are evaluated in daily life under everyday life situations of high arousal negative affect. Findings show the value of technology-delivered interventions in supporting affect regulation that can augment prior strategies being implemented by individuals in their daily lives. The thesis is concluded with a discussion of research contributions and future directions

Biosensing and Actuation—Platforms Coupling Body Input-Output Modalities for Affective Technologies

Research in the use of ubiquitous technologies, tracking systems and wearables within mental health domains is on the rise. In recent years, affective technologies have gained traction and garnered the interest of interdisciplinary fields as the research on such technologies matured. However, while the role of movement and bodily experience to affective experience is well-established, how to best address movement and engagement beyond measuring cues and signals in technology-driven interactions has been unclear. In a joint industry-academia effort, we aim to remodel how affective technologies can help address body and emotional self-awareness. We present an overview of biosignals that have become standard in low-cost physiological monitoring and show how these can be matched with methods and engagements used by interaction designers skilled in designing for bodily engagement and aesthetic experiences. Taking both strands of work together offers unprecedented design opportunities that inspire further research. Through first-person soma design, an approach that draws upon the designer’s felt experience and puts the sentient body at the forefront, we outline a comprehensive work for the creation of novel interactions in the form of couplings that combine biosensing and body feedback modalities of relevance to affective health. These couplings lie within the creation of design toolkits that have the potential to render rich embodied interactions to the designer/user. As a result we introduce the concept of “orchestration”. By orchestration, we refer to the design of the overall interaction: coupling sensors to actuation of relevance to the affective experience; initiating and closing the interaction; habituating; helping improve on the users’ body awareness and engagement with emotional experiences; soothing, calming, or energising, depending on the affective health condition and the intentions of the designer. Through the creation of a range of prototypes and couplings we elicited requirements on broader orchestration mechanisms. First-person soma design lets researchers look afresh at biosignals that, when experienced through the body, are called to reshape affective technologies with novel ways to interpret biodata, feel it, understand it and reflect upon our bodies

Exploring Personalized Vibrotactile and Thermal Patterns for Affect Regulation

The growing HCI interest in wellbeing has led to the emerging area of haptics for affect regulation. In such technologies, distinct haptic patterns are usually designed by researchers; however, current work provides a limited reflection on the rationale for the implemented patterns or the choice of haptic modality. We also know little about how people may benefit from engagement in designing such patterns and what design principles underpin them. We explored vibrotactile and thermal modalities to address these gaps and report on a study with 23 participants. These created haptic patterns for affect regulation during stress elicitation. Findings indicate that subjective and objective measures of anxiety and stress were lower in participants who received haptic patterns than those who did not, and highlighted key experiential qualities of vibrotactile and thermal patterns, and their potential for affect regulation. These open up new design opportunities for affect regulation technologies, including supporting implicit affect regulation through entrainment of slow bodily rhythms, decoupling it from predominant vibrotactile modality, designing thermal biofeedback patterns, and supporting personalized and adaptive patterns