Attentiveness Map Estimation for Haptic Teleoperation of Mobile Robot Obstacle Avoidance and Approach

Haptic feedback can improve safety of teleoperated robots when situational awareness is limited or operators are inattentive. Standard potential field approaches increase haptic resistance as an obstacle is approached, which is desirable when the operator is unaware of the obstacle but undesirable when the movement is intentional, such as when the operator wishes to inspect or manipulate an object. This paper presents a novel haptic teleoperation framework that estimates the operator's attentiveness to obstacles and dampens haptic feedback for intentional movement. A biologically-inspired attention model is developed based on computational working memory theories to integrate visual saliency estimation with spatial mapping. The attentiveness map is generated in real-time, and our system renders lower haptic forces for obstacles that the operator is estimated to be aware of. Experimental results in simulation show that the proposed framework outperforms haptic teleoperation without attentiveness estimation in terms of task performance, robot safety, and user experience.Comment: Accepted by IEEE RA-

Integration of touch attention mechanisms to improve the robotic haptic exploration of surfaces

This text presents the integration of touch attention mechanisms to improve the efficiency of the action-perception loop, typically involved in active haptic exploration tasks of surfaces by robotic hands. The progressive inference of regions of the workspace that should be probed by the robotic system uses information related with haptic saliency extracted from the perceived haptic stimulus map (exploitation) and a “curiosity”-inducing prioritisation based on the reconstruction's inherent uncertainty and inhibition-of-return mechanisms (exploration), modulated by top-down influences stemming from current task objectives, updated at each exploration iteration. This work also extends the scope of the top-down modulation of information presented in a previous work, by integrating in the decision process the influence of shape cues of the current exploration path. The Bayesian framework proposed in this work was tested in a simulation environment. A scenario made of three different materials was explored autonomously by a robotic system. The experimental results show that the system was able to perform three different haptic discontinuity following tasks with a good structural accuracy, demonstrating the selectivity and generalization capability of the attention mechanisms. These experiments confirmed the fundamental contribution of the haptic saliency cues to the success and accuracy of the execution of the tasks

Integration and disruption effects of shape and texture in haptic search

In a search task, where one has to search for the presence of a target among distractors, the target is sometimes easily found, whereas in other searches it is much harder to find. The performance in a search task is influenced by the identity of the target, the identity of the distractors and the differences between the two. In this study, these factors were manipulated by varying the target and distractors in shape (cube or sphere) and roughness (rough or smooth) in a haptic search task. Participants had to grasp a bundle of items and determine as fast as possible whether a predefined target was present or not. It was found that roughness and edges were relatively salient features and the search for the presence of these features was faster than for their absence. If the task was easy, the addition of these features could also disrupt performance, even if they were irrelevant for the search task. Another important finding was that the search for a target that differed in two properties from the distractors was faster than a task with only a single property difference, although this was only found if the two target properties were non-salient. This means that shape and texture can be effectively integrated. Finally, it was found that edges are more beneficial to a search task than disrupting, whereas for roughness this was the other way round

Touch attention Bayesian models for robotic active haptic exploration of heterogeneous surfaces

This work contributes to the development of active haptic exploration strategies of surfaces using robotic hands in environments with an unknown structure. The architecture of the proposed approach consists two main Bayesian models, implementing the touch attention mechanisms of the system. The model πper perceives and discriminates different categories of materials (haptic stimulus) integrating compliance and texture features extracted from haptic sensory data. The model πtar actively infers the next region of the workspace that should be explored by the robotic system, integrating the task information, the permanently updated saliency and uncertainty maps extracted from the perceived haptic stimulus map, as well as, inhibition-of-return mechanisms. The experimental results demonstrate that the Bayesian model πper can be used to discriminate 10 different classes of materials with an average recognition rate higher than 90%. The generalization capability of the proposed models was demonstrated experimentally. The ATLAS robot, in the simulation, was able to perform the following of a discontinuity between two regions made of different materials with a divergence smaller than 1cm (30 trials). The tests were performed in scenarios with 3 different configurations of the discontinuity. The Bayesian models have demonstrated the capability to manage the uncertainty about the structure of the surfaces and sensory noise to make correct motor decisions from haptic percepts

Multimodality in VR: A survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer

Multimodality in {VR}: {A} Survey

Virtual reality has the potential to change the way we create and consume content in our everyday life. Entertainment, training, design and manufacturing, communication, or advertising are all applications that already benefit from this new medium reaching consumer level. VR is inherently different from traditional media: it offers a more immersive experience, and has the ability to elicit a sense of presence through the place and plausibility illusions. It also gives the user unprecedented capabilities to explore their environment, in contrast with traditional media. In VR, like in the real world, users integrate the multimodal sensory information they receive to create a unified perception of the virtual world. Therefore, the sensory cues that are available in a virtual environment can be leveraged to enhance the final experience. This may include increasing realism, or the sense of presence; predicting or guiding the attention of the user through the experience; or increasing their performance if the experience involves the completion of certain tasks. In this state-of-the-art report, we survey the body of work addressing multimodality in virtual reality, its role and benefits in the final user experience. The works here reviewed thus encompass several fields of research, including computer graphics, human computer interaction, or psychology and perception. Additionally, we give an overview of different applications that leverage multimodal input in areas such as medicine, training and education, or entertainment; we include works in which the integration of multiple sensory information yields significant improvements, demonstrating how multimodality can play a fundamental role in the way VR systems are designed, and VR experiences created and consumed

Deep neural network model of haptic saliency

Haptic exploration usually involves stereotypical systematic movements that are adapted to the task. Here we tested whether exploration movements are also driven by physical stimulus features. We designed haptic stimuli, whose surface relief varied locally in spatial frequency, height, orientation, and anisotropy. In Experiment 1, participants subsequently explored two stimuli in order to decide whether they were same or different. We trained a variational autoencoder to predict the spatial distribution of touch duration from the surface relief of the haptic stimuli. The model successfully predicted where participants touched the stimuli. It could also predict participants' touch distribution from the stimulus' surface relief when tested with two new groups of participants, who performed a different task (Exp. 2) or explored different stimuli (Exp. 3). We further generated a large number of virtual surface reliefs (uniformly expressing a certain combination of features) and correlated the model's responses with stimulus properties to understand the model's preferences in order to infer which stimulus features were preferentially touched by participants. Our results indicate that haptic exploratory behavior is to some extent driven by the physical features of the stimuli, with e.g. edge-like structures, vertical and horizontal patterns, and rough regions being explored in more detail