Assisting Human Motion-Tasks with Minimal, Real-time Feedback

Teaching physical motions such as riding, exercising, swimming, etc. to human beings is hard. Coaches face difficulties in communicating their feedback verbally and cannot correct the student mid-action; teaching videos are two dimensional and suffer from perspective distortion. Systems that track a user and provide him real-time feedback have many potential applications: as an aid to the visually challenged, improving rehabilitation, improving exercise routines such as weight training or yoga, teaching new motion tasks, synchronizing motions of multiple actors, etc. It is not easy to deliver real-time feedback in a way that is easy to interpret, yet unobtrusive enough to not distract the user from the motion task. I have developed motion feedback systems that provide real-time feedback to achieve or improve human motion tasks. These systems track the user\u27s actions with simple sensors, and use tiny vibration motors as feedback devices. Vibration motors provide feedback that is both intuitive and minimally intrusive. My systems\u27 designs are simple, flexible, and extensible to large-scale, full-body motion tasks. The systems that I developed as part of this thesis address two classes of motion tasks: configuration tasks and trajectory tasks. Configuration tasks guide the user to a target configuration. My systems for configuration tasks use a motion-capture system to track the user. Configuration-task systems restrict the user\u27s motions to a set of motion primitives, and guide the user to the target configuration by executing a sequence of motion-primitives. Trajectory tasks assume that the user understands the motion task. The systems for trajectory tasks provide corrective feedback that assists the user in improving their performance. This thesis presents the design, implementation, and results of user experiments with the prototype systems I have developed

Enhancing tele-operation - Investigating the effect of sensory feedback on performance

The decline in the number of healthcare service providers in comparison to the growing numbers of service users prompts the development of technologies to improve the efficiency of healthcare services. One such technology which could offer support are assistive robots, remotely tele-operated to provide assistive care and support for older adults with assistive care needs and people living with disabilities. Tele-operation makes it possible to provide human-in-the-loop robotic assistance while also addressing safety concerns in the use of autonomous robots around humans. Unlike many other applications of robot tele-operation, safety is particularly significant as the tele-operated assistive robots will be used in close proximity to vulnerable human users. It is therefore important to provide as much information about the robot (and the robot workspace) as possible to the tele-operators to ensure safety, as well as efficiency. Since robot tele-operation is relatively unexplored in the context of assisted living, this thesis explores different feedback modalities that may be employed to communicate sensor information to tele-operators. The thesis presents research as it transitioned from identifying and evaluating additional feedback modalities that may be used to supplement video feedback, to exploring different strategies for communicating the different feedback modalities. Due to the fact that some of the sensors and feedback needed are not readily available, different design iterations were carried out to develop the necessary hardware and software for the studies carried out. The first human study was carried out to investigate the effect of feedback on tele-operator performance. Performance was measured in terms of task completion time, ease of use of the system, number of robot joint movements, and success or failure of the task. The effect of verbal feedback between the tele-operator and service users was also investigated. Feedback modalities have differing effects on performance metrics and as a result, the choice of optimal feedback may vary from task to task. Results show that participants preferred scenarios with verbal feedback relative to scenarios without verbal feedback, which also reflects in their performance. Gaze metrics from the study also showed that it may be possible to understand how tele-operators interact with the system based on their areas of interest as they carry out tasks. This findings suggest that such studies can be used to improve the design of tele-operation systems.The need for social interaction between the tele-operator and service user suggests that visual and auditory feedback modalities will be engaged as tasks are carried out. This further reduces the number of available sensory modalities through which information can be communicated to tele-operators. A wrist-worn Wi-Fi enabled haptic feedback device was therefore developed and a study was carried out to investigate haptic sensitivities across the wrist. Results suggest that different locations on the wrist have varying sensitivities to haptic stimulation with and without video distraction, duration of haptic stimulation, and varying amplitudes of stimulation. This suggests that dynamic control of haptic feedback can be used to improve haptic perception across the wrist, and it may also be possible to display more than one type of sensor data to tele-operators during a task. The final study carried out was designed to investigate if participants can differentiate between different types of sensor data conveyed through different locations on the wrist via haptic feedback. The effect of increased number of attempts on performance was also investigated. Total task completion time decreased with task repetition. Participants with prior gaming and robot experience had a more significant reduction in total task completion time when compared to participants without prior gaming and robot experience. Reduction in task completion time was noticed for all stages of the task but participants with additional feedback had higher task completion time than participants without supplementary feedback. Reduction in task completion time varied for different stages of the task. Even though gripper trajectory reduced with task repetition, participants with supplementary feedback had longer gripper trajectories than participants without supplementary feedback, while participants with prior gaming experience had shorter gripper trajectories than participants without prior gaming experience. Perceived workload was also found to reduce with task repetition but perceived workload was higher for participants with feedback reported higher perceived workload than participants without feedback. However participants without feedback reported higher frustration than participants without feedback.Results show that the effect of feedback may not be significant where participants can get necessary information from video feedback. However, participants were fully dependent on feedback when video feedback could not provide requisite information needed.The findings presented in this thesis have potential applications in healthcare, and other applications of robot tele-operation and feedback. Findings can be used to improve feedback designs for tele-operation systems to ensure safe and efficient tele-operation. The thesis also provides ways visual feedback can be used with other feedback modalities. The haptic feedback designed in this research may also be used to provide situational awareness for the visually impaired

Cooperative human-robot haptic navigation

International audienceThis paper proposes a novel use of haptic feedback for human navigation with a mobile robot. Assuming that a path-planner has provided a mobile robot with an obstacle-free trajectory, the vehicle must steer the human from an initial to a desired target position by only interacting with him/her via a custom-designed vibro-tactile bracelet. The subject is free to decide his/her own pace and a warning vibrational signal is generated by the bracelet only when a large deviation with respect to the planned trajectory is detected by the vision sensor on-board the robot. This leads to a cooperative navigation system that is less intrusive, more flexible and easy-to-use than the ones existing in literature. The effectiveness of the proposed system is demonstrated via extensive real-world experiments

A hierarchical sensorimotor control framework for human-in-the-loop robotic hands.

Human manual dexterity relies critically on touch. Robotic and prosthetic hands are much less dexterous and make little use of the many tactile sensors available. We propose a framework modeled on the hierarchical sensorimotor controllers of the nervous system to link sensing to action in human-in-the-loop, haptically enabled, artificial hands

A Mechanical Hand-Tracking System with Tactile Feedback Designed for Telemanipulation

: In this paper, we present a mechanical hand-tracking system with tactile feedback designed for fine manipulation in teleoperation scenarios. Alternative tracking methods based on artificial vision and data gloves have become an asset for virtual reality interaction. Yet, occlusions, lack of precision, and the absence of effective haptic feedback beyond vibrotactile still appear as a limit for teleoperation applications. In this work, we propose a methodology to design a linkage mechanism for hand pose tracking purposes, preserving complete finger mobility. Presentation of the method is followed by design and implementation of a working prototype, and by evaluation of the tracking accuracy using optical markers. Moreover, a teleoperation experiment involving a dexterous robotic arm and hand was proposed to ten participants. It investigated the effectiveness and repeatability of the hand tracking with combined haptic feedback during a proposed pick and place manipulation tasks

Development of collaborative strategies in joint action

Many tasks in daily life involve coordinating movements between two or more individuals. A couple of dancers, a team of players, two workers carrying a load or a therapist interacting with a patient are just a few examples. Acting in collaboration or joint action is a crucial human ability, and our sensorimotor system is shaped to support this capability efficiently. When two partners have different goals but may benefit from collaborating, they face the challenge of negotiating a joint strategy. To do this, first and foremost both subjects need to know their partner\u2019s state and current strategy. It is unclear how the collaboration would be affected if information about the partner is unreliable or incomplete. This work intends to investigate the development of collaborative strategies in joint action. To this purpose, I developed a dedicated experimental apparatus and task. I also developed a general computational framework \u2013 based on differential game theory \u2013 for the description and implementation of interactive behaviours of two subjects performing a joint motor task. The model allows to simulate any joint sensorimotor action in which the joint dynamics can be represented as a linear dynamical system and each agent\u2019s task is formulated in terms of a quadratic cost functional. The model also accounts for imperfect information about dyad dynamics and partner\u2019s actions, and can predict the development of joint action through repeated performance. A first experimental study, focused on how the development of joint action is affected by incomplete and unreliable information. We found that information about the partner not only affects the speed at which a collaborative strategy is achieved (less information, slower learning) but also optimality of the collaboration. In particular, when information about the partner is reduced, the learned strategy is characterised by the development of alternating patterns of leader-follower roles, whereas greater information leads to a more synchronous behaviour. Simulations with a computational model based on game theory suggest that synchronous behaviours are close to optimal in a game theoretic sense (Nash equilibrium). The emergence of roles is a compensation strategy which minimises the need to estimate partner\u2019s intentions and is, therefore, more robust to incomplete information. A second study addresses how physical interaction develops between adults with Autism spectrum disorder (ASD) and typically developing subjects. ASD remains mostly a mystery and has therefore generated some theories trying to explain their cognitive disabilities, which involve an impaired ability to interact with other human partners. Although preliminary due to the small number of subjects, our results suggest that ASD subjects display heterogeneity in establishing a collaboration, which can be only partly explained with their ability to perceive haptic force. This work is a first attempt to establish a sensorimotor theory of joint action. It may provide new insights into the development of robots that are capable of establishing optimal collaborations with human partners, for instance in the context of robot-assisted rehabilitation