Spatio-Temporal Avoidance of Predicted Occupancy in Human-Robot Collaboration

This paper addresses human-robot collaboration (HRC) challenges of integrating predictions of human activity to provide a proactive-n-reactive response capability for the robot. Prior works that consider current or predicted human poses as static obstacles are too nearsighted or too conservative in planning, potentially causing delayed robot paths. Alternatively, time-varying prediction of human poses would enable robot paths that avoid anticipated human poses, synchronized dynamically in time and space. Herein, a proactive path planning method, denoted STAP, is presented that uses spatiotemporal human occupancy maps to find robot trajectories that anticipate human movements, allowing robot passage without stopping. In addition, STAP anticipates delays from robot speed restrictions required by ISO/TS 15066 speed and separation monitoring (SSM). STAP also proposes a sampling-based planning algorithm based on RRT* to solve the spatio-temporal motion planning problem and find paths of minimum expected duration. Experimental results show STAP generates paths of shorter duration and greater average robot-human separation distance throughout tasks. Additionally, STAP more accurately estimates robot trajectory durations in HRC, which are useful in arriving at proactive-n-reactive robot sequencing.Comment: 7 pages, 7 figures. Accepted at IEEE ROMAN 202

Operator awareness in human–robot collaboration through wearable vibrotactile feedback

In industrial scenarios, requiring human–robot collaboration, the understanding between the human operator and his/her robot coworker is paramount. On the one side, the robot has to detect human intentions, and on the other side, the human needs to be aware of what is happening during the collaborative task. In this letter, we address the first issue by predicting human behavior through a new recursive Bayesian classifier, exploiting head, and hand tracking data. Human awareness is tackled by endowing the human with a vibrotactile ring that sends acknowledgments to the user during critical phases of the collaborative task. The proposed solution has been assessed in a human–robot collaboration scenario, and we found that adding haptic feedback is particularly helpful to improve the performance when the human–robot cooperation task is performed by nonskilled subjects. We believe that predicting operator's intention and equipping him/her with wearable interface, able to give information about the prediction reliability, are essential features to improve performance in a human–robot collaboration in industrial environments

Real-time Human Workload Estimation and Its Application in Adaptive Haptic Shared Control

Automated vehicles (AVs) are promising to have the potential to reduce driving-related injuries and deaths. However, autonomous driving technology is currently limited in its scope and reliability, giving rise to the semi-autonomous driving model, where the autonomy and the human share the control of the vehicle. Workload, despite being an important human factor, has not yet been considered when designing adaptive shared control. Recently, researchers have started to apply machine learning techniques to classify mental workload into different levels. However, most of these studies have adopted either a single-model-single-feature approach or a single-model-all-features approach. However different machine learning models are suitable for different features, how to leverage different models for different features is critical. To address these shortcomings and research gaps, the goals of this dissertation were to (1) examine whether and to what extent haptic shared control performance can be improved by incorporating operators' workload; (2) develop a computational model for workload estimation, and the model should be able to leverage different machine learning models that work best for different features; and (3) investigate the generalizability of the workload estimation model. To address these research goals, this dissertation was composed of four research phases with two pilot studies and four human subject experiments. (1) Collaborating with Yifan Weng, Dr. Tulga Ersal, and Prof. Jeffrey Stein from the Department of Mechanical Engineering at the University of Michigan, we developed a teleoperated dual-task shared control simulation platform where the human shared control of a ground vehicle with autonomy while performing a surveillance task simultaneously. In addition, we developed a real-time eye-tracking system based on Tobii Pro Glasses 2 to measure the human gaze points in a world frame and pupil sizes. (2) We proposed a workload-adaptive haptic shared control scheme together with our collaborators. We conducted two human subject experiments during this phase. The results indicated that the proposed workload-adaptive haptic shared control scheme can reduce human workload, increase human trust in the system, increase driving performance, and reduce human effort without sacrificing surveillance task performance. (3) We proposed a Bayesian inference model for workload estimation that can leverage the different machine learning models that work best for different features. Specifically, we used support-vector machines (SVMs) for pupil size change, the Hidden Markov Model (HMM) for gaze trajectory, SVMs for fixation feature, and Gaussian Mixture Models (GMMs) for fixation trajectory. The empirical results indicated that our proposed model achieved a 0.82 F1 score for workload imposed by varying surveillance task urgency. (4) We investigated the generalizability of our proposed Bayesian inference model for workload estimation by conducting two human subject experiments with 24 participants and using different factors to impose human workload, i.e., obstacle headway and driving speed. The results indicated that our proposed model achieved a 0.68 F1 score for the workload imposed by obstacle avoidance and the personalized version of our proposed model can distinguish the workload imposed by different driving speeds under high surveillance task urgency.PHDRoboticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169623/1/ruikunl_1.pd