Effects of alarms on control of robot teams

Annunciator driven supervisory control (ADSC) is a widely used technique for directing human attention to control systems otherwise beyond their capabilities. ADSC requires associating abnormal parameter values with alarms in such a way that operator attention can be directed toward the involved subsystems or conditions. This is hard to achieve in multirobot control because it is difficult to distinguish abnormal conditions for states of a robot team. For largely independent tasks such as foraging, however, self-reflection can serve as a basis for alerting the operator to abnormalities of individual robots. While the search for targets remains unalarmed the resulting system approximates ADSC. The described experiment compares a control condition in which operators perform a multirobot urban search and rescue (USAR) task without alarms with ADSC (freely annunciated) and with a decision aid that limits operator workload by showing only the top alarm. No differences were found in area searched or victims found, however, operators in the freely annunciated condition were faster in detecting both the annunciated failures and victims entering their cameras' fields of view. Copyright 2011 by Human Factors and Ergonomics Society, Inc. All rights reserved

Mixed Initiative Systems for Human-Swarm Interaction: Opportunities and Challenges

Human-swarm interaction (HSI) involves a number of human factors impacting human behaviour throughout the interaction. As the technologies used within HSI advance, it is more tempting to increase the level of swarm autonomy within the interaction to reduce the workload on humans. Yet, the prospective negative effects of high levels of autonomy on human situational awareness can hinder this process. Flexible autonomy aims at trading-off these effects by changing the level of autonomy within the interaction when required; with mixed-initiatives combining human preferences and automation's recommendations to select an appropriate level of autonomy at a certain point of time. However, the effective implementation of mixed-initiative systems raises fundamental questions on how to combine human preferences and automation recommendations, how to realise the selected level of autonomy, and what the future impacts on the cognitive states of a human are. We explore open challenges that hamper the process of developing effective flexible autonomy. We then highlight the potential benefits of using system modelling techniques in HSI by illustrating how they provide HSI designers with an opportunity to evaluate different strategies for assessing the state of the mission and for adapting the level of autonomy within the interaction to maximise mission success metrics.Comment: Author version, accepted at the 2018 IEEE Annual Systems Modelling Conference, Canberra, Australi

Attention Allocation for Human Multi-Robot Control: Cognitive Analysis based on Behavior Data and Hidden States

Human multi-robot interaction exploits both the human operator’s high-level decision-making skills and the robotic agents’ vigorous computing and motion abilities. While controlling multi-robot teams, an operator’s attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator’s attentional resources, a robot with self reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots’ normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot’s request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots’ failures) tasks, regardless of the system’s reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users’ underlying cognitive strategies and may fail to reflect the user’s true intent. As understanding users’ intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators’ underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments

A discrete event simulation model for unstructured supervisory control of unmanned vehicles

Thesis (S.B.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 33).Most current Unmanned Vehicle (UV) systems consist of teams of operators controlling a single UV. Technological advances will likely lead to the inversion of this ratio, and automation of low level tasking. These advances will also lead to a growth in UV use in large-scale applications such as urban search and rescue, which will require the use of both teams of operators and teams of UVs. This growth will in turn require research and development in the area of team supervisory control of multiple UVs. Human-in-the- loop experimentation is often used during this research but can be time consuming and expensive. The time and cost of experimentation can often be drastically reduced by using predictive models. However there is a lack of such models in the area of multiple-operator supervisory control of multiple- UVs. This problem is addressed in this thesis through the following method: First, current predictive models of human supervisory control of UVs are analyzed, and attributes of systems related to this modeling space are identified. Second, a queuing-based multiple-operator multiple-vehicle discrete event simulation model (MO-MUVDES) is developed which captures these attributes, including the ability to predict performance in situations with low observable exogenous event arrivals. MO-MUVDES also incorporates traditional system variables such as level of vehicle autonomy, vehicle and operator team structure, and operator switching strategy. The accuracy and robustness of the MO-MUVDES model were measured by a two-stage validation process using data from a human-in-the-loop supervisory control experiment, and a Monte Carlo simulation. The first stage of the validation process used data from the experiment as input for the MOMUVDES model which was then used to generate predictions of operator performance. In the second stage of validation, a sensitivity analysis was performed on the MO-MUVDES model. This validation process achieved confidence in the model's ability to predict operator performance and a measurement of the robustness of the model under varying input conditions. Additionally, the process indicated that discrete event simulation is an effective technique for modeling team supervisory control of UVs in a situation where exogenous event arrivals are not clearly observable. As a result, the MO-MUVDES model could be used to reduce development time for systems within its modeled space.by Anthony D. McDonald.S.B

Using Discrete Event Simulation to Model Multi-Robot Multi-Operator Teamwork

With the increasing need for teams of operators in controlling multiple robots, it is important to understand how to construct the team and support team processes. While running experiments can be time consuming and expensive, the use of simulation models is an alternative method. In this study, we built a discrete event simulation model that represents multi-robot multi-operator teamwork. Preliminary results show that the model can generate performance measures consistent with experimental results.This research is sponsored by the Office for Naval Research and the Air Force Office of Scientific Research

Digital twin-enabled human-robot collaborative teaming towards sustainable and healthy built environments

Development of sustainable and healthy built environments (SHBE) is highly advocated to achieve collective societal good. Part of the pathway to SHBE is the engagement of robots to manage the ever-complex facilities for tasks such as inspection and disinfection. However, despite the increasing advancements of robot intelligence, it is still “mission impossible” for robots to independently undertake such open-ended problems as facility management, calling for a need to “team up” the robots with humans. Leveraging digital twin's ability to capture real-time data and inform decision-making via dynamic simulation, this study aims to develop a human-robot teaming framework for facility management to achieve sustainability and healthiness in the built environments. A digital twin-enabled prototype system is developed based on the framework. Case studies showed that the framework can safely and efficiently incorporate robotics into facility management tasks (e.g., patrolling, inspection, and cleaning) by allowing humans to plan, oversee, manage, and cooperate with the robot via the digital twin's bi-directional mechanism. The study lays out a high-level framework, under which purposeful efforts can be made to unlock digital twin's full potential in collaborating humans and robots in facility management towards SHBE

Asynchronous control with ATR for large robot teams

In this paper, we discuss and investigate the advantages of an asynchronous display, called "image queue", tested for an urban search and rescue foraging task. The image queue approach mines video data to present the operator with a relevant and comprehensive view of the environment by selecting a small number of images that together cover large portions of the area searched. This asynchronous approach allows operators to search through a large amount of data gathered by autonomous robot teams, and allows comprehensive and scalable displays to obtain a network-centric perspective for unmanned ground vehicles (UGVs). In the reported experiment automatic target recognition (ATR) was used to augment utilities based on visual coverage in selecting imagery for presentation to the operator. In the cued condition a box was drawn in the region in which a possible target was detected. In the no-cue condition no box was drawn although the target detection probability continued to play a role in the selection of imagery. We found that operators using the image queue displays missed fewer victims and relied on teleoperation less often than those using streaming video. Image queue users in the no-cue condition did better in avoiding false alarms and reported lower workload than those in the cued condition. Copyright 2011 by Human Factors and Ergonomics Society, Inc. All rights reserved

Auction-Based Task Allocation and Motion Planning for Multi-Robot Systems with Human Supervision

This paper presents a task allocation strategy for a multi-robot system with a human supervisor. The multi-robot system consists of a team of heterogeneous robots with different capabilities that operate in a dynamic scenario that can change in the robots’ capabilities or in the operational requirements. The human supervisor can intervene in the operation scenario by approving the final plan before its execution or forcing a robot to execute a specific task. The proposed task allocation strategy leverages an auction-based method in combination with a sampling-based multi-goal motion planning. The latter is used to evaluate the costs of execution of tasks based on realistic features of paths. The proposed architecture enables the allocation of tasks accounting for priorities and precedence constraints, as well as the quick re-allocation of tasks after a dynamic perturbation occurs –a crucial feature when the human supervisor preempts the outcome of the algorithm and makes manual adjustments. An extensive simulation campaign in a rescue scenario validates our approach in dynamic scenarios comprising a sensor failure of a robot, a total failure of a robot, and a human-driven re-allocation. We highlight the benefits of the proposed multi-goal strategy by comparing it with single-goal motion planning strategies at the state of the art. Finally, we provide evidence for the system efficiency by demonstrating the powerful synergistic combination of the auction-based allocation and the multi-goal motion planning approach