Neglect Benevolence in Human-Swarm Interaction with Communication Latency

In practical applications of robot swarms with bio-inspired behaviors, a human operator will need to exert control over the swarm to fulfill the mission objectives. In many operational settings, human operators are remotely located and the communication environment is harsh. Hence, there exists some latency in information (or control command) transfer between the human and the swarm. In this paper, we conduct experiments of human-swarm interaction to investigate the effects of communication latency on the performance of a human-swarm system in a swarm foraging task. We develop and investigate the concept of neglect benevolence, where a human operator allows the swarm to evolve on its own and stabilize before giving new commands. Our experimental results indicate that operators exploited neglect benevolence in different ways to develop successful strategies in the foraging task. Furthermore, we show experimentally that the use of a predictive display can help mitigate the adverse effects of communication latency

Operator Objective Function Guidance for a Real-time Unmanned Vehicle Scheduling Algorithm

Advances in autonomy have made it possible to invert the typical operator-to-unmanned-vehicle ratio so that asingle operator can now control multiple heterogeneous unmanned vehicles. Algorithms used in unmanned-vehicle path planning and task allocation typically have an objective function that only takes into account variables initially identified by designers with set weightings. This can make the algorithm seemingly opaque to an operator and brittle under changing mission priorities. To address these issues, it is proposed that allowing operators to dynamically modify objective function weightings of an automated planner during a mission can have performance benefits. A multiple-unmanned-vehicle simulation test bed was modified so that operators could either choose one variable or choose any combination of equally weighted variables for the automated planner to use in evaluating mission plans. Results from a human-participant experiment showed that operators rated their performance and confidence highest when using the dynamic objective function with multiple objectives. Allowing operators to adjust multiple objectives resulted in enhanced situational awareness, increased spare mental capacity, fewer interventions to modify the objective function, and no significant differences in mission performance. Adding this form of flexibility and transparency to automation in future unmanned vehicle systems could improve performance, engender operator trust, and reduce errors.Aurora Flight Sciences, U.S. Office of Naval Researc

Attention Allocation for Human Multi-Robot Control: Cognitive Analysis based on Behavior Data and Hidden States

Human multi-robot interaction exploits both the human operator’s high-level decision-making skills and the robotic agents’ vigorous computing and motion abilities. While controlling multi-robot teams, an operator’s attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator’s attentional resources, a robot with self reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots’ normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot’s request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots’ failures) tasks, regardless of the system’s reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users’ underlying cognitive strategies and may fail to reflect the user’s true intent. As understanding users’ intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators’ underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments

Operator Scheduling Strategies in Supervisory Control of Multiple UAVs

The application of network centric operations to time-constrained command and control environments will mean that human operators will be increasingly responsible for multiple simultaneous supervisory control tasks. One such futuristic application will be the control of multiple unmanned aerial vehicles (UAVs) by a single operator. To achieve such performance in complex, time critical, and high risk settings, automated systems will be required both to guarantee rapid system response as well as manageable workload for operators. Through the development of a simulation test bed for human supervisory control of multiple independent UAVs by a single operator, this paper presents recent efforts to investigate workload mitigation strategies as a function of increasing automation. A humanin- the-loop experiment revealed that under low workload conditions, operators’ cognitive strategies were relatively robust across increasing levels of automated decision support. However, when provided with explicit automated recommendations and with the ability to negotiate with external agencies for delays in arrival times for targets, operators inappropriately fixated on the need to globally optimize their schedules. In addition, without explicit visual representation of uncertainty, operators tended to treated all probabilities uniformly. This study also revealed that operators that reached cognitive saturation adapted two very distinct management strategies, which led to varying degrees of success. Lastly, operators with management-by-exception decision support exhibited evidence of automation bias.This research was sponsored by Boeing Phantom Works

Coordination Strategies for Human Supervisory Control of Robotic Teams

Autonomous mobile sensor teams are crucial to many civilian and military applications. These robotic teams often operate within a larger supervisory system, involving human operators who oversee the mission and analyze sensory data. Here, both the human and the robotic system sub-components, as well as interactions between them, must be carefully considered in designing effective mission coordination strategies. This dissertation explores a series of representative sub-problems relating to the analysis and coordination of both mobile sensors and human operators within supervisory systems. The content herein is presented in three parts: Part I focuses on coordinating operator behavior independently (operator-focused methods), Part II focuses on coordinating mobile-sensor behavior independently (sensor-focused methods), and Part III focuses on jointly coordinating both operator and mobile sensor behavior (joint methods). The content herein is primarily motivated by a particular application in which Unmanned Aerial Vehicles collect visual imagery to be analyzed by a remotely located operator, although many of the results apply to any system of similar architecture. Specifically, with regard to operator-focused methods, Chapter 2 illustrates how physiological sensing, namely eye tracking, may provide aid in modeling operator behavior and assessing the usability of user interfaces. The results of a pilot usability study in which human observers interact with a supervisory control interface are presented, and eye-tracking data is correlated with various usability metrics. Chapter 3 develops robust scheduling algorithms for determining the ordering in which operators should process sensory tasks to both boost performance and decrease variance. A scenario-based, Mixed-Integer Linear Program (MILP) framework is presented, and is assessed in a series of numerical studies. With regard to sensor-focused methods, Chapters 4 and 5 consider two types of supervisory surveillance missions:Chapter 4 develops a cloud-based coverage strategy for persistent surveillance of planar regions. The scheme operates in a dynamic environment, only requiring sporadic, unplanned data exchanges between a central cloud and the sensors in the field. The framework is shown to provide collision avoidance and, in certain cases, produce convergence to a Pareto-optimal coverage configuration. In chapter 5, a heuristic routing scheme is discussed to produce Dubins tours for persistent surveillance of discrete targets, each with associated visibility and dwell-time constraints. Under some assumptions, the problem is posed as a constrained optimization that seeks a minimum-length tour, while simultaneously constraining the time required to reach the first target. A sampling-based scheme is used to approximate solutions to the constrained optimization. This approach is also shown to have desirable resolution completeness properties.Finally, Chapter 6 explores joint methods for coordinating both operator and sensor behavior in the context of a discrete surveillance mission (similar to that of Chapter 5), in which UAVs collect imagery of static targets to be analyzed by the human operator.In particular, a method is proposed to simultaneously construct UAV routes and operator schedules, with the goal of maintaining the operator's task load within a high-performance regime and preventing unnecessary UAV loitering. The full routing/scheduling problem is posed as a mixed-integer (non-linear) program, which can be equivalently represented as a MILP through the addition of auxiliary variables. For scalability, a MILP-based receding-horizon method is proposed to incrementally construct suboptimal solutions to the full optimization problem, which can be extended using a scenario-based approach (similar to that of Chapter 3) to incorporate robustness to operator uncertainty

Mixed Initiative Systems for Human-Swarm Interaction: Opportunities and Challenges

Human-swarm interaction (HSI) involves a number of human factors impacting human behaviour throughout the interaction. As the technologies used within HSI advance, it is more tempting to increase the level of swarm autonomy within the interaction to reduce the workload on humans. Yet, the prospective negative effects of high levels of autonomy on human situational awareness can hinder this process. Flexible autonomy aims at trading-off these effects by changing the level of autonomy within the interaction when required; with mixed-initiatives combining human preferences and automation's recommendations to select an appropriate level of autonomy at a certain point of time. However, the effective implementation of mixed-initiative systems raises fundamental questions on how to combine human preferences and automation recommendations, how to realise the selected level of autonomy, and what the future impacts on the cognitive states of a human are. We explore open challenges that hamper the process of developing effective flexible autonomy. We then highlight the potential benefits of using system modelling techniques in HSI by illustrating how they provide HSI designers with an opportunity to evaluate different strategies for assessing the state of the mission and for adapting the level of autonomy within the interaction to maximise mission success metrics.Comment: Author version, accepted at the 2018 IEEE Annual Systems Modelling Conference, Canberra, Australi

Global vs. local decision support for multiple independent UAV schedule management

As unmanned aerial vehicles (UAVs) become increasingly autonomous, time-critical and complex single-operator systems will require advance prediction and mitigation of schedule conflicts. However, actions that mitigate a current schedule conflict may create future schedule problems. Decision support is needed allowing an operator to evaluate different mission schedule management options in real-time. This paper describes two decision support visualisations for single-operator supervisory control of four independent UAVs performing a time-critical targeting mission. A configural display common to both visualisations, called StarVis, graphically depicts current schedule problems, as well as projections of potential local and global schedule problems. Results from an experiment showed that subjects using the locally optimal StarVis implementation had better performance, higher situational awareness, and no significant increase in workload over a more globally optimal implementation of StarVis. This research effort highlights how the same decision support design applied at different abstraction levels can produce different performance results.This research was sponsored by Mitre, Inc

NASA space station automation: AI-based technology review. Executive summary

Research and Development projects in automation technology for the Space Station are described. Artificial Intelligence (AI) based technologies are planned to enhance crew safety through reduced need for EVA, increase crew productivity through the reduction of routine operations, increase space station autonomy, and augment space station capability through the use of teleoperation and robotics

NASA space station automation: AI-based technology review

Research and Development projects in automation for the Space Station are discussed. Artificial Intelligence (AI) based automation technologies are planned to enhance crew safety through reduced need for EVA, increase crew productivity through the reduction of routine operations, increase space station autonomy, and augment space station capability through the use of teleoperation and robotics. AI technology will also be developed for the servicing of satellites at the Space Station, system monitoring and diagnosis, space manufacturing, and the assembly of large space structures

Technology assessment of advanced automation for space missions

Six general classes of technology requirements derived during the mission definition phase of the study were identified as having maximum importance and urgency, including autonomous world model based information systems, learning and hypothesis formation, natural language and other man-machine communication, space manufacturing, teleoperators and robot systems, and computer science and technology