ARDebug: an augmented reality tool for analysing and debugging swarm robotic systems

Despite growing interest in collective robotics over the past few years, analysing and debugging the behaviour of swarm robotic systems remains a challenge due to the lack of appropriate tools. We present a solution to this problem-ARDebug: an open-source, cross-platform, and modular tool that allows the user to visualise the internal state of a robot swarm using graphical augmented reality techniques. In this paper we describe the key features of the software, the hardware required to support it, its implementation, and usage examples. ARDebug is specifically designed with adoption by other institutions in mind, and aims to provide an extensible tool that other researchers can easily integrate with their own experimental infrastructure

ARDebug: An Augmented Reality Tool for Analysing and Debugging Swarm Robotic Systems

Despite growing interest in collective robotics over the past few years, analysing and debugging the behaviour of swarm robotic systems remains a challenge due to the lack of appropriate tools. We present a solution to this problem—ARDebug: an open-source, cross-platform, and modular tool that allows the user to visualise the internal state of a robot swarm using graphical augmented reality techniques. In this paper we describe the key features of the software, the hardware required to support it, its implementation, and usage examples. ARDebug is specifically designed with adoption by other institutions in mind, and aims to provide an extensible tool that other researchers can easily integrate with their own experimental infrastructure

Agent Teaming Situation Awareness (ATSA): A Situation Awareness Framework for Human-AI Teaming

The rapid advancements in artificial intelligence (AI) have led to a growing trend of human-AI teaming (HAT) in various fields. As machines continue to evolve from mere automation to a state of autonomy, they are increasingly exhibiting unexpected behaviors and human-like cognitive/intelligent capabilities, including situation awareness (SA). This shift has the potential to enhance the performance of mixed human-AI teams over all-human teams, underscoring the need for a better understanding of the dynamic SA interactions between humans and machines. To this end, we provide a review of leading SA theoretical models and a new framework for SA in the HAT context based on the key features and processes of HAT. The Agent Teaming Situation Awareness (ATSA) framework unifies human and AI behavior, and involves bidirectional, and dynamic interaction. The framework is based on the individual and team SA models and elaborates on the cognitive mechanisms for modeling HAT. Similar perceptual cycles are adopted for the individual (including both human and AI) and the whole team, which is tailored to the unique requirements of the HAT context. ATSA emphasizes cohesive and effective HAT through structures and components, including teaming understanding, teaming control, and the world, as well as adhesive transactive part. We further propose several future research directions to expand on the distinctive contributions of ATSA and address the specific and pressing next steps.Comment: 52 pages,5 figures, 1 tabl

A Survey of Multi-Agent Human-Robot Interaction Systems

This article presents a survey of literature in the area of Human-Robot Interaction (HRI), specifically on systems containing more than two agents (i.e., having multiple humans and/or multiple robots). We identify three core aspects of ``Multi-agent" HRI systems that are useful for understanding how these systems differ from dyadic systems and from one another. These are the Team structure, Interaction style among agents, and the system's Computational characteristics. Under these core aspects, we present five attributes of HRI systems, namely Team size, Team composition, Interaction model, Communication modalities, and Robot control. These attributes are used to characterize and distinguish one system from another. We populate resulting categories with examples from recent literature along with a brief discussion of their applications and analyze how these attributes differ from the case of dyadic human-robot systems. We summarize key observations from the current literature, and identify challenges and promising areas for future research in this domain. In order to realize the vision of robots being part of the society and interacting seamlessly with humans, there is a need to expand research on multi-human -- multi-robot systems. Not only do these systems require coordination among several agents, they also involve multi-agent and indirect interactions which are absent from dyadic HRI systems. Adding multiple agents in HRI systems requires advanced interaction schemes, behavior understanding and control methods to allow natural interactions among humans and robots. In addition, research on human behavioral understanding in mixed human-robot teams also requires more attention. This will help formulate and implement effective robot control policies in HRI systems with large numbers of heterogeneous robots and humans; a team composition reflecting many real-world scenarios.Comment: 23 pages, 7 figure

Adaptive and learning-based formation control of swarm robots

Autonomous aerial and wheeled mobile robots play a major role in tasks such as search and rescue, transportation, monitoring, and inspection. However, these operations are faced with a few open challenges including robust autonomy, and adaptive coordination based on the environment and operating conditions, particularly in swarm robots with limited communication and perception capabilities. Furthermore, the computational complexity increases exponentially with the number of robots in the swarm. This thesis examines two different aspects of the formation control problem. On the one hand, we investigate how formation could be performed by swarm robots with limited communication and perception (e.g., Crazyflie nano quadrotor). On the other hand, we explore human-swarm interaction (HSI) and different shared-control mechanisms between human and swarm robots (e.g., BristleBot) for artistic creation. In particular, we combine bio-inspired (i.e., flocking, foraging) techniques with learning-based control strategies (using artificial neural networks) for adaptive control of multi- robots. We first review how learning-based control and networked dynamical systems can be used to assign distributed and decentralized policies to individual robots such that the desired formation emerges from their collective behavior. We proceed by presenting a novel flocking control for UAV swarm using deep reinforcement learning. We formulate the flocking formation problem as a partially observable Markov decision process (POMDP), and consider a leader-follower configuration, where consensus among all UAVs is used to train a shared control policy, and each UAV performs actions based on the local information it collects. In addition, to avoid collision among UAVs and guarantee flocking and navigation, a reward function is added with the global flocking maintenance, mutual reward, and a collision penalty. We adapt deep deterministic policy gradient (DDPG) with centralized training and decentralized execution to obtain the flocking control policy using actor-critic networks and a global state space matrix. In the context of swarm robotics in arts, we investigate how the formation paradigm can serve as an interaction modality for artists to aesthetically utilize swarms. In particular, we explore particle swarm optimization (PSO) and random walk to control the communication between a team of robots with swarming behavior for musical creation

Route Planning and Operator Allocation in Robot Fleets

In this thesis, we address various challenges related to optimal planning and task allocation in a robot fleet supervised by remote human operators. The overarching goal is to enhance the performance and efficiency of the robot fleets by planning routes and scheduling operator assistance while accounting for limited human availability. The thesis consists of three main problems, each of which focuses on a specific aspect of the system. The first problem pertains to optimal planning for a robot in a collaborative human-robot team, where the human supervisor is intermittently available to assist the robot to complete its tasks faster. Specifically, we address the challenge of computing the fastest route between two configurations in an environment with time constraints on how long the robot can wait for assistance at intermediate configurations. We consider the application of robot navigation in a city environment, where different routes can have distinct speed limits and different time constraints on how long a robot is allowed to wait. Our proposed approach utilizes the concepts of budget and critical departure times, enabling optimal solution and enhanced scalability compared to existing methods. Extensive comparisons with baseline algorithms on a city road network demonstrate its effectiveness and ability to achieve high-quality solutions. Furthermore, we extend the problem to the multi-robot case, where the challenge lies in prioritizing robots when multiple service requests arrive simultaneously. To address this challenge, we present a greedy algorithm that efficiently prioritizes service requests in a batch and has a remarkably good performance compared to the optimal solution. The next problem focuses on allocating human operators to robots in a fleet, considering each robot's specified route and the potential for failures and getting stuck. Conventional techniques used to solve such problems face scalability issues due to exponential growth of state and action spaces with the number of robots and operators. To overcome these, we derive conditions for a technical requirement called indexability, thereby enabling the use of the Whittle index heuristic. Our key insight is to leverage the structure of the value function of individual robots, resulting in conditions that can be easily verified separately for each state of each robot. We apply these conditions to two types of transitions commonly seen in supervised robot fleets. Through numerical simulations, we demonstrate the efficacy of Whittle index policy as a near-optimal scalable approach that outperforms existing scalable methods. Finally, we investigate the impact of interruptions on human supervisors overseeing a fleet of robots. Human supervisors in such systems are primarily responsible for monitoring robots, but can also be assigned with secondary tasks. These tasks can act as interruptions and can be categorized as either intrinsic, i.e., being directly related to the monitoring task, or extrinsic, i.e., being unrelated. Through a user study involving $39$ participants, the findings reveal that task performance remains relatively unaffected by interruptions, and is primarily dependent on the number of robots being monitored. However, extrinsic interruptions led to a significant increase in perceived workload, creating challenges in switching between tasks. These results highlight the importance of managing user workload by limiting extrinsic interruptions in such supervision systems. Overall, this thesis contributes to the field of robot planning and operator allocation in collaborative human-robot teams. By incorporating human assistance, addressing scalability challenges, and understanding the impact of interruptions, we aim to enhance the performance and usability of robot fleets. Our work introduces optimal planning methods and efficient allocation strategies, empowering the seamless operation of robot fleets in real-world scenarios. Additionally, we provide valuable insights into user workload, shedding light on the interactions between humans and robots in such systems. We hope that our research promotes the widespread adoption of robot fleets and facilitates their integration into various domains, ultimately driving advancements in the field

Dynamic virtual reality user interface for teleoperation of heterogeneous robot teams

This research investigates the possibility to improve current teleoperation control for heterogeneous robot teams using modern Human-Computer Interaction (HCI) techniques such as Virtual Reality. It proposes a dynamic teleoperation Virtual Reality User Interface (VRUI) framework to improve the current approach to teleoperating heterogeneous robot teams

Human factors of semi-autonomous robots for urban search and rescue

During major disasters or other emergencies, Urban Search and Rescue (USAR) teams are responsible for extricating casualties safely from collapsed urban structures. The rescue work is dangerous due to possible further collapse, fire, dust or electricity hazards. Sometimes the necessary precautions and checks can last several hours before rescuers are safe to start the search for survivors. Remote controlled rescue robots provide the opportunity to support human rescuers to search the site for trapped casualties while they remain in a safe place. The research reported in this thesis aimed to understand how robot behaviour and interface design can be applied to utilise the benefits of robot autonomy and how to inform future human-robot collaborative systems. The data was analysed in the context of USAR missions when using semi-autonomous remote controlled robot systems. The research focussed on the influence of robot feedback, robot reliability, task complexity, and transparency. The influence of these factors on trust, workload, and performance was examined. The overall goal of the research was to make the life of rescuers safer and enhance their performance to help others in distress. Data obtained from the studies conducted for this thesis showed that semi-autonomous robot reliability is still the most dominant factor influencing trust, workload, and team performance. A robot with explanatory feedback was perceived as more competent, more efficient and less malfunctioning. The explanatory feedback was perceived as a clearer type of communication compared to concise robot feedback. Higher levels of robot transparency were perceived as more trustworthy. However, single items on the trust questionnaire were manipulated and further investigation is necessary. However, neither explanatory feedback from the robot nor robot transparency, increased team performance or mediated workload levels. Task complexity mainly influenced human-robot team performance and the participants’ control allocation strategy. Participants allowed the robot to find more targets and missed more robot errors in the high complexity conditions compared to the low task complexity conditions. Participants found more targets manually in the low complexity tasks. In addition, the research showed that recording the observed robot performance (the performance of the robot that was witnessed by the participant) can help to identify the cause of contradicting results: participants might not have noticed some of the robots mistakes and therefore they were not able to distinguish between the robot reliability levels. Furthermore, the research provided a foundation of knowledge regarding the real world application of USAR in the United Kingdom. This included collecting knowledge via an autoethnographic approach about working processes, command structures, currently used technical equipment, and attitudes of rescuers towards robots. Also, recommendations about robot behaviour and interface design were collected throughout the research. However, recommendations made in the thesis include consideration of the overall outcome (mission performance) and the perceived usefulness of the system in order to support the uptake of the technology in real world applications. In addition, autonomous features might not be appropriate in all USAR applications. When semi-autonomous robot trials were compared to entirely manual operation, only the robot with an average of 97% reliability significantly increased the team performance and reduced the time needed to complete the USAR scenario compared to the manually operated robot. Unfortunately, such high robot success levels do not exist to date. This research has contributed to our understanding of the factors influencing human-robot collaboration in USAR operations, and provided guidance for the next generation of autonomous robots

Human factors of semi-autonomous robots for urban search and rescue

During major disasters or other emergencies, Urban Search and Rescue (USAR) teams are responsible for extricating casualties safely from collapsed urban structures. The rescue work is dangerous due to possible further collapse, fire, dust or electricity hazards. Sometimes the necessary precautions and checks can last several hours before rescuers are safe to start the search for survivors. Remote controlled rescue robots provide the opportunity to support human rescuers to search the site for trapped casualties while they remain in a safe place. The research reported in this thesis aimed to understand how robot behaviour and interface design can be applied to utilise the benefits of robot autonomy and how to inform future human-robot collaborative systems. The data was analysed in the context of USAR missions when using semi-autonomous remote controlled robot systems. The research focussed on the influence of robot feedback, robot reliability, task complexity, and transparency. The influence of these factors on trust, workload, and performance was examined. The overall goal of the research was to make the life of rescuers safer and enhance their performance to help others in distress. Data obtained from the studies conducted for this thesis showed that semi-autonomous robot reliability is still the most dominant factor influencing trust, workload, and team performance. A robot with explanatory feedback was perceived as more competent, more efficient and less malfunctioning. The explanatory feedback was perceived as a clearer type of communication compared to concise robot feedback. Higher levels of robot transparency were perceived as more trustworthy. However, single items on the trust questionnaire were manipulated and further investigation is necessary. However, neither explanatory feedback from the robot nor robot transparency, increased team performance or mediated workload levels. Task complexity mainly influenced human-robot team performance and the participants’ control allocation strategy. Participants allowed the robot to find more targets and missed more robot errors in the high complexity conditions compared to the low task complexity conditions. Participants found more targets manually in the low complexity tasks. In addition, the research showed that recording the observed robot performance (the performance of the robot that was witnessed by the participant) can help to identify the cause of contradicting results: participants might not have noticed some of the robots mistakes and therefore they were not able to distinguish between the robot reliability levels. Furthermore, the research provided a foundation of knowledge regarding the real world application of USAR in the United Kingdom. This included collecting knowledge via an autoethnographic approach about working processes, command structures, currently used technical equipment, and attitudes of rescuers towards robots. Also, recommendations about robot behaviour and interface design were collected throughout the research. However, recommendations made in the thesis include consideration of the overall outcome (mission performance) and the perceived usefulness of the system in order to support the uptake of the technology in real world applications. In addition, autonomous features might not be appropriate in all USAR applications. When semi-autonomous robot trials were compared to entirely manual operation, only the robot with an average of 97% reliability significantly increased the team performance and reduced the time needed to complete the USAR scenario compared to the manually operated robot. Unfortunately, such high robot success levels do not exist to date. This research has contributed to our understanding of the factors influencing human-robot collaboration in USAR operations, and provided guidance for the next generation of autonomous robots