Multi-Operator Gesture Control of Robotic Swarms Using Wearable Devices

The theory and design of effective interfaces for human interaction with multi-robot systems has recently gained significant interest. Robotic swarms are multi-robot systems where local interactions between robots and neighbors within their spatial neighborhood generate emergent collective behaviors. Most prior work has studied interfaces for human interaction with remote swarms, but swarms also have great potential in applications working alongside humans, motivating the need for interfaces for local interaction. Given the collective nature of swarms, human interaction may occur at many levels of abstraction ranging from swarm behavior selection to teleoperation. Wearable gesture control is an intuitive interaction modality that can meet this requirement while keeping operator hands usually unencumbered. In this paper, we present an interaction method using a gesture-based wearable device with a limited number of gestures for robust control of a complex system: a robotic swarm. Experiments conducted with a real robot swarm compare performance in single and two-operator conditions illustrating the effectiveness of the method. Results show human operators using our interaction method are able to successfully complete the task in all trials, illustrating the effectiveness of the method, with better performance in the two-operator condition, indicating separation of function is beneficial for our method. The primary contribution of our work is the development and demonstration of interaction methods that allow robust control of a difficult to understand multi robot system using only the noisy inputs typical of smartphones and other on-body sensor driven devices

Asynchronous Visualization of Spatiotemporal Information for Multiple Moving Targets

In the modern information age, the quantity and complexity of spatiotemporal data is increasing both rapidly and continuously. Sensor systems with multiple feeds that gather multidimensional spatiotemporal data will result in information clusters and overload, as well as a high cognitive load for users of these systems. To meet future safety-critical situations and enhance time-critical decision-making missions in dynamic environments, and to support the easy and effective managing, browsing, and searching of spatiotemporal data in a dynamic environment, we propose an asynchronous, scalable, and comprehensive spatiotemporal data organization, display, and interaction method that allows operators to navigate through spatiotemporal information rather than through the environments being examined, and to maintain all necessary global and local situation awareness. To empirically prove the viability of our approach, we developed the Event-Lens system, which generates asynchronous prioritized images to provide the operator with a manageable, comprehensive view of the information that is collected by multiple sensors. The user study and interaction mode experiments were designed and conducted. The Event-Lens system was discovered to have a consistent advantage in multiple moving-target marking-task performance measures. It was also found that participants’ attentional control, spatial ability, and action video gaming experience affected their overall performance

Integrating Affective Expressions into Robot-Assisted Search and Rescue to Improve Human-Robot Communication

Unexplained or ambiguous behaviours of rescue robots can lead to inefficient collaborations between humans and robots in robot-assisted SAR teams. To date, rescue robots do not have the ability to interact with humans on a social level, which is believed to be an essential ability that can improve the quality of interactions. This thesis research proposes to bring affective robot expressions into the SAR context to provide rescue robots social capabilities. The first experiment presented in Chapter 3 investigates whether there is consensus in mapping emotions to messages/situations in Urban Search and Rescue (USAR) scenarios, where efficiency and effectiveness of interactions are crucial to success. We studied mappings between 10 specific messages, presented in two different communication styles, reflecting common situations that might happen during search and rescue missions and the emotions exhibited by robots in those situations. The data was obtained through a Mechanical Turk study with 78 participants. The findings support the feasibility of using emotions as an additional communication channel to improve multi-modal human-robot interaction for urban search and rescue robots and suggest that these mappings are robust, i.e., are not affected by the robot’s communication style. The second experiment was conducted on Amazon Mechanical Turk as well with 223 participants. We used Affect Control Theory (ACT) as a method for deriving the mappings between situations and emotions (similar to the ones in the first experiment) and as an alternative method to obtaining mappings that can be adjusted for different emotion sets (Chapter 4). The results suggested that there is consistency in the choice of emotions for a robot to show in different situations between the two methods used in the first and second experiment, indicating the feasibility of using emotions as an additional modality in SAR robots. After validating the feasibility of bringing emotions to SAR context based on the findings from the first two experiments, we created affective expressions based on Evaluation, Potency and Activity (EPA) dimensions of ACT with the help of LED lights on a rescue robot called Husky. We evaluated the effect of emotions on rescue workers’ situational awareness through an online Amazon Mechanical Turk Study with 151 participants (Chapter 5). Findings indicated that participants who saw Husky with affective expressions (conveyed through lights) had better perception accuracy of the situation happening in the disaster scene than participants who saw the videos of the Husky robot without any affective lights. In other words, Husky with affective lights improved participants’ situational awareness

Dynamic virtual reality user interface for teleoperation of heterogeneous robot teams

This research investigates the possibility to improve current teleoperation control for heterogeneous robot teams using modern Human-Computer Interaction (HCI) techniques such as Virtual Reality. It proposes a dynamic teleoperation Virtual Reality User Interface (VRUI) framework to improve the current approach to teleoperating heterogeneous robot teams

Teams organization and performance in multi-human/multi-robot teams

We are developing a theory for human control of robot teams based on considering how control varies across different task allocations. Our current work focuses on domains such as foraging in which robots perform largely independent tasks. The present study addresses the interaction between automation and organization of human teams in controlling large robot teams performing an Urban Search and Rescue (USAR) task. We identify three subtasks: perceptual search-visual search for victims, assistance-teleoperation to assist robot, and navigation-path planning and coordination. For the studies reported here, navigation was selected for automation because it involves weak dependencies among robots making it more complex and because it was shown in an earlier experiment to be the most difficult. Two possible ways to organize operators were identified as assignment of robots to particular operators or as a shared pool in which operators service robots from the population as needed. The experiment compares two member teams of operators controlling teams of 12 robots each in the assigned robots conditions or sharing control of 24 robots in the shared pool conditions using either waypoint control or autonomous path planning. We identify three self organizing team strategies in the shared pool condition: joint control operators share full authority over robots, mixed control in which one operator takes primary control while the other acts as an assistant, and split control in which operators divide the robots with each controlling a subteam. Automating path planning improved system performance. Effects of team organization favored operator teams who shared authority for the pool of robots. ©2010 IEEE

Human-Robot Teams – Paving the Way for the Teams of the Future

Thanks to advances in artificial intelligence and robotics, robots, and especially social robots that can naturally interact with humans, are now found in more and more areas of our lives. At the same time, teams have been the norm in organizations for decades. To bring these two circumstances together, this dissertation addresses the use of social robots together with humans in teams, so-called human-robot teams (HRTs). This work aims to advance knowledge about HRTs and important underlying mechanisms in the establishment of such teams, thereby providing insights in two aspects. First, a structured and universal definition of HRTs is derived from the various perspectives of extant research, and based on a comprehensive literature overview, important characteristics and influencing factors of HRTs as well as research gaps in HRT research are identified. Second, insights into the underlying mechanisms of the establishment of human-robot teams are provided for settings with social robots in two different team roles: team assistant and lower-level (team) manager. For this purpose, this dissertation contains three research studies that cover the currently largely unexplored area of social robots' use in organizational teams at both the employee and lower-level manager levels. The first study (conceptual study) provides a foundation for this dissertation and beyond by developing a structured and universal definition of HRTs. It also structures extant research on HRTs and proposes an agenda for future research on HRTs based on research gaps identified in a comprehensive literature review that includes 194 studies on HRTs. The second and third studies (empirical studies 1 and 2) use empirical online studies to address two of the research gaps identified in the conceptual study. They examine the underlying mechanisms in decisions for social robots in two different team roles: team assistant (empirical study 1) and team manager (empirical study 2). By looking at expectations and experiences of taskwork-/performance-related and teamwork-related/relational features of social robots using polynomial regressions and response surface analyses, these studies rely on expectation disconfirmation theory to provide a detailed investigation of the underlying mechanisms of organizational decisions for social robots. Empirical study 1 thereby shows that for teamwork, positive disconfirmation and high levels of experiences lead to higher acceptance of humanoid and android robotic team assistants, and similar results emerge for a humanoid robot’s taskwork skills. In contrast, for taskwork skills of android team assistants, high levels of positive disconfirmation lead to lower robot acceptance. For robotic lower-level managers, empirical study 2 shows that there are discrepancies in the evaluation of performance-related usefulness and relational attitude. While for usefulness a slight overfulfilment of expectations leads to a positive impact on the readiness to work with, before evaluations decrease with greater overfulfillment, for attitude increasing positive experiences are associated with (decreasing) positive evaluations of readiness. In summary, this dissertation contributes to scientific research on HRTs by advancing the understanding of HRTs, providing a structured and universal definition of HRTs, and suggesting avenues for future research. The systematic investigation of underlying mechanisms for the selection of different types of social robots for different team roles provides a holistic view of this new form of organizational teams. In addition to the research contributions, this thesis also provides practical guidance for the successful establishment of HRTs in organizations