Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards

Wizard of Oz (WoZ) as a prototyping method has been used to simulate intelligent user interfaces, particularly for speech-based systems. However, as our societies' expectations on artificial intelligence (AI) grows, the question remains whether a single Wizard is sufficient for it to simulate smarter systems and more complex interactions. Optimistic visions of 'what artificial intelligence (AI) can do' places demands on WoZ platforms to simulate smarter systems and more complex interactions. This raises the question of whether the typical approach of employing a single Wizard is sufficient. Moreover, while existing work has employed multiple Wizards in WoZ studies, a multi-Wizard approach has not been systematically studied in terms of feasibility, effectiveness, and challenges. We offer Wizundry, a real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. We outline the design and technical specifications of our open-source platform, which we iterated over two design phases. We report on two studies in which participant-Wizards were tasked with negotiating how to cooperatively simulate an interface that can handle natural speech for dictation and text editing as well as other intelligent text processing tasks. We offer qualitative findings on the Multi-Wizard experience for Dyads and Triads of Wizards. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.Comment: 34 page

Movement Acts in Breakdown Situations : How a Robot’s Recovery Procedure Affects Participants’ Opinions

Funding Information: Funding information : This research was partly funded by the Research Council of Norway as part of the Multimodal Elderly Care Systems (MECS) project, under grant agreement 247697. Publisher Copyright: © 2021 Trenton Schulz et al., published by De Gruyter.Recovery procedures are targeted at correcting issues encountered by robots. What are people’s opinions of a robot during these recovery procedures? During an experiment that examined how a mobile robot moved, the robot would unexpectedly pause or rotate itself to recover from a navigation problem. The serendipity of the recovery procedure and people’s understanding of it became a case study to examine how future study designs could consider breakdowns better and look at suggestions for better robot behaviors in such situations. We present the original experiment with the recovery procedure. We then examine the responses from the participants in this experiment qualitatively to see how they interpreted the breakdown situation when it occurred. Responses could be grouped into themes of sentience, competence, and the robot’s forms. The themes indicate that the robot’s movement communicated different information to different participants. This leads us to introduce the concept of movement acts to help examine the explicit and implicit parts of communication in movement. Given that we developed the concept looking at an unexpected breakdown, we suggest that researchers should plan for the possibility of breakdowns in experiments and examine and report people’s experience around a robot breakdown to further explore unintended robot communication.Peer reviewe

Route Planning and Operator Allocation in Robot Fleets

In this thesis, we address various challenges related to optimal planning and task allocation in a robot fleet supervised by remote human operators. The overarching goal is to enhance the performance and efficiency of the robot fleets by planning routes and scheduling operator assistance while accounting for limited human availability. The thesis consists of three main problems, each of which focuses on a specific aspect of the system. The first problem pertains to optimal planning for a robot in a collaborative human-robot team, where the human supervisor is intermittently available to assist the robot to complete its tasks faster. Specifically, we address the challenge of computing the fastest route between two configurations in an environment with time constraints on how long the robot can wait for assistance at intermediate configurations. We consider the application of robot navigation in a city environment, where different routes can have distinct speed limits and different time constraints on how long a robot is allowed to wait. Our proposed approach utilizes the concepts of budget and critical departure times, enabling optimal solution and enhanced scalability compared to existing methods. Extensive comparisons with baseline algorithms on a city road network demonstrate its effectiveness and ability to achieve high-quality solutions. Furthermore, we extend the problem to the multi-robot case, where the challenge lies in prioritizing robots when multiple service requests arrive simultaneously. To address this challenge, we present a greedy algorithm that efficiently prioritizes service requests in a batch and has a remarkably good performance compared to the optimal solution. The next problem focuses on allocating human operators to robots in a fleet, considering each robot's specified route and the potential for failures and getting stuck. Conventional techniques used to solve such problems face scalability issues due to exponential growth of state and action spaces with the number of robots and operators. To overcome these, we derive conditions for a technical requirement called indexability, thereby enabling the use of the Whittle index heuristic. Our key insight is to leverage the structure of the value function of individual robots, resulting in conditions that can be easily verified separately for each state of each robot. We apply these conditions to two types of transitions commonly seen in supervised robot fleets. Through numerical simulations, we demonstrate the efficacy of Whittle index policy as a near-optimal scalable approach that outperforms existing scalable methods. Finally, we investigate the impact of interruptions on human supervisors overseeing a fleet of robots. Human supervisors in such systems are primarily responsible for monitoring robots, but can also be assigned with secondary tasks. These tasks can act as interruptions and can be categorized as either intrinsic, i.e., being directly related to the monitoring task, or extrinsic, i.e., being unrelated. Through a user study involving $39$ participants, the findings reveal that task performance remains relatively unaffected by interruptions, and is primarily dependent on the number of robots being monitored. However, extrinsic interruptions led to a significant increase in perceived workload, creating challenges in switching between tasks. These results highlight the importance of managing user workload by limiting extrinsic interruptions in such supervision systems. Overall, this thesis contributes to the field of robot planning and operator allocation in collaborative human-robot teams. By incorporating human assistance, addressing scalability challenges, and understanding the impact of interruptions, we aim to enhance the performance and usability of robot fleets. Our work introduces optimal planning methods and efficient allocation strategies, empowering the seamless operation of robot fleets in real-world scenarios. Additionally, we provide valuable insights into user workload, shedding light on the interactions between humans and robots in such systems. We hope that our research promotes the widespread adoption of robot fleets and facilitates their integration into various domains, ultimately driving advancements in the field

How to address smart homes with a social robot? A multi-modal corpus of user interactions with an intelligent environment

Holthaus P, Leichsenring C, Bernotat J, et al. How to address smart homes with a social robot? A multi-modal corpus of user interactions with an intelligent environment. In: Calzolari N, ed. LREC 2016, Tenth International Conference on Language Resources and Evaluation. [Proceedings]. Paris: European Language Resources Association (ELRA); 2016: 3440-3446.In order to explore intuitive verbal and non-verbal interfaces in smart environments we recorded user interactions with an intelligent apartment. Besides offering various interactive capabilities itself, the apartment is also inhabited by a social robot that is available as a humanoid interface. This paper presents a multi-modal corpus that contains goal-directed actions of naive users in attempts to solve a number of predefined tasks. Alongside audio and video recordings, our data-set consists of large amount of temporally aligned sensory data and system behavior provided by the environment and its interactive components. Non-verbal system responses such as changes in light or display contents, as well as robot and apartment utterances and gestures serve as a rich basis for later in-depth analysis. Manual annotations provide further information about meta data like the current course of study and user behavior including the incorporated modality, all literal utterances, language features, emotional expressions, foci of attention, and addressees

Autonomous decision-making for socially interactive robots

Mención Internacional en el título de doctorThe aim of this thesis is to present a novel decision-making system based on bio-inspired concepts to decide the actions to make during the interaction between humans and robots. We use concepts from nature to make the robot may behave analogously to a living being for a better acceptance by people. The system is applied to autonomous Socially Interactive Robots that works in environments with users. These objectives are motivated by the need of having robots collaborating, entertaining or helping in educational tasks for real situations with children or elder people where the robot has to behave socially. Moreover, the decision-making system can be integrated into this kind of robots in order to learn how to act depending on the user profile the robot is interacting with. The decision-making system proposed in this thesis is a solution to all these issues in addition to a complement for interactive learning in HRI. We also show real applications of the system proposed applying it in an educational scenario, a situation where the robot can learn and interact with different kinds of people. The last goal of this thesis is to develop a robotic architecture that is able to learn how to behave in different contexts where humans and robots coexist. For that purpose, we design a modular and portable robotic architecture that is included in several robots. Including well-known software engineering techniques together with innovative agile software development procedures that produces an easily extensible architecture.El objetivo de esta tesis es presentar un novedoso sistema de toma de decisiones basado en conceptos bioinspirados para decidir las acciones a realizar durante la interacción entre personas y robots. Usamos conceptos de la naturaleza para hacer que el robot pueda comportarse análogamente a un ser vivo para una mejor aceptación por las personas. El sistema está desarrollado para que se pueda aplicar a los llamados Robots Socialmente Interactivos que están destinados a entornos con usuarios. Estos objetivos están motivados por la necesidad de tener robots en tareas de colaboración, entretenimiento o en educación en situaciones reales con niños o personas mayores en las cuales el robot debe comportarse siguiendo las normas sociales. Además, el sistema de toma de decisiones es integrado en estos tipos de robots con el fin de que pueda aprender a actuar dependiendo del perfil de usuario con el que el robot está interactuando. El sistema de toma de decisiones que proponemos en esta tesis es una solución a todos estos desafíos además de un complemento para el aprendizaje interactivo en la interacción humano-robot. También mostramos aplicaciones reales del sistema propuesto aplicándolo en un escenario educativo, una situación en la que el robot puede aprender e interaccionar con diferentes tipos de personas. El último objetivo de esta tesis es desarrollar un arquitectura robótica que sea capaz de aprender a comportarse en diferentes contextos donde las personas y los robots coexistan. Con ese propósito, diseñamos una arquitectura robótica modular y portable que está incluida en varios robots. Incluyendo técnicas bien conocidas de ingeniería del software junto con procedimientos innovadores de desarrollo de sofware ágil que producen una arquitectura fácilmente extensible.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Fabio Bonsignorio.- Secretario: María Dolores Blanco Rojas.- Vocal: Martin Stoele

Crowd of oz : A crowd-powered social robotics system for stress management

Coping with stress is crucial for a healthy lifestyle. In the past, a great deal of research has been conducted to use socially assistive robots as a therapy to alleviate stress and anxiety related problems. However, building a fully autonomous social robot which can deliver psycho-therapeutic solutions is a very challenging endeavor due to limitations in artificial intelligence (AI). To overcome AI’s limitations, researchers have previously introduced crowdsourcing-based teleoperation methods, which summon the crowd’s input to control a robot’s functions. However, in the context of robotics, such methods have only been used to support the object manipulation, navigational, and training tasks. It is not yet known how to leverage real-time crowdsourcing (RTC) to process complex therapeutic conversational tasks for social robotics. To fill this gap, we developed Crowd of Oz (CoZ), an open-source system that allows Softbank’s Pepper robot to support such conversational tasks. To demonstrate the potential implications of this crowd-powered approach, we investigated how effectively, crowd workers recruited in real-time can teleoperate the robot’s speech, in situations when the robot needs to act as a life coach. We systematically varied the number of workers who simultaneously handle the speech of the robot (N = 1, 2, 4, 8) and investigated the concomitant effects for enabling RTC for social robotics. Additionally, we present Pavilion, a novel and open-source algorithm for managing the workers’ queue so that a required number of workers are engaged or waiting. Based on our findings, we discuss salient parameters that such crowd-powered systems must adhere to, so as to enhance their performance in response latency and dialogue quality. © 2020 by the authors. Licensee MDPI, Basel, Switzerland