Interactive Imitation Learning in Robotics: A Survey

Interactive Imitation Learning (IIL) is a branch of Imitation Learning (IL) where human feedback is provided intermittently during robot execution allowing an online improvement of the robot's behavior. In recent years, IIL has increasingly started to carve out its own space as a promising data-driven alternative for solving complex robotic tasks. The advantages of IIL are its data-efficient, as the human feedback guides the robot directly towards an improved behavior, and its robustness, as the distribution mismatch between the teacher and learner trajectories is minimized by providing feedback directly over the learner's trajectories. Nevertheless, despite the opportunities that IIL presents, its terminology, structure, and applicability are not clear nor unified in the literature, slowing down its development and, therefore, the research of innovative formulations and discoveries. In this article, we attempt to facilitate research in IIL and lower entry barriers for new practitioners by providing a survey of the field that unifies and structures it. In addition, we aim to raise awareness of its potential, what has been accomplished and what are still open research questions. We organize the most relevant works in IIL in terms of human-robot interaction (i.e., types of feedback), interfaces (i.e., means of providing feedback), learning (i.e., models learned from feedback and function approximators), user experience (i.e., human perception about the learning process), applications, and benchmarks. Furthermore, we analyze similarities and differences between IIL and RL, providing a discussion on how the concepts offline, online, off-policy and on-policy learning should be transferred to IIL from the RL literature. We particularly focus on robotic applications in the real world and discuss their implications, limitations, and promising future areas of research

SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents

Building embodied autonomous agents capable of participating in social interactions with humans is one of the main challenges in AI. Within the Deep Reinforcement Learning (DRL) field, this objective motivated multiple works on embodied language use. However, current approaches focus on language as a communication tool in very simplified and non-diverse social situations: the "naturalness" of language is reduced to the concept of high vocabulary size and variability. In this paper, we argue that aiming towards human-level AI requires a broader set of key social skills: 1) language use in complex and variable social contexts; 2) beyond language, complex embodied communication in multimodal settings within constantly evolving social worlds. We explain how concepts from cognitive sciences could help AI to draw a roadmap towards human-like intelligence, with a focus on its social dimensions. As a first step, we propose to expand current research to a broader set of core social skills. To do this, we present SocialAI, a benchmark to assess the acquisition of social skills of DRL agents using multiple grid-world environments featuring other (scripted) social agents. We then study the limits of a recent SOTA DRL approach when tested on SocialAI and discuss important next steps towards proficient social agents. Videos and code are available at https://sites.google.com/view/socialai.Comment: under review. This paper extends and generalizes work in arXiv:2104.1320

Rating-based Reinforcement Learning

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforcement learning paradigms, based on human relative preferences over sample pairs, the proposed rating-based reinforcement learning approach is based on human evaluation of individual trajectories without relative comparisons between sample pairs. The rating-based reinforcement learning approach builds on a new prediction model for human ratings and a novel multi-class loss function. We conduct several experimental studies based on synthetic ratings and real human ratings to evaluate the effectiveness and benefits of the new rating-based reinforcement learning approach.Comment: This is an extended version of the paper "Rating-based Reinforcement Learning" accepted to the 38th Annual AAAI Conference on Artificial Intelligenc

Complex System Governance Leadership

The purpose of this research was to develop a systems theory-based framework for leadership in governance of complex systems. Recognizing complexity and uncertainty as norms for the environments in which organizations exist encouraged researchers to suggest complexity theory, complex systems, and complex adaptive systems as appropriate for addressing these conditions. Complex System Governance (CSG), based in systems theory, management cybernetics, and governance, endeavors to provide for the design, execution and evolution of functions that provide control, communication, coordination, and integration at the metasystem level to support operations and continued system existence (viability). From a management cybernetics perspective, CSG leadership has a role in the design of the metasystem that provides governance functions for a complex system. Similarly, leadership assures the existence of conditions necessary for the requisite metasystem functions to be enabled, executed, and evolved sufficiently for continued system viability. In this research, CSG leadership functions were examined from a system theoretic perspective. An extensive body of leadership literature provides insight into leadership from a number of perspectives including leadership as personal traits, leadership as a set of skills, or leadership as a process or relationship. Systems theory conceptual foundations applied to CSG leadership functions are not represented in this literature thus resulting in a gap. This research contributes to addressing that gap by linking systems theory to leadership functions for CSG. The research was a journey of discovery with no pre-established hypotheses that could be tested using deductive approaches, therefore, an inductive approach supportive of exploring, understanding (gaining insight) and discovery was employed. As the purpose was to develop a systems theory-based framework for leadership in governance of complex systems, theory construction was required. As a recognized methodology to discover theory from data, Grounded Theory was chosen as the research methodology. The framework that resulted from this research presents a novel contribution to CSG leadership that is grounded in systems theory and management cybernetics. It also provides practitioners the opportunity to develop novel approaches for facilitating anticipation, identification, and remediation of leadership issues

Assessing Performance, Role Sharing, and Control Mechanisms in Human-Human Physical Interaction for Object Manipulation

abstract: Object manipulation is a common sensorimotor task that humans perform to interact with the physical world. The first aim of this dissertation was to characterize and identify the role of feedback and feedforward mechanisms for force control in object manipulation by introducing a new feature based on force trajectories to quantify the interaction between feedback- and feedforward control. This feature was applied on two grasp contexts: grasping the object at either (1) predetermined or (2) self-selected grasp locations (“constrained” and “unconstrained”, respectively), where unconstrained grasping is thought to involve feedback-driven force corrections to a greater extent than constrained grasping. This proposition was confirmed by force feature analysis. The second aim of this dissertation was to quantify whether force control mechanisms differ between dominant and non-dominant hands. The force feature analysis demonstrated that manipulation by the dominant hand relies on feedforward control more than the non-dominant hand. The third aim was to quantify coordination mechanisms underlying physical interaction by dyads in object manipulation. The results revealed that only individuals with worse solo performance benefit from interpersonal coordination through physical couplings, whereas the better individuals do not. This work showed that naturally emerging leader-follower roles, whereby the leader in dyadic manipulation exhibits significant greater force changes than the follower. Furthermore, brain activity measured through electroencephalography (EEG) could discriminate leader and follower roles as indicated power modulation in the alpha frequency band over centro-parietal areas. Lastly, this dissertation suggested that the relation between force and motion (arm impedance) could be an important means for communicating intended movement direction between biological agents.Dissertation/ThesisDoctoral Dissertation Biomedical Engineering 201

Toward Real-Time Decentralized Reinforcement Learning using Finite Support Basis Functions

This paper addresses the design and implementation of complex Reinforcement Learning (RL) behaviors where multi-dimensional action spaces are involved, as well as the need to execute the behaviors in real-time using robotic platforms with limited computational resources and training times. For this purpose, we propose the use of decentralized RL, in combination with finite support basis functions as alternatives to Gaussian RBF, in order to alleviate the effects of the curse of dimensionality on the action and state spaces respectively, and to reduce the computation time. As testbed, a RL based controller for the in-walk kick in NAO robots, a challenging and critical problem for soccer robotics, is used. The reported experiments show empirically that our solution saves up to 99.94% of execution time and 98.82% of memory consumption during execution, without diminishing performance compared to classical approaches.Comment: Accepted in the RoboCup Symposium 2017. Final version will be published at Springe

How sustainability actions are influenced by management control systems: a levers of control perspective

openLa tesi si propone di identificare i molteplici usi del sistema di controllo manageriale e come questi siano d'aiuto ad implementare la sostenibilità nella strategia aziendale, attraverso lo studio delle quattro leve di controllo descritte da Simons (1995) che inquadrano i sistemi di controllo manageriale. Attraverso l'analisi di un campione di 18 aziende italiane medio-grandi, la tesi ha l'obiettivo di portare i concetti puramente teorici già trattati nella letteratura a riguardo ad un livello di esaminazione più empirico e pragmatico. In particolare, il capitolo empirico analizza: l'adattamento degli obiettivi aziendali, del processo di formulazione dei KPI e dei sistemi di di misurazione delle performance e di remunerazione per permettere l' integrazione della sostenibilità all'interno della strategia; le opportunità e le incertezze che risultano dalla nuova strategia di sostenibilità; i rischi e le limitazioni che risultano dall'introduzione della sostenibilità nelle azioni e nei processi aziendali; l'influenza dei valori e della mission aziendale dall'inserimento delle azioni di sostenibilità.This thesis aims to identify the different uses of MCS and how they help to implement the sustainability into the business strategy, through the study of the four levers of control described by Simons (1995) that frame managerial control systems. Through the analysis of a sample of eighteen medium-large Italian companies, the paper is intended to bring merely theoretical concepts already detailed in the literature on the subject, to a more empirical and pragmatic level of examination. In particular, the empirical section analyses: the adaptation of the objectives, the KPI’s formulation process and performance and reward systems to enable the integration of sustainability within the strategy; the opportunities and uncertainties that results from the new sustainable strategy; the risks and limitation coming from the employment of sustainability within actions and processes; the influence of values and mission by the introduction of sustainable activities