Multiform Adaptive Robot Skill Learning from Humans

Object manipulation is a basic element in everyday human lives. Robotic manipulation has progressed from maneuvering single-rigid-body objects with firm grasping to maneuvering soft objects and handling contact-rich actions. Meanwhile, technologies such as robot learning from demonstration have enabled humans to intuitively train robots. This paper discusses a new level of robotic learning-based manipulation. In contrast to the single form of learning from demonstration, we propose a multiform learning approach that integrates additional forms of skill acquisition, including adaptive learning from definition and evaluation. Moreover, going beyond state-of-the-art technologies of handling purely rigid or soft objects in a pseudo-static manner, our work allows robots to learn to handle partly rigid partly soft objects with time-critical skills and sophisticated contact control. Such capability of robotic manipulation offers a variety of new possibilities in human-robot interaction.Comment: Accepted to 2017 Dynamic Systems and Control Conference (DSCC), Tysons Corner, VA, October 11-1

Robot Composite Learning and the Nunchaku Flipping Challenge

Advanced motor skills are essential for robots to physically coexist with humans. Much research on robot dynamics and control has achieved success on hyper robot motor capabilities, but mostly through heavily case-specific engineering. Meanwhile, in terms of robot acquiring skills in a ubiquitous manner, robot learning from human demonstration (LfD) has achieved great progress, but still has limitations handling dynamic skills and compound actions. In this paper, we present a composite learning scheme which goes beyond LfD and integrates robot learning from human definition, demonstration, and evaluation. The method tackles advanced motor skills that require dynamic time-critical maneuver, complex contact control, and handling partly soft partly rigid objects. We also introduce the "nunchaku flipping challenge", an extreme test that puts hard requirements to all these three aspects. Continued from our previous presentations, this paper introduces the latest update of the composite learning scheme and the physical success of the nunchaku flipping challenge

Data-driven learning for robot physical intelligence

The physical intelligence, which emphasizes physical capabilities such as dexterous manipulation and dynamic mobility, is essential for robots to physically coexist with humans. Much research on robot physical intelligence has achieved success on hyper robot motor capabilities, but mostly through heavily case-specific engineering. Meanwhile, in terms of robot acquiring skills in a ubiquitous manner, robot learning from human demonstration (LfD) has achieved great progress, but still has limitations handling dynamic skills and compound actions. In this dissertation, a composite learning scheme which goes beyond LfD and integrates robot learning from human definition, demonstration, and evaluation is proposed. This method tackles advanced motor skills that require dynamic time-critical maneuver, complex contact control, and handling partly soft partly rigid objects. Besides, the power of crowdsourcing is brought to tackle case-specific engineering problem in the robot physical intelligence. Crowdsourcing has demonstrated great potential in recent development of artificial intelligence. Constant learning from a large group of human mentors breaks the limit of learning from one or a few mentors in individual cases, and has achieved success in image recognition, translation, and many other cyber applications. A robot learning scheme that allows a robot to synthesize new physical skills using knowledge acquired from crowdsourced human mentors is proposed. The work is expected to provide a long-term and big-scale measure to produce advanced robot physical intelligence

Efficient learning of sequential tasks for collaborative robots: a neurodynamic approach

Dissertação de mestrado integrado em Engenharia Eletrónica, Industrial e ComputadoresIn the recent years, there has been an increasing demand for collaborative robots able to interact and co operate with ordinary people in several human environments, sharing physical space and working closely with people in joint tasks, both within industrial and domestic environments. In some scenarios, these robots will come across tasks that cannot be fully designed beforehand, resulting in a need for flexibility and adaptation to the changing environments. This dissertation aims to endow robots with the ability to acquire knowledge of sequential tasks using the Programming by Demonstration (PbD) paradigm. Concretely, it extends the learning models - based on Dynamic Neural Fields (DNFs) - previously developed in the Mobile and Anthropomorphic Robotics Laboratory (MARLab), at the University of Minho, to the collaborative robot Sawyer, which is amongst the newest collaborative robots on the market. The main goal was to endow Sawyer with the ability to learn a sequential task from tutors’ demonstrations, through a natural and efficient process. The developed work can be divided into three main tasks: (1) first, a previously developed neuro-cognitive control architecture for extracting the sequential structure of a task was implemented and tested in Sawyer, combined with a Short-Term Memory (STM) mechanism to memorize a sequence in one-shot, aiming to reduce the number of demonstration trials; (2) second, the previous model was extended to incorporate workspace information and action selection in a Human-Robot Collaboration (HRC) scenario where robot and human co worker coordinate their actions to construct the structure; and (3) third, the STM mechanism was also extended to memorize ordinal and temporal aspects of the sequence, demonstrated by tutors with different behavior time scales. The models implemented contributed to a more intuitive and practical interaction with the robot for human co-workers. The STM model made the learning possible from few demonstrations to comply with the requirement of being an efficient method for learning. Moreover, the recall of the memorized information allowed Sawyer to evolve from being in a learning position to be in a teaching one, obtaining the capability of assisting inexperienced co-workers.Nos últimos anos, tem havido uma crescente procura por robôs colaborativos capazes de interagir e cooperar com pessoas comuns em vários ambientes, partilhando espaço físico e trabalhando em conjunto, tanto em ambientes industriais como domésticos. Em alguns cenários, estes robôs serão confrontados com tarefas que não podem ser previamente planeadas, o que resulta numa necessidade de existir flexibilidade e adaptação ao ambiente que se encontra em constante mudança. Esta dissertação pretende dotar robôs com a capacidade de adquirir conhecimento de tarefas sequenciais utilizando técnicas de Programação por Demonstração. De forma a continuar o trabalho desenvolvido no Laboratório de Robótica Móvel e Antropomórfica da Universidade do Minho, esta dissertação visa estender os modelos de aprendizagem previamente desenvolvidos ao robô colaborativo Sawyer, que é um dos mais recentes no mercado. O principal objetivo foi dotar o robô com a capacidade de aprender tarefas sequenciais por demonstração, através de um processo natural e eficiente. O trabalho desenvolvido pode ser dividido em três tarefas principais: (1) em primeiro lugar, uma arquitetura de controlo baseada em modelos neurocognitivos, desenvolvida anteriormente, para aprender a estrutura de uma tarefa sequencial foi implementada e testada no robô Sawyer, conjugada com um mecanismo de Short Term Memory que permitiu memorizar uma sequência apenas com uma demonstração, para reduzir o número de demonstrações necessárias; (2) em segundo lugar, o modelo anterior foi estendido para englobar informação acerca do espaço de trabalho e seleção de ações num cenário de Colaboração Humano-Robô em que ambos coordenam as suas ações para construir a tarefa; (3) em terceiro lugar, o mecanismo de Short-Term Memory foi também estendido para memorizar informação ordinal e temporal de uma sequência de passos demonstrada por tutores com comportamentos temporais diferentes. Os modelos implementados contribuíram para uma interação com o robô mais intuitiva e prática para os co-workers humanos. O mecanismo de Short-Term Memory permitiu que a aprendizagem fosse realizada a partir de poucas demonstrações, para cumprir com o requisito de ser um método de aprendizagem eficiente. Além disso, a informação memorizada permitiu ao Sawyer evoluir de uma posição de aprendizagem para uma posição em que é capaz de instruir co-workers inexperientes.This work was carried out within the scope of the project “PRODUTECH SIF - Soluções para a Indústria do Futuro”, reference POCI-01-0247-FEDER-024541, cofunded by “Fundo Europeu de Desenvolvimento Regional (FEDER)”, through “Programa Operacional Competitividade e Internacionalização (POCI)”

Complementary Actions

Human beings come into the world wired for social interaction. At the fourteenth week of gestation, twin fetuses already display interactive movements specifically directed towards their co- twin. Readiness for social interaction is also clearly expressed by the newborn who imitate facial gestures, suggesting that there is a common representation mediating action observation and execution. While actions that are observed and those that are planned seem to be functionally equivalent, it is unclear if the visual representation of an observed action inevitably leads to its motor representation. This is particularly true with regard to complementary actions (from the Latin complementum ; i.e. that fills up), a specific class of movements which differ, while interacting, with observed ones. In geometry, angles are defined as complementary if they form a right angle. In art and design, complementary colors are color pairs that, when combined in the right proportions, produce white or black. As a working definition, complementary actions refer here to any form of social interaction wherein two (or more) individuals complete each other\u2019s actions in a balanced way. Successful complementary interactions are founded on the abilities:\ua0 (1)\ua0 to simulate another person\u2019s movements; (2)\ua0 to predict another person\u2019s future action/ s; (3)\ua0to produce an appropriate congruent/ incongruent response that completes the other person\u2019s action/ s; and (4)\ua0to integrate the predicted effects of one\u2019s own and another person\u2019s actions. It is the neurophysiological mechanism that underlies this process which forms the main theme of this chapte

A survey of object goal navigation

Object Goal Navigation (ObjectNav) refers to an agent navigating to an object in an unseen environment, which is an ability often required in the accomplishment of complex tasks. Though it has drawn increasing attention from researchers in the Embodied AI community, there has not been a contemporary and comprehensive survey of ObjectNav. In this survey, we give an overview of this field by summarizing more than 70 recent papers. First, we give the preliminaries of the ObjectNav: the definition, the simulator, and the metrics. Then, we group the existing works into three categories: 1) end-to-end methods that directly map the observations to actions, 2) modular methods that consist of a mapping module, a policy module, and a path planning module, and 3) zero-shot methods that use zero-shot learning to do navigation. Finally, we summarize the performance of existing works and the main failure modes and discuss the challenges of ObjectNav. This survey would provide comprehensive information for researchers in this field to have a better understanding of ObjectNav. Note to Practitioners —This work was motivated by the increased interest in real-world applications of mobile robots. Object Goal Navigation (ObjectNav), which is an important task in these applications, requires an agent to find an object in an unseen environment. To accomplish that, the agent needs to be equipped with the capability to move in the environment, decide where to go, and recognize the object categories. So far, most works on ObjectNav have been done in a simulation environment. We present an overview of the existing works in ObjectNav and introduce them in three categories. Additionally, we analyze the current performance of ObjectNav and the challenges for future research. This paper provides researchers and practitioners with a comprehensive overview of the developed methods in ObjectNav, which can help them to have a good understanding of this task and develop suitable solutions for applications in the real world

Human-Machine Communication: Complete Volume. Volume 2

This is the complete volume of HMC Volume 2

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

Human-Machine Communication: Complete Volume. Volume 4

This is the complete volume of HMC Volume 4