XSkill: Cross Embodiment Skill Discovery

Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework. The benchmark, code, and qualitative results are on https://xskill.cs.columbia.edu

Towards endowing collaborative robots with fast learning for minimizing tutors’ demonstrations: what and when to do?

Programming by demonstration allows non-experts in robot programming to train the robots in an intuitive manner. However, this learning paradigm requires multiple demonstrations of the same task, which can be time-consuming and annoying for the human tutor. To overcome this limitation, we propose a fast learning system – based on neural dynamics – that permits collaborative robots to memorize sequential information from single task demonstrations by a human-tutor. Important, the learning system allows not only to memorize long sequences of sub-goals in a task but also the time interval between them. We implement this learning system in Sawyer (a collaborative robot from Rethink Robotics) and test it in a construction task, where the robot observes several human-tutors with different preferences on the sequential order to perform the task and different behavioral time scales. After learning, memory recall (of what and when to do a sub-task) allows the robot to instruct inexperienced human workers, in a particular human-centered task scenario.POFC - Programa Operacional Temático Factores de Competitividade(POCI-01-0247-FEDER-024541

From speech recognition to instruction learning

In robotics, learning from demonstration is a research field that has gained a lot of attention since it provides an easy and intuitive way to teach a robot complex movements. When teaching a human it is common to use natural language to provide additional information. Regardless of that, it is rarely investigated how speech can be used in addition for teaching a robot with learning from demonstration. Therefore this thesis examines how natural language is already used in robotics and how the learning process can be supported further by using speech. Furthermore a learning from demonstration system is implemented based on task spaces that are an intuitive abstraction of the state of the robot and its relation to the environment and therefore are more correlated with spoken instructions. Additional a task space selection instruction and a correction instruction are introduced to use natural language to support the learning process. They are evaluated on a real robot in an experiment where the robot has to draw a point.In der Robotik ist learning from demonstration ein Forschungsfeld, welches viel Beachtung bekommen hat, da es ermöglicht, einen Roboter auf einfache und intuitive Weise komplexe Bewegungen beizubringen. Sofern man einem Menschen etwas beibringt, ist es üblich, natürliche Sprache zu benutzen um zusätzliche Informationen zu vermitteln. Ungeachtet dessen ist es kaum erforscht worden, wie Sprache zusätzlich genutzt werden kann, um Robotern etwas mittels learning from demonstration beizubringen. Daher untersucht diese Thesis, wie natürliche Sprache in der Robotik bereits genutzt wird und wie der Lernprozess mittels Sprache besser unterstützt werden kann. Im Weiteren wird ein learning from demonstration System auf Basis von task spaces implementiert, welche eine intuitive Abstraktion des Zustands des Roboters und dessen Relation zur Umgebung sind und daher stärker mit gesprochenen Anweisungen verknüpft sind. Zusätzlich wird eine task space-Auswahl-Instruktion und eine Verbesserungs-Instruktion vorgestellt, um natürliche Sprache zur Unterstützung des Lernprozesses zu nutzen. Diese werden auf einem echten Roboter mittels eines Experiments evaluiert, bei dem der Roboter einen Punkt zeichnen muss

Robot Learning Dual-Arm Manipulation Tasks by Trial-and-Error and Multiple Human Demonstrations

In robotics, there is a need of an interactive and expedite learning method as experience is expensive. In this research, we propose two different methods to make a humanoid robot learn manipulation tasks: Learning by trial-and-error, and Learning from demonstrations. Just like the way a child learns a new task assigned to him by trying all possible alternatives and further learning from his mistakes, the robot learns in the same manner in learning by trial-and error. We used Q-learning algorithm, in which the robot tries all the possible ways to do a task and creates a matrix that consists of Q-values based on the rewards it received for the actions performed. Using this method, the robot was made to learn dance moves based on a music track. Robot Learning from Demonstrations (RLfD) enable a human user to add new capabilities to a robot in an intuitive manner without explicitly reprogramming it. In this method, the robot learns skill from demonstrations performed by a human teacher. The robot extracts features from each demonstration called as key-points and learns a model of the demonstrated task or trajectory using Hidden Markov Model (HMM). The learned model is further used to produce a generalized trajectory. In the end, we discuss the differences between two developed systems and make conclusions based on the experiments performed

GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators

Imitation learning from human demonstrations is a powerful framework to teach robots new skills. However, the performance of the learned policies is bottlenecked by the quality, scale, and variety of the demonstration data. In this paper, we aim to lower the barrier to collecting large and high-quality human demonstration data by proposing GELLO, a general framework for building low-cost and intuitive teleoperation systems for robotic manipulation. Given a target robot arm, we build a GELLO controller that has the same kinematic structure as the target arm, leveraging 3D-printed parts and off-the-shelf motors. GELLO is easy to build and intuitive to use. Through an extensive user study, we show that GELLO enables more reliable and efficient demonstration collection compared to commonly used teleoperation devices in the imitation learning literature such as VR controllers and 3D spacemouses. We further demonstrate the capabilities of GELLO for performing complex bi-manual and contact-rich manipulation tasks. To make GELLO accessible to everyone, we have designed and built GELLO systems for 3 commonly used robotic arms: Franka, UR5, and xArm. All software and hardware are open-sourced and can be found on our website: https://wuphilipp.github.io/gello/

Handover Control for Human-Robot and Robot-Robot Collaboration

Modern scenarios in robotics involve human-robot collaboration or robot-robot cooperation in unstructured environments. In human-robot collaboration, the objective is to relieve humans from repetitive and wearing tasks. This is the case of a retail store, where the robot could help a clerk to refill a shelf or an elderly customer to pick an item from an uncomfortable location. In robot-robot cooperation, automated logistics scenarios, such as warehouses, distribution centers and supermarkets, often require repetitive and sequential pick and place tasks that can be executed more efficiently by exchanging objects between robots, provided that they are endowed with object handover ability. Use of a robot for passing objects is justified only if the handover operation is sufficiently intuitive for the involved humans, fluid and natural, with a speed comparable to that typical of a human-human object exchange. The approach proposed in this paper strongly relies on visual and haptic perception combined with suitable algorithms for controlling both robot motion, to allow the robot to adapt to human behavior, and grip force, to ensure a safe handover. The control strategy combines model-based reactive control methods with an event-driven state machine encoding a human-inspired behavior during a handover task, which involves both linear and torsional loads, without requiring explicit learning from human demonstration. Experiments in a supermarket-like environment with humans and robots communicating only through haptic cues demonstrate the relevance of force/tactile feedback in accomplishing handover operations in a collaborative task