Real-time robotic tasks for cyber-physical avatars

Although modern robots can perform complex tasks using sophisticated algorithms that are specialized to a particular task and environment, creating robots capable of completing tasks in unstructured environments without human guidance (e.g., through teleoperation) remains a challenge. In this research, we present a framework to meet this challenge for a "cyberphysical avatar," which is defined to be a semi-autonomous robotic system that adjusts to an unstructured environment and performs physical tasks subject to critical timing constraints while under human supervision. This thesis first realizes a cyberphysical avatar that integrates three key technologies: (1) whole body-compliant control, (2) skill acquisition from machine learning (neuroevolution methods and deep learning), and (3) vision-based control through visual servoing. Body-compliant control is essential for operator safety because avatars perform cooperative tasks in close proximity to humans; machine learning enables "programming" avatars such that they can be used by non-experts for a large array of tasks, some unforeseen, in an unstructured environment; the visual servoing technique is indispensable for facilitating feedback control in human avatar interaction. This thesis proposes and demonstrates a systematically incremental approach to automating robotic tasks by decomposing a non-trivial task into stages, each of which may be automated by integrating the aforementioned techniques. We design and implement the controllers for two semi-autonomous robots that integrate three key techniques for grasping and pick-and-place tasks. While a general theory is beyond reach, we present a study on the tradeoffs between three design metrics for robotic task systems: (1) the amount of training effort for the robots to perform the task, (2) the time available to complete the task when the command is given, and (3) the quality of the result of the performed task. The tradeoff study in this design space uses the imprecise computation model as a framework to evaluate specific types of tasks: (1) grasping an unknown object and (2) placing the object in a target position. We demonstrate the generality of our integration methodology by applying it to two different robots, Dreamer and Hoppy. Our approach is evaluated by the performance of the robots in trading off between task completion time, training time and task completion success rate, in an environment similar to those in the recent Amazon Picking Challenge.Computer Science

Combined heuristic task and motion planning for bi-manual robots

Planning efficiently at task and motion levels allows the setting of new challenges for robotic manipulation problems, like for instance constrained table-top problems for bi-manual robots. In this scope, the appropriate combination of task and motion planning levels plays an important role. Accordingly, a heuristic-based task and motion planning approach is proposed, in which the computation of the heuristic addresses a geometrically relaxed problem, i.e., it only reasons upon objects placements, grasp poses, and inverse kinematics solutions. Motion paths are evaluated lazily, i.e., only after an action has been selected by the heuristic. This reduces the number of calls to the motion planner, while backtracking is reduced because the heuristic captures most of the geometric constraints. The approach has been validated in simulation and on a real robot, with different classes of table-top manipulation problems. Empirical comparison with recent approaches solving similar problems is also reported, showing that the proposed approach results in significant improvement both in terms of planing time and success rate.Peer ReviewedPostprint (author's final draft

towards intelligent autonomous sorting of unclassified nuclear wastes

Sorting of old and mixed nuclear waste is an essential process in nuclear decommissioning operations. The main bottleneck is manual picking and separation of the materials using remotely operated arms, which is slow and error prone especially with small items. Automation of the process is therefore desirable. In the framework of the newly funded European project ECHORD++, experiment RadioRoSo, a pilot robotic cell is being developed and validated against industrial requirements on a range of sorting tasks. Industrial robots, custom gripper, vision feedback and new manipulation skills will be developed. This paper presents application context, cell layout and sorting approach

Combining task and motion planning for mobile manipulators

Aplicat embargament des de la data de defensa fins el dia 31/12/2019Premi Extraordinari de Doctorat, promoció 2018-2019. Àmbit d’Enginyeria IndustrialThis thesis addresses the combination of task and motion planning which deals with different types of robotic manipulation problems. Manipulation problems are referred to as mobile manipulation, collaborative multiple mobile robots tasks, and even higher dimensional tasks (like bi-manual robots or mobile manipulators). Task and motion planning problems needs to obtain a geometrically feasible manipulation plan through symbolic and geometric search space. The combination of task and motion planning levels has emerged as a challenging issue as the failure leads robots to dead-end tasks due to geometric constraints. In addition, task planning is combined with physics-based motion planning and information to cope with manipulation tasks in which interactions between robots and objects are required, or also a low-cost feasible plan in terms of power is looked for. Moreover, combining task and motion planning frameworks is enriched by introducing manipulation knowledge. It facilitates the planning process and aids to provide the way of executing symbolic actions. Combining task and motion planning can be considered under uncertain information and with human-interaction. Uncertainty can be viewed in the initial state of the robot world or the result of symbolic actions. To deal with such issues, contingent-based task and motion planning is proposed using a perception system and human knowledge. Also, robots can ask human for those tasks which are difficult or infeasible for the purpose of collaboration. An implementation framework to combine different types of task and motion planning is presented. All the required modules and tools are also illustrated. As some task planning algorithms are implemented in Prolog or C++ languages and our geometric reasoner is developed in C++, the flow of information between different languages is explained.Aquesta tesis es centra en les eines de planificació combinada a nivell de tasca i a nivell de moviments per abordar diferents problemes de manipulació robòtica. Els problemes considerats són de navegació de robots mòbil enmig de obstacles no fixes, tasques de manipulació cooperativa entre varis robots mòbils, i tasques de manipulació de dimensió més elevada com les portades a terme amb robots bi-braç o manipuladors mòbils. La planificació combinada de tasques i de moviments ha de cercar un pla de manipulació que sigui geomètricament realitzable, a través de d'un espai de cerca simbòlic i geomètric. La combinació dels nivells de planificació de tasca i de moviments ha sorgit com un repte ja que les fallades degudes a les restriccions geomètriques poden portar a tasques sense solució. Addicionalment, la planificació a nivell de tasca es combina amb informació de la física de l'entorn i amb mètodes de planificació basats en la física, per abordar tasques de manipulació en les que la interacció entre el robot i els objectes és necessària, o també si es busca un pla realitzable i amb un baix cost en termes de potència. A més, el marc proposat per al combinació de la planificació a nivell de tasca i a nivell de moviments es millora mitjançant l'ús de coneixement, que facilita el procés de planificació i ajuda a trobar la forma d'executar accions simbòliques. La combinació de nivells de planificació també es pot considerar en casos d'informació incompleta i en la interacció humà-robot. La incertesa es considera en l'estat inicial i en el resultat de les accions simbòliques. Per abordar aquest problema, es proposa la planificació basada en contingències usant un sistema de percepció i el coneixement de l'operari humà. Igualment, els robots poden demanar col·laboració a l'operari humà per a que realitzi aquelles accions que són difícils o no realitzables pel robot. Es presenta també un marc d'implementació per a la combinació de nivells de planificació usant diferents mètodes, incloent tots els mòduls i eines necessàries. Com que alguns algorismes estan implementats en Prolog i d'altres en C++, i el mòdul de raonament geomètric proposat està desenvolupat en C++, es detalla el flux d'informació entre diferents llenguatges.Award-winningPostprint (published version

Object search and retrieval in indoor environment using a Mobile Manipulator

Robots are increasingly viewed as service agents in offices and homes. In many countries where the average population is aging, robots can be used for elderly care. This Thesis explores one such possibility using a mobile manipulator robot. Such robots have a mobile base to move from one place to another and an arm to pick and place objects. This Thesis considers a problem where the mobile manipulator needs to search for an object in an environment and bring it to some location. The optimal object search is formulated in terms of the popular traveling salesman problem (TSP) that computes the optimal sequence in which the Robot can visit all the possible locations where the object can possibly be. Prior information about the more likely locations is brought in by scaling the edge-weight of the TSP graph through the probabilities of the location. The Thesis can combine the output of TSP with navigation and manipulation planning built on top of Robot Operating Systems (ROS) to build the complete object search and retrieval pipeline. The results of the Thesis are validated both in simulation and actual hardware experiments

Hierarchical Visual Policy Learning for Long-Horizon Robot Manipulation in Densely Cluttered Scenes

In this work, we focus on addressing the long-horizon manipulation tasks in densely cluttered scenes. Such tasks require policies to effectively manage severe occlusions among objects and continually produce actions based on visual observations. We propose a vision-based Hierarchical policy for Cluttered-scene Long-horizon Manipulation (HCLM). It employs a high-level policy and three options to select and instantiate three parameterized action primitives: push, pick, and place. We first train the pick and place options by behavior cloning (BC). Subsequently, we use hierarchical reinforcement learning (HRL) to train the high-level policy and push option. During HRL, we propose a Spatially Extended Q-update (SEQ) to augment the updates for the push option and a Two-Stage Update Scheme (TSUS) to alleviate the non-stationary transition problem in updating the high-level policy. We demonstrate that HCLM significantly outperforms baseline methods in terms of success rate and efficiency in diverse tasks. We also highlight our method's ability to generalize to more cluttered environments with more additional blocks

Task planning using physics-based heuristics on manipulation actions

© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.In order to solve mobile manipulation problems, the efficient combination of task and motion planning is usually required. Moreover, the incorporation of physics-based information has recently been taken into account in order to plan the tasks in a more realistic way. In the present paper, a task and motion planning framework is proposed based on a modified version of the Fast-Forward task planner that is guided by physics-based knowledge. The proposal uses manipulation knowledge for reasoning on symbolic literals (both in offline and online modes) taking into account geometric information in order to evaluate the applicability as well as feasibility of actions while evaluating the heuristic cost. It results in an efficient search of the state space and in the obtention of low-cost physically-feasible plans. The proposal has been implemented and is illustrated with a manipulation problem consisting of a mobile robot and some fixed and manipulatable objects.Peer ReviewedPostprint (author's final draft

Task Planning and Execution for Human Robot Team Performing a Shared Task in a Shared Workspace

A cyber-physical system is developed to enable a human-robot team to perform a shared task in a shared workspace. The system setup is suitable for the implementation of a tabletop manipulation task, a common human-robot collaboration scenario. The system integrates elements that exist in the physical (real) and the virtual world. In this work, we report the insights we gathered throughout our exploration in understanding and implementing task planning and execution for human-robot team

Reasoning about space for human-robot interaction

L'interaction Homme-Robot est un domaine de recherche qui se développe de manière exponentielle durant ces dernières années, ceci nous procure de nouveaux défis au raisonnement géométrique du robot et au partage d'espace. Le robot pour accomplir une tâche, doit non seulement raisonner sur ses propres capacités, mais également prendre en considération la perception humaine, c'est à dire "Le robot doit se placer du point de vue de l'humain". Chez l'homme, la capacité de prise de perspective visuelle commence à se manifester à partir du 24ème mois. Cette capacité est utilisée pour déterminer si une autre personne peut voir un objet ou pas. La mise en place de ce genre de capacités sociales améliorera les capacités cognitives du robot et aidera le robot pour une meilleure interaction avec les hommes. Dans ce travail, nous présentons un mécanisme de raisonnement spatial de point de vue géométrique qui utilise des concepts psychologiques de la "prise de perspective" et "de la rotation mentale" dans deux cadres généraux: - La planification de mouvement pour l'interaction homme-robot: le robot utilise "la prise de perspective égocentrique" pour évaluer plusieurs configurations où le robot peut effectuer différentes tâches d'interaction. - Une interaction face à face entre l'homme et le robot : le robot emploie la prise de point de vue de l'humain comme un outil géométrique pour comprendre l'attention et l'intention humaine afin d'effectuer des tâches coopératives.Human Robot Interaction is a research area that is growing exponentially in last years. This fact brings new challenges to the robot's geometric reasoning and space sharing abilities. The robot should not only reason on its own capacities but also consider the actual situation by looking from human's eyes, thus "putting itself into human's perspective". In humans, the "visual perspective taking" ability begins to appear by 24 months of age and is used to determine if another person can see an object or not. The implementation of this kind of social abilities will improve the robot's cognitive capabilities and will help the robot to perform a better interaction with human beings. In this work, we present a geometric spatial reasoning mechanism that employs psychological concepts of "perspective taking" and "mental rotation" in two general frameworks: - Motion planning for human-robot interaction: where the robot uses "egocentric perspective taking" to evaluate several configurations where the robot is able to perform different tasks of interaction. - A face-to-face human-robot interaction: where the robot uses perspective taking of the human as a geometric tool to understand the human attention and intention in order to perform cooperative tasks