Knowledge-oriented task and motion planning for multiple mobile robots

This is an Accepted Manuscript of an article published by Taylor & Francis in Journal of experimental and theoretical artificial intelligence, published online: 30 Nov 2018 available online: https://www.tandfonline.com/doi/abs/10.1080/0952813X.2018.1544280Robotic systems composed of several mobile robots moving in human environments pose several problems at perception, planning and control levels. In these environments, there may be obstacles obstructing the paths, which robots can remove by pushing or pulling them. At planning level, therefore, an efficient combination of task and motion planning is required. Even more if we assume a cooperative system in which robots can collaborate with each other by e.g. pushing together a heavy obstacle or by one robot clearing the way to another one. In this paper, we cope with this problem by proposing ¿-TMP, a smart combination of an heuristic task planner based on the Fast Forward method, a physics-based motion planner, and reasoning processes over the ontologies that code the knowledge on the problem. The significance of the proposal relies on how geometric and physics information is used within the computation of the heuristics in order to guide the symbolic search, i.e. how an artificial intelligence planning method is combined with low-level motion planning to achieve a feasible sequence of actions (composed of collision-free motions plus physically-feasible push/pull actions). The proposal has been validated with several simulated scenarios (using up to five robots that need to collaborate with each other to reach the goal state), showing how the method is able to solve challenging situations and also find an efficient solution in terms of power.Peer ReviewedPostprint (author's final draft

Contingent task and motion planning under uncertainty for human–robot interactions

Manipulation planning under incomplete information is a highly challenging task for mobile manipulators. Uncertainty can be resolved by robot perception modules or using human knowledge in the execution process. Human operators can also collaborate with robots for the execution of some difficult actions or as helpers in sharing the task knowledge. In this scope, a contingent-based task and motion planning is proposed taking into account robot uncertainty and human–robot interactions, resulting a tree-shaped set of geometrically feasible plans. Different sorts of geometric reasoning processes are embedded inside the planner to cope with task constraints like detecting occluding objects when a robot needs to grasp an object. The proposal has been evaluated with different challenging scenarios in simulation and a real environment.Postprint (published version

PMK : a knowledge processing framework for autonomous robotics perception and manipulation

Autonomous indoor service robots are supposed to accomplish tasks, like serve a cup, which involve manipulation actions. Particularly, for complex manipulation tasks which are subject to geometric constraints, spatial information and a rich semantic knowledge about objects, types, and functionality are required, together with the way in which these objects can be manipulated. In this line, this paper presents an ontological-based reasoning framework called Perception and Manipulation Knowledge (PMK) that includes: (1) the modeling of the environment in a standardized way to provide common vocabularies for information exchange in human-robot or robot-robot collaboration, (2) a sensory module to perceive the objects in the environment and assert the ontological knowledge, (3) an evaluation-based analysis of the situation of the objects in the environment, in order to enhance the planning of manipulation tasks. The paper describes the concepts and the implementation of PMK, and presents an example demonstrating the range of information the framework can provide for autonomous robots.Peer ReviewedPostprint (published version

Semantic coupling of path planning and a primitive action of a task plan for the simulation of manipulation tasks in a virtual 3D environment

This work deals with the simulation of complex manipulation tasks in virtual environments. Validating such complex tasks, possibly to be performed under strong geometric constraints, requires considering task and path planning jointly. The contribution of this work focuses on using task-related information at the path planning level. We propose an ontology-based approach to a) model the 3D environment where the simulated task is executed, based on an original multi-level environment model involving higher abstraction level data than the purely geometric models traditionally used, and b) automatically define path planning queries for the primitive ctions of a task plan, together with task-related geometric constraints on these queries. This approach allows the improvement of the state of the art from two points of view. First, our joint task and path planning approach allows the improvement of path planning through better semantic control of the path planning process. Second, if compared to hard-coded geometric constraints, the proposed ontology-based approach introduces a more flexible ay of defining geometric constraints through an inference process, and can be adapted to different applications of manipulation tasks

Physics-based motion planning for grasping and manipulation

This thesis develops a series of knowledge-oriented physics-based motion planning algorithms for grasping and manipulation in cluttered an uncertain environments. The main idea is to use high-level knowledge-based reasoning to define the manipulation constraints that define the way how robot should interact with the objects in the environment. These interactions are modeled by incorporating the physics-based model of rigid body dynamics in planning. The first part of the thesis is focused on the techniques to integrate the knowledge with physics-based motion planning. The knowledge is represented in terms of ontologies, a prologbased knowledge inference process is introduced that defines the manipulation constraints. These constraints are used in the state validation procedure of sampling-based kinodynamic motion planners. The state propagator of the motion planner is replaced by a physics-engine that takes care of the kinodynamic and physics-based constraints. To make the interaction humanlike, a low-level physics-based reasoning process is introduced that dynamically varies the control bounds by evaluating the physical properties of the objects. As a result, power efficient motion plans are obtained. Furthermore, a framework has been presented to incorporate linear temporal logic within physics-based motion planning to handle complex temporal goals. The second part of this thesis develops physics-based motion planning approaches to plan in cluttered and uncertain environments. The uncertainty is considered in 1) objects’ poses due to sensing and due to complex robot-object or object-object interactions; 2) uncertainty in the contact dynamics (such as friction coefficient); 3) uncertainty in robot controls. The solution is framed with sampling-based kinodynamic motion planners that solve the problem in open-loop, i.e., it considers uncertainty while planning and computes the solution in such a way that it successfully moves the robot from the start to the goal configuration even if there is uncertainty in the system. To implement the above stated approaches, a knowledge-oriented physics-based motion planning tool is presented. It is developed by extending The Kautham Project, a C++ based tool for sampling-based motion planning. Finally, the current research challenges and future research directions to extend the above stated approaches are discussedEsta tesis desarrolla una serie de algoritmos de planificación del movimientos para la aprehensión y la manipulación de objetos en entornos desordenados e inciertos, basados en la física y el conocimiento. La idea principal es utilizar el razonamiento de alto nivel basado en el conocimiento para definir las restricciones de manipulación que definen la forma en que el robot debería interactuar con los objetos en el entorno. Estas interacciones se modelan incorporando en la planificación el modelo dinámico de los sólidos rígidos. La primera parte de la tesis se centra en las técnicas para integrar el conocimiento con la planificación del movimientos basada en la física. Para ello, se representa el conocimiento mediante ontologías y se introduce un proceso de razonamiento basado en Prolog para definir las restricciones de manipulación. Estas restricciones se usan en los procedimientos de validación del estado de los algoritmos de planificación basados en muestreo, cuyo propagador de estado se susituye por un motor basado en la física que tiene en cuenta las restricciones físicas y kinodinámicas. Además se ha implementado un proceso de razonamiento de bajo nivel que permite adaptar los límites de los controles aplicados a las propiedades físicas de los objetos. Complementariamente, se introduce un marco de desarrollo para la inclusión de la lógica temporal lineal en la planificación de movimientos basada en la física. La segunda parte de esta tesis extiende el enfoque a planificación del movimiento basados en la física en entornos desordenados e inciertos. La incertidumbre se considera en 1) las poses de los objetos debido a la medición y a las interacciones complejas robot-objeto y objeto-objeto; 2) incertidumbre en la dinámica de los contactos (como el coeficiente de fricción); 3) incertidumbre en los controles del robot. La solución se enmarca en planificadores kinodinámicos basados en muestro que solucionan el problema en lazo abierto, es decir que consideran la incertidumbre en la planificación para calcular una solución robusta que permita mover al robot de la configuración inicial a la final a pesar de la incertidumbre. Para implementar los enfoques mencionados anteriormente, se presenta una herramienta de planificación del movimientos basada en la física y guiada por el conocimiento, desarrollada extendiendo The Kautham Project, una herramienta implementada en C++ para la planificación de movimientos basada en muestreo. Finalmente, de discute los retos actuales y las futuras lineas de investigación a seguir para extender los enfoques presentados

Combining task and motion planning for mobile manipulators

Aplicat embargament des de la data de defensa fins el dia 31/12/2019Premi Extraordinari de Doctorat, promoció 2018-2019. Àmbit d’Enginyeria IndustrialThis thesis addresses the combination of task and motion planning which deals with different types of robotic manipulation problems. Manipulation problems are referred to as mobile manipulation, collaborative multiple mobile robots tasks, and even higher dimensional tasks (like bi-manual robots or mobile manipulators). Task and motion planning problems needs to obtain a geometrically feasible manipulation plan through symbolic and geometric search space. The combination of task and motion planning levels has emerged as a challenging issue as the failure leads robots to dead-end tasks due to geometric constraints. In addition, task planning is combined with physics-based motion planning and information to cope with manipulation tasks in which interactions between robots and objects are required, or also a low-cost feasible plan in terms of power is looked for. Moreover, combining task and motion planning frameworks is enriched by introducing manipulation knowledge. It facilitates the planning process and aids to provide the way of executing symbolic actions. Combining task and motion planning can be considered under uncertain information and with human-interaction. Uncertainty can be viewed in the initial state of the robot world or the result of symbolic actions. To deal with such issues, contingent-based task and motion planning is proposed using a perception system and human knowledge. Also, robots can ask human for those tasks which are difficult or infeasible for the purpose of collaboration. An implementation framework to combine different types of task and motion planning is presented. All the required modules and tools are also illustrated. As some task planning algorithms are implemented in Prolog or C++ languages and our geometric reasoner is developed in C++, the flow of information between different languages is explained.Aquesta tesis es centra en les eines de planificació combinada a nivell de tasca i a nivell de moviments per abordar diferents problemes de manipulació robòtica. Els problemes considerats són de navegació de robots mòbil enmig de obstacles no fixes, tasques de manipulació cooperativa entre varis robots mòbils, i tasques de manipulació de dimensió més elevada com les portades a terme amb robots bi-braç o manipuladors mòbils. La planificació combinada de tasques i de moviments ha de cercar un pla de manipulació que sigui geomètricament realitzable, a través de d'un espai de cerca simbòlic i geomètric. La combinació dels nivells de planificació de tasca i de moviments ha sorgit com un repte ja que les fallades degudes a les restriccions geomètriques poden portar a tasques sense solució. Addicionalment, la planificació a nivell de tasca es combina amb informació de la física de l'entorn i amb mètodes de planificació basats en la física, per abordar tasques de manipulació en les que la interacció entre el robot i els objectes és necessària, o també si es busca un pla realitzable i amb un baix cost en termes de potència. A més, el marc proposat per al combinació de la planificació a nivell de tasca i a nivell de moviments es millora mitjançant l'ús de coneixement, que facilita el procés de planificació i ajuda a trobar la forma d'executar accions simbòliques. La combinació de nivells de planificació també es pot considerar en casos d'informació incompleta i en la interacció humà-robot. La incertesa es considera en l'estat inicial i en el resultat de les accions simbòliques. Per abordar aquest problema, es proposa la planificació basada en contingències usant un sistema de percepció i el coneixement de l'operari humà. Igualment, els robots poden demanar col·laboració a l'operari humà per a que realitzi aquelles accions que són difícils o no realitzables pel robot. Es presenta també un marc d'implementació per a la combinació de nivells de planificació usant diferents mètodes, incloent tots els mòduls i eines necessàries. Com que alguns algorismes estan implementats en Prolog i d'altres en C++, i el mòdul de raonament geomètric proposat està desenvolupat en C++, es detalla el flux d'informació entre diferents llenguatges.Award-winningPostprint (published version

An ontology-based approach towards coupling task and path planning for the simulation of manipulation tasks

This work deals with the simulation and the validation of complex manipulation tasks under strong geometric constraints in virtual environments. The targeted applications relate to the industry 4.0 framework; as up-to-date products are more and more integrated and the economic competition increases, industrial companies express the need to validate, from design stage on, not only the static CAD models of their products but also the tasks (e.g., assembly or maintenance) related to their Product Lifecycle Management (PLM). The scientific community looked at this issue from two points of view: - Task planning decomposes a manipulation task to be realized into a sequence of primitive actions (i.e., a task plan) - Path planning computes collision-free trajectories, notably for the manipulated objects. It traditionally uses purely geometric data, which leads to classical limitations (possible high computational processing times, low relevance of the proposed trajectory concerning the task to be performed, or failure); recent works have shown the interest of using higher abstraction level data. Joint task and path planning approaches found in the literature usually perform a classical task planning step, and then check out the feasibility of path planning requests associated with the primitive actions of this task plan. The link between task and path planning has to be improved, notably because of the lack of loopback between the path planning level and the task planning level: - The path planning information used to question the task plan is usually limited to the motion feasibility where richer information such as the relevance or the complexity of the proposed path would be needed; - path planning queries traditionally use purely geometric data and/or “blind” path planning methods (e.g., RRT), and no task-related information is used at the path planning level Our work focuses on using task level information at the path planning level. The path planning algorithm considered is RRT; we chose such a probabilistic algorithm because we consider path planning for the simulation and the validation of complex tasks under strong geometric constraints. We propose an ontology-based approach to use task level information to specify path planning queries for the primitive actions of a task plan. First, we propose an ontology to conceptualize the knowledge about the 3D environment in which the simulated task takes place. The environment where the simulated task takes place is considered as a closed part of 3D Cartesian space cluttered with mobile/fixed obstacles (considered as rigid bodies). It is represented by a digital model relying on a multilayer architecture involving semantic, topologic and geometric data. The originality of the proposed ontology lies in the fact that it conceptualizes heterogeneous knowledge about both the obstacles and the free space models. Second, we exploit this ontology to automatically generate a path planning query associated to each given primitive action of a task plan. Through a reasoning process involving the primitive actions instantiated in the ontology, we are able to infer the start and the goal configurations, as well as task-related geometric constraints. Finally, a multi-level path planner is called to generate the corresponding trajectory. The contributions of this work have been validated by full simulation of several manipulation tasks under strong geometric constraints. The results obtained demonstrate that using task-related information allows better control on the RRT path planning algorithm involved to check the motion feasibility for the primitive actions of a task plan, leading to lower computational time and more relevant trajectories for primitive actions

Producció científica de l'ETSEIB a Futur. Articles de revista i publicacions a congressos dels investigadors de l'ETSEIB durant l'any 2019

Postprint (author's final draft

Physics-based motion planning for grasping and manipulation

This thesis develops a series of knowledge-oriented physics-based motion planning algorithms for grasping and manipulation in cluttered an uncertain environments. The main idea is to use high-level knowledge-based reasoning to define the manipulation constraints that define the way how robot should interact with the objects in the environment. These interactions are modeled by incorporating the physics-based model of rigid body dynamics in planning. The first part of the thesis is focused on the techniques to integrate the knowledge with physics-based motion planning. The knowledge is represented in terms of ontologies, a prologbased knowledge inference process is introduced that defines the manipulation constraints. These constraints are used in the state validation procedure of sampling-based kinodynamic motion planners. The state propagator of the motion planner is replaced by a physics-engine that takes care of the kinodynamic and physics-based constraints. To make the interaction humanlike, a low-level physics-based reasoning process is introduced that dynamically varies the control bounds by evaluating the physical properties of the objects. As a result, power efficient motion plans are obtained. Furthermore, a framework has been presented to incorporate linear temporal logic within physics-based motion planning to handle complex temporal goals. The second part of this thesis develops physics-based motion planning approaches to plan in cluttered and uncertain environments. The uncertainty is considered in 1) objects’ poses due to sensing and due to complex robot-object or object-object interactions; 2) uncertainty in the contact dynamics (such as friction coefficient); 3) uncertainty in robot controls. The solution is framed with sampling-based kinodynamic motion planners that solve the problem in open-loop, i.e., it considers uncertainty while planning and computes the solution in such a way that it successfully moves the robot from the start to the goal configuration even if there is uncertainty in the system. To implement the above stated approaches, a knowledge-oriented physics-based motion planning tool is presented. It is developed by extending The Kautham Project, a C++ based tool for sampling-based motion planning. Finally, the current research challenges and future research directions to extend the above stated approaches are discussedEsta tesis desarrolla una serie de algoritmos de planificación del movimientos para la aprehensión y la manipulación de objetos en entornos desordenados e inciertos, basados en la física y el conocimiento. La idea principal es utilizar el razonamiento de alto nivel basado en el conocimiento para definir las restricciones de manipulación que definen la forma en que el robot debería interactuar con los objetos en el entorno. Estas interacciones se modelan incorporando en la planificación el modelo dinámico de los sólidos rígidos. La primera parte de la tesis se centra en las técnicas para integrar el conocimiento con la planificación del movimientos basada en la física. Para ello, se representa el conocimiento mediante ontologías y se introduce un proceso de razonamiento basado en Prolog para definir las restricciones de manipulación. Estas restricciones se usan en los procedimientos de validación del estado de los algoritmos de planificación basados en muestreo, cuyo propagador de estado se susituye por un motor basado en la física que tiene en cuenta las restricciones físicas y kinodinámicas. Además se ha implementado un proceso de razonamiento de bajo nivel que permite adaptar los límites de los controles aplicados a las propiedades físicas de los objetos. Complementariamente, se introduce un marco de desarrollo para la inclusión de la lógica temporal lineal en la planificación de movimientos basada en la física. La segunda parte de esta tesis extiende el enfoque a planificación del movimiento basados en la física en entornos desordenados e inciertos. La incertidumbre se considera en 1) las poses de los objetos debido a la medición y a las interacciones complejas robot-objeto y objeto-objeto; 2) incertidumbre en la dinámica de los contactos (como el coeficiente de fricción); 3) incertidumbre en los controles del robot. La solución se enmarca en planificadores kinodinámicos basados en muestro que solucionan el problema en lazo abierto, es decir que consideran la incertidumbre en la planificación para calcular una solución robusta que permita mover al robot de la configuración inicial a la final a pesar de la incertidumbre. Para implementar los enfoques mencionados anteriormente, se presenta una herramienta de planificación del movimientos basada en la física y guiada por el conocimiento, desarrollada extendiendo The Kautham Project, una herramienta implementada en C++ para la planificación de movimientos basada en muestreo. Finalmente, de discute los retos actuales y las futuras lineas de investigación a seguir para extender los enfoques presentados.Postprint (published version

Knowledge-oriented task and motion planning for multiple mobile robots

This is an Accepted Manuscript of an article published by Taylor & Francis in Journal of experimental and theoretical artificial intelligence, published online: 30 Nov 2018 available online: https://www.tandfonline.com/doi/abs/10.1080/0952813X.2018.1544280Robotic systems composed of several mobile robots moving in human environments pose several problems at perception, planning and control levels. In these environments, there may be obstacles obstructing the paths, which robots can remove by pushing or pulling them. At planning level, therefore, an efficient combination of task and motion planning is required. Even more if we assume a cooperative system in which robots can collaborate with each other by e.g. pushing together a heavy obstacle or by one robot clearing the way to another one. In this paper, we cope with this problem by proposing ¿-TMP, a smart combination of an heuristic task planner based on the Fast Forward method, a physics-based motion planner, and reasoning processes over the ontologies that code the knowledge on the problem. The significance of the proposal relies on how geometric and physics information is used within the computation of the heuristics in order to guide the symbolic search, i.e. how an artificial intelligence planning method is combined with low-level motion planning to achieve a feasible sequence of actions (composed of collision-free motions plus physically-feasible push/pull actions). The proposal has been validated with several simulated scenarios (using up to five robots that need to collaborate with each other to reach the goal state), showing how the method is able to solve challenging situations and also find an efficient solution in terms of power.Peer Reviewe