Automation in handling uncertainty in semantic-knowledge based robotic task-planning by using Markov Logic Networks

Generating plans in real world environments by mobile robot planner is a challenging task due to the uncertainty and environment dynamics. Therefore, task-planning should take in its consideration these issues when generating plans. Semantic knowledge domain has been proposed as a source of information for deriving implicit information and generating semantic plans. This paper extends the Semantic-Knowledge Based (SKB) plan generation to take into account the uncertainty in existing of objects, with their types and properties, and proposes a new approach to construct plans based on probabilistic values which are derived from Markov Logic Networks (MLN). An MLN module is established for probabilistic learning and inferencing together with semantic information to provide a basis for plausible learning and reasoning services in supporting of robot task-planning. In addition, an algorithm has been devised to construct MLN from semantic knowledge. By providing a means of modeling uncertainty in system architecture, task-planning serves as a supporting tool for robotic applications that can benefit from probabilistic inference within a semantic domain. This approach is illustrated using test scenarios run in a domestic environment using a mobile robot

Probabilistic techniques in semantic mapping for mobile robotics

Los mapas semánticos son representaciones del mundo que permiten a un robot entender no sólo los aspectos espaciales de su lugar de trabajo, sino también el significado de sus elementos (objetos, habitaciones, etc.) y como los humanos interactúan con ellos (e.g. funcionalidades, eventos y relaciones). Para conseguirlo, un mapa semántico añade a las representaciones puramente espaciales, tales como mapas geométricos o topológicos, meta-información sobre los tipos de elementos y relaciones que pueden encontrarse en el entorno de trabajo. Esta meta-información, denominada conocimiento semántico o de sentido común, se codifica típicamente en Bases de Conocimiento. Un ejemplo de este tipo de información podría ser: "los frigoríficos son objetos grandes, con forma rectangular, colocados normalmente en las cocinas, y que pueden contener comida perecedera y medicación". Codificar y manejar este conocimiento semántico permite al robot razonar acerca de la información obtenida de un cierto lugar de trabajo, así como inferir nueva información con el fin de ejecutar eficientemente tareas de alto nivel como "¡hola robot! llévale la medicación a la abuela, por favor". La presente tesis propone la utilización de técnicas probabilísticas para construir y mantener mapas semánticos, lo cual presenta tres ventajas principales en comparación con los enfoques tradicionales: i) permite manejar incertidumbre (proveniente de los sensores imprecisos del robot y de los modelos empleados), ii) provee representaciones del entorno coherentes por medio del aprovechamiento de las relaciones contextuales entre los elementos observados (e.g. los frigoríficos usualmente se encuentran en las cocinas) desde un punto de vista holístico, y iii) produce valores de certidumbre que reflejan el grado de exactitud de la comprensión del robot acerca de su entorno. Específicamente, las contribuciones presentadas pueden agruparse en dos temas principales. El primer conjunto de contribuciones se basa en el problema del reconocimiento de objetos y/o habitaciones, ya que los sistemas de mapeo semántico deben contar con algoritmos de reconocimiento fiables para la construcción de representaciones válidas. Para ello se ha explorado la utilización de Modelos Gráficos Probabilísticos (Probabilistic Graphical Models o PGMs en inglés) con el fin de aprovechar las relaciones de contexto entre objetos y/o habitaciones a la vez que se maneja la incertidumbre inherente al problema de reconocimiento, y el empleo de Bases de Conocimiento para mejorar su desempeño de distintos modos, e.g., detectando resultados incoherentes, proveyendo información a priori, reduciendo la complejidad de los algoritmos de inferencia probabilística, generando ejemplos de entrenamiento sintéticos, habilitando el aprendizaje a partir de experiencias pasadas, etc. El segundo grupo de contribuciones acomoda los resultados probabilísticos provenientes de los algoritmos de reconocimiento desarrollados en una nueva representación semántica, denominada Multiversal Semantic Map (MvSmap). Este mapa gestiona múltiples interpretaciones del espacio de trabajo del robot, llamadas universos, los cuales son anotados con la probabilidad de ser los correctos de acuerdo con el conocimiento actual del robot. Así, este enfoque proporciona una creencia fundamentada sobre la exactitud de la comprensión del robot sobre su entorno, lo que le permite operar de una manera más eficiente y coherente. Los algoritmos probabilísticos propuestos han sido testeados concienzudamente y comparados con otros enfoques actuales e innovadores empleando conjuntos de datos del estado del arte. De manera adicional, esta tesis también contribuye con dos conjuntos de datos, UMA-Offices and Robot@Home, los cuales contienen información sensorial capturada en distintos entornos de oficinas y casas, así como dos herramientas software, la librería Undirected Probabilistic Graphical Models in C++ (UPGMpp), y el conjunto de herramientas Object Labeling Toolkit (OLT), para el trabajo con Modelos Gráficos Probabilísticos y el procesamiento de conjuntos de datos respectivamente

Interpretable task planning and learning for autonomous robotic surgery with logic programming

This thesis addresses the long-term goal of full (supervised) autonomy in surgery, characterized by dynamic environmental (anatomical) conditions, unpredictable workflow of execution and workspace constraints. The scope is to reach autonomy at the level of sub-tasks of a surgical procedure, i.e. repetitive, yet tedious operations (e.g., dexterous manipulation of small objects in a constrained environment, as needle and wire for suturing). This will help reducing time of execution, hospital costs and fatigue of surgeons during the whole procedure, while further improving the recovery time for the patients. A novel framework for autonomous surgical task execution is presented in the first part of this thesis, based on answer set programming (ASP), a logic programming paradigm, for task planning (i.e., coordination of elementary actions and motions). Logic programming allows to directly encode surgical task knowledge, representing emph{plan reasoning methodology} rather than a set of pre-defined plans. This solution introduces several key advantages, as reliable human-like interpretable plan generation, real-time monitoring of the environment and the workflow for ready adaptation and failure recovery. Moreover, an extended review of logic programming for robotics is presented, motivating the choice of ASP for surgery and providing an useful guide for robotic designers. In the second part of the thesis, a novel framework based on inductive logic programming (ILP) is presented for surgical task knowledge learning and refinement. ILP guarantees fast learning from very few examples, a common drawback of surgery. Also, a novel action identification algorithm is proposed based on automatic environmental feature extraction from videos, dealing for the first time with small and noisy datasets collecting different workflows of executions under environmental variations. This allows to define a systematic methodology for unsupervised ILP. All the results in this thesis are validated on a non-standard version of the benchmark training ring transfer task for surgeons, which mimics some of the challenges of real surgery, e.g. constrained bimanual motion in small space

Semantic based task planning for domestic service robots

Task Planning is developed for an autonomous mobile robot in order to support the robot to accomplish tasks in various degrees of environmental complexity. This environment can be fixed or deterministic (as in a factory), dynamic (as in the human domestic household), or non-deterministic (as in the space exploration). The robot should be provided with a reliable planning system in order to face its major challenge of being certain that its plan to accomplish a task is generated correctly, regardless of the dynamic or uncertain elements of its environment. This thesis is focused on providing the robot task planner with the ability to generate its plans reliably and detect the failures in generating correct plans. Previous approaches for generating plans depended mainly on action effects (explicit effects) that are encoded in the action model. This means that the action effects should cover most of the characteristics of the newly generated world state. However, this extra information can complicate the action model, especially in the real world. In this thesis, a semantic knowledge base is proposed to derive and check implicit information about the effects of actions during plan generation. For example, this approach would inform the robot, that it had entered a bedroom because it has recorded at least one bed and zero ovens. When a robot enters a room, the implicit expectations are derived from a semantic knowledge base about that type of room. These expectations should be verified in order to make sure the robot is in the correct room. The main contributions of this thesis are as follows: The concept of using the Semantic Knowledge Base (SKB) to support the robot task planner under deterministic conditions has been defined. A new model of high-level robot actions has been developed, and this model represents the details of robot action as ontology. This model is thus known as the Semantic Action Model (SAM). An algorithm that integrates SKB and SAMs has also been developed. This algorithm creates the “planning domain” in the Planning Domain Definition Language (PDDL) style. This is used as input to the planner to generate the plan for robot tasks. Then, a general purpose planning algorithm has also been defined, which can support planning under deterministic conditions, and is based on using ontology to represent SKB. ii A further contribution relates to the development of a probabilistic approach to deal with uncertainty in semantic knowledge based task planning. This approach shows how uncertainties in action effects and world states are taken into account by the planning system. This contribution also served to resolve situations of confusion in finding an object relevant to the successful generation of an action during task planning. The accuracy related to this type of planning in navigation scenario, on average, is (90.10%). An additional contribution is using the planning system to respond to unexpected situations which are caused by lack of information. This contribution is formalised as a general approach that models cases of incomplete information as a planning problem. This approach includes a sequence of steps for modelling and generating a plan of actions to collect the necessary information from the knowledge base to support the robot planner in generating its plan. This results in developing a new type of action which is known as a Semantic Action Model for Information Gathering (SAM_IG). These actions have the ability to access the knowledge base to retrieve the necessary information to support the planning system when it is faced with incomplete information. The information gathering approach is also used to gather the necessary information in order to check the implicit expectations of the generated actions. The correct classification related to this type of planning in navigation scenario, on average, is (92.83 %). Another contribution is concerned with solving the problem of missing information, which is using the methods for measuring concept similarity in order to extend the robot world state with new similar objects to the original one in the action model. This results in developing Semantic Realisation and Refreshment Module (SRRM) which has the ability to estimate the similarities between objects and the quality of the alternative plans. The quality of the alternative plans could be similar to the original plan, in average, 92.1%. The results reported in this thesis have been tested and verified in simulation experiments under the Robot Operating System (ROS) middleware. The performance of the planning system has been evaluated by using the planning time and other known metrics. These results show that using semantic knowledge can lead to high performance and reliability in generating robot plans during its operatio

Grounded Semantic Reasoning for Robotic Interaction with Real-World Objects

Robots are increasingly transitioning from specialized, single-task machines to general-purpose systems that operate in unstructured environments, such as homes, offices, and warehouses. In these real-world domains, robots need to manipulate novel objects while adapting to changes in environments and goals. Semantic knowledge, which concisely describes target domains with symbols, can potentially reveal the meaningful patterns shared between problems and environments. However, existing robots are yet to effectively reason about semantic data encoding complex relational knowledge or jointly reason about symbolic semantic data and multimodal data pertinent to robotic manipulation (e.g., object point clouds, 6-DoF poses, and attributes detected with multimodal sensing). This dissertation develops semantic reasoning frameworks capable of modeling complex semantic knowledge grounded in robot perception and action. We show that grounded semantic reasoning enables robots to more effectively perceive, model, and interact with objects in real-world environments. Specifically, this dissertation makes the following contributions: (1) a survey providing a unified view for the diversity of works in the field by formulating semantic reasoning as the integration of knowledge sources, computational frameworks, and world representations; (2) a method for predicting missing relations in large-scale knowledge graphs by leveraging type hierarchies of entities, effectively avoiding ambiguity while maintaining generalization of multi-hop reasoning patterns; (3) a method for predicting unknown properties of objects in various environmental contexts, outperforming prior knowledge graph and statistical relational learning methods due to the use of n-ary relations for modeling object properties; (4) a method for purposeful robotic grasping that accounts for a broad range of contexts (including object visual affordance, material, state, and task constraint), outperforming existing approaches in novel contexts and for unknown objects; (5) a systematic investigation into the generalization of task-oriented grasping that includes a benchmark dataset of 250k grasps, and a novel graph neural network that incorporates semantic relations into end-to-end learning of 6-DoF grasps; (6) a method for rearranging novel objects into semantically meaningful spatial structures based on high-level language instructions, more effectively capturing multi-object spatial constraints than existing pairwise spatial representations; (7) a novel planning-inspired approach that iteratively optimizes placements of partially observed objects subject to both physical constraints and semantic constraints inferred from language instructions.Ph.D