Differentiable world programs

L'intelligence artificielle (IA) moderne a ouvert de nouvelles perspectives prometteuses pour la création de robots intelligents. En particulier, les architectures d'apprentissage basées sur le gradient (réseaux neuronaux profonds) ont considérablement amélioré la compréhension des scènes 3D en termes de perception, de raisonnement et d'action. Cependant, ces progrès ont affaibli l'attrait de nombreuses techniques ``classiques'' développées au cours des dernières décennies. Nous postulons qu'un mélange de méthodes ``classiques'' et ``apprises'' est la voie la plus prometteuse pour développer des modèles du monde flexibles, interprétables et exploitables : une nécessité pour les agents intelligents incorporés. La question centrale de cette thèse est : ``Quelle est la manière idéale de combiner les techniques classiques avec des architectures d'apprentissage basées sur le gradient pour une compréhension riche du monde 3D ?''. Cette vision ouvre la voie à une multitude d'applications qui ont un impact fondamental sur la façon dont les agents physiques perçoivent et interagissent avec leur environnement. Cette thèse, appelée ``programmes différentiables pour modèler l'environnement'', unifie les efforts de plusieurs domaines étroitement liés mais actuellement disjoints, notamment la robotique, la vision par ordinateur, l'infographie et l'IA. Ma première contribution---gradSLAM--- est un système de localisation et de cartographie simultanées (SLAM) dense et entièrement différentiable. En permettant le calcul du gradient à travers des composants autrement non différentiables tels que l'optimisation non linéaire par moindres carrés, le raycasting, l'odométrie visuelle et la cartographie dense, gradSLAM ouvre de nouvelles voies pour intégrer la reconstruction 3D classique et l'apprentissage profond. Ma deuxième contribution - taskography - propose une sparsification conditionnée par la tâche de grandes scènes 3D encodées sous forme de graphes de scènes 3D. Cela permet aux planificateurs classiques d'égaler (et de surpasser) les planificateurs de pointe basés sur l'apprentissage en concentrant le calcul sur les attributs de la scène pertinents pour la tâche. Ma troisième et dernière contribution---gradSim--- est un simulateur entièrement différentiable qui combine des moteurs physiques et graphiques différentiables pour permettre l'estimation des paramètres physiques et le contrôle visuomoteur, uniquement à partir de vidéos ou d'une image fixe.Modern artificial intelligence (AI) has created exciting new opportunities for building intelligent robots. In particular, gradient-based learning architectures (deep neural networks) have tremendously improved 3D scene understanding in terms of perception, reasoning, and action. However, these advancements have undermined many ``classical'' techniques developed over the last few decades. We postulate that a blend of ``classical'' and ``learned'' methods is the most promising path to developing flexible, interpretable, and actionable models of the world: a necessity for intelligent embodied agents. ``What is the ideal way to combine classical techniques with gradient-based learning architectures for a rich understanding of the 3D world?'' is the central question in this dissertation. This understanding enables a multitude of applications that fundamentally impact how embodied agents perceive and interact with their environment. This dissertation, dubbed ``differentiable world programs'', unifies efforts from multiple closely-related but currently-disjoint fields including robotics, computer vision, computer graphics, and AI. Our first contribution---gradSLAM---is a fully differentiable dense simultaneous localization and mapping (SLAM) system. By enabling gradient computation through otherwise non-differentiable components such as nonlinear least squares optimization, ray casting, visual odometry, and dense mapping, gradSLAM opens up new avenues for integrating classical 3D reconstruction and deep learning. Our second contribution---taskography---proposes a task-conditioned sparsification of large 3D scenes encoded as 3D scene graphs. This enables classical planners to match (and surpass) state-of-the-art learning-based planners by focusing computation on task-relevant scene attributes. Our third and final contribution---gradSim---is a fully differentiable simulator that composes differentiable physics and graphics engines to enable physical parameter estimation and visuomotor control, solely from videos or a still image

Localization, Navigation and Activity Planning for Wheeled Agricultural Robots – A Survey

Source at:https://fruct.org/publications/volume-32/fruct32/High cost, time intensive work, labor shortages and inefficient strategies have raised the need of employing mobile robotics to fully automate agricultural tasks and fulfil the requirements of precision agriculture. In order to perform an agricultural task, the mobile robot goes through a sequence of sub operations and integration of hardware and software systems. Starting with localization, an agricultural robot uses sensor systems to estimate its current position and orientation in field, employs algorithms to find optimal paths and reach target positions. It then uses techniques and models to perform feature recognition and finally executes the agricultural task through an end effector. This article, compiled through scrutinizing the current literature, is a step-by-step approach of the strategies and ways these sub-operations are performed and integrated together. An analysis has also been done on the limitations in each sub operation, available solutions, and the ongoing research focus

Towards adaptive and autonomous humanoid robots: from vision to actions

Although robotics research has seen advances over the last decades robots are still not in widespread use outside industrial applications. Yet a range of proposed scenarios have robots working together, helping and coexisting with humans in daily life. In all these a clear need to deal with a more unstructured, changing environment arises. I herein present a system that aims to overcome the limitations of highly complex robotic systems, in terms of autonomy and adaptation. The main focus of research is to investigate the use of visual feedback for improving reaching and grasping capabilities of complex robots. To facilitate this a combined integration of computer vision and machine learning techniques is employed. From a robot vision point of view the combination of domain knowledge from both imaging processing and machine learning techniques, can expand the capabilities of robots. I present a novel framework called Cartesian Genetic Programming for Image Processing (CGP-IP). CGP-IP can be trained to detect objects in the incoming camera streams and successfully demonstrated on many different problem domains. The approach requires only a few training images (it was tested with 5 to 10 images per experiment) is fast, scalable and robust yet requires very small training sets. Additionally, it can generate human readable programs that can be further customized and tuned. While CGP-IP is a supervised-learning technique, I show an integration on the iCub, that allows for the autonomous learning of object detection and identification. Finally this dissertation includes two proof-of-concepts that integrate the motion and action sides. First, reactive reaching and grasping is shown. It allows the robot to avoid obstacles detected in the visual stream, while reaching for the intended target object. Furthermore the integration enables us to use the robot in non-static environments, i.e. the reaching is adapted on-the- fly from the visual feedback received, e.g. when an obstacle is moved into the trajectory. The second integration highlights the capabilities of these frameworks, by improving the visual detection by performing object manipulation actions

Becoming Human with Humanoid

Nowadays, our expectations of robots have been significantly increases. The robot, which was initially only doing simple jobs, is now expected to be smarter and more dynamic. People want a robot that resembles a human (humanoid) has and has emotional intelligence that can perform action-reaction interactions. This book consists of two sections. The first section focuses on emotional intelligence, while the second section discusses the control of robotics. The contents of the book reveal the outcomes of research conducted by scholars in robotics fields to accommodate needs of society and industry

Control visual de un robot móvil mediante una cámara cenital

This research project addresses the problem of controlling the motion of a small mobile robot by means of visual feedback provided by an overhead camera. This visual servoing problem has been previously addressed by many researchers due to its multiple applications to real world problems. In this document, we propose a software platform that rely on low cost hardware components to solve it. Based on the imagery supplied by the overhead camera, the proposed system is capable of precisely locating and tracking the robot within a planar ground workspace, using the CAMShift algorithm, as well as finding out its orientation at every moment. Then, an error measurement is defined between current and desired positions of the robot in the Cartesian plane (Position-Based Visual Servoing). In order to generate the suitable motion commands that lead the robot towards its destination, we make use of mathematical equations that model the control of the robot. The platform has been especially designed regarding its application to real time problems. One of the central goals of this work is analyzing the viability of the proposed system and the level of accuracy that it is capable of achieving taking into account the low cost components on which it is based. The validation of the system has come as a result of the real time experiments that have been conducted. Firstly, an exhaustive battery testing that comprehends 1400 experiments has been conducted in order to find a suitable set of parameter values that polished the control equations. Secondly, we have implemented three different applications to test these new control values: tracing a trajectory defined by a fixed set of points, pursuing a mobile target and integrating our system with a blockprogramming platform from which it receives a set of destination points to be followed. Having successfully completed all these tasks, we conclude that the proposed robotic system has well proven its feasibility and effectiveness facing the addressed visual servoing problem.Este proyecto de investigación aborda el problema de controlar el movimiento de un pequeño robot móvil por medio del feedback visual proporcionado por una cámara cenital. Este problema de control visual de servos ya ha sido abordado previamente por multitud de investigadores debido a sus múltiples aplicaciones a problemas del mundo real. En este documento, se propone una plataforma software que depende de componentes hardware de bajo coste para resolverlo. Basado en imágenes suministradas por la cámara cenital, el sistema propuesto es capaz de localizar y seguir de forma precisa al robot dentro de un entorno de trabajo en el plano del suelo, usando para ello el algoritmo de tracking CAMShift, así como averiguar su orientación en cada momento. Después, una medida de error se define entre la posición actual del robot y la deseada en el plano Cartesiano (control visual de servos basado en posición (PBVS)). Para generar los comandos de movimiento aporpiados que lleven al robot a su destino, hacemos uso de ecuaciones matemáticas que modelizan el control del robot. La plataforma ha sido especialmente diseñada teniendo en cuenta su aplicación a problemas en tiempo real. Uno de los objetivos centrales de este trabajo es analizar la viabilidad del sistema propuesto y el nivel de precisión que es capaz de obtener teniendo en cuenta los componentes de bajo coste en los que se basa. La validación del sistema viene dada como resultado de los experimentos en tiempo real que se han llevado a cabo. Primeramente, una exhaustiva batería de pruebas que comprende 1400 experimentos ha sido ejecutada con el fin de obtener un set de valores para los parámetros que puliesen las ecuaciones de control. A continuación, hemos implementado tres aplicaciones diferentes para probar estos nuevos valores de control: trazar una trayectoria definida por un conjunto de puntos fijos, perseguir un objetivo móvil e integrar nuestro sistema con la plataforma de programación por bloques desde la que recibe el conjunto de puntos a seguir. Habiendo completado todas estas tareas satisfactoriamente, concluimos que el sistema robótico propuesto ha demostrado con holgura su viabilidad y efectividad frente al problema de control visual de servos abordado

Autonomous Navigation for Unmanned Aerial Systems - Visual Perception and Motion Planning

L'abstract è presente nell'allegato / the abstract is in the attachmen

An Application-centric Perspective on Industrial Artificial Intelligence

Advances in Artificial Intelligence have made its application increasingly relevant to all types of Information Systems. One area where researchers and practitioners see massive potential is the interface between Artificial Intelligence-empowered Information Systems and industrial processes. This explicit area of Industrial Artificial Intelligence and Industry 4.0 has been a popular topic of recent work and has opened up new research streams and applications. However, given the increasing number of publications, it is difficult to discern where the research field is heading. In our work, we conduct a systematic literature review of 296 scientific articles to provide a comprehensive overview of the current state of Industrial Artificial Intelligence in terms of research streams and application areas. We present both a metadata analysis as well as an application-specific analysis. Our results reveal insights into 14 major application areas as well as several findings on applied algorithms and approaches of Industrial Artificial Intelligence

Towards an infrastructure for preparation and control of intelligent automation systems

In an attempt to handle some of the challenges of modern production, intelligent automation systems offer solutions that are flexible, adaptive, and collaborative. Contrary to traditional solutions, intelligent automation systems emerged just recently and thus lack the supporting tools and infrastructure that traditional systems nowadays take for granted. To support efficient development, commissioning, and control of such systems, this thesis summarizes various lessons learned during years of implementation. Based on what was learned, this thesis investigates key features of infrastructure for modern and flexible intelligent automation systems, as well as a number of important design solutions. For example, an important question is raised whether to decentralize the global state or to give complete access to the main controller.Moreover, in order to develop such systems, a framework for virtual preparation and commissioning is presented, with the main goal to offer support for engineers. As traditional virtual commissioning solutions are not intended for preparing highly flexible, collaborative, and dynamic systems, this framework aims to provide some of the groundwork and point to a direction for fast and integrated preparation and virtual commissioning of such systems.Finally, this thesis summarizes some of the investigations made on planning as satisfiability, in order to evaluate how different methods improve planning performance. Throughout the thesis, an industrial material kitting use case exemplifies presented perspectives, lessons learned, and frameworks