Agent and object aware tracking and mapping methods for mobile manipulators

The age of the intelligent machine is upon us. They exist in our factories, our warehouses, our military, our hospitals, on our roads, and on the moon. Most of these things we call robots. When placed in a controlled or known environment such as an automotive factory or a distribution warehouse they perform their given roles with exceptional efficiency, achieving far more than is within reach of a humble human being. Despite the remarkable success of intelligent machines in such domains, they have yet to make a full-hearted deployment into our homes. The missing link between the robots we have now and the robots that are soon to come to our houses is perception. Perception as we mean it here refers to a level of understanding beyond the collection and aggregation of sensory data. Much of the available sensory information is noisy and unreliable, our homes contain many reflective surfaces, repeating textures on large flat surfaces, and many disruptive moving elements, including humans. These environments change over time, with objects frequently moving within and between rooms. This idea of change in an environment is fundamental to robotic applications, as in most cases we expect them to be effectors of such change. We can identify two particular challenges1 that must be solved for robots to make the jump to less structured environments - how to manage noise and disruptive elements in observational data, and how to understand the world as a set of changeable elements (objects) which move over time within a wider environment. In this thesis we look at one possible approach to solving each of these problems. For the first challenge we use proprioception aboard a robot with an articulated arm to handle difficult and unreliable visual data caused both by the robot and the environment. We use sensor data aboard the robot to improve the pose tracking of a visual system when the robot moves rapidly, with high jerk, or when observing a scene with little visual variation. For the second challenge, we build a model of the world on the level of rigid objects, and relocalise them both as they change location between different sequences and as they move. We use semantics, image keypoints, and 3D geometry to register and align objects between sequences, showing how their position has moved between disparate observations.Open Acces

Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age

Simultaneous Localization and Mapping (SLAM)consists in the concurrent construction of a model of the environment (the map), and the estimation of the state of the robot moving within it. The SLAM community has made astonishing progress over the last 30 years, enabling large-scale real-world applications, and witnessing a steady transition of this technology to industry. We survey the current state of SLAM. We start by presenting what is now the de-facto standard formulation for SLAM. We then review related work, covering a broad set of topics including robustness and scalability in long-term mapping, metric and semantic representations for mapping, theoretical performance guarantees, active SLAM and exploration, and other new frontiers. This paper simultaneously serves as a position paper and tutorial to those who are users of SLAM. By looking at the published research with a critical eye, we delineate open challenges and new research issues, that still deserve careful scientific investigation. The paper also contains the authors' take on two questions that often animate discussions during robotics conferences: Do robots need SLAM? and Is SLAM solved

Distributed Robotic Vision for Calibration, Localisation, and Mapping

This dissertation explores distributed algorithms for calibration, localisation, and mapping in the context of a multi-robot network equipped with cameras and onboard processing, comparing against centralised alternatives where all data is transmitted to a singular external node on which processing occurs. With the rise of large-scale camera networks, and as low-cost on-board processing becomes increasingly feasible in robotics networks, distributed algorithms are becoming important for robustness and scalability. Standard solutions to multi-camera computer vision require the data from all nodes to be processed at a central node which represents a significant single point of failure and incurs infeasible communication costs. Distributed solutions solve these issues by spreading the work over the entire network, operating only on local calculations and direct communication with nearby neighbours. This research considers a framework for a distributed robotic vision platform for calibration, localisation, mapping tasks where three main stages are identified: an initialisation stage where calibration and localisation are performed in a distributed manner, a local tracking stage where visual odometry is performed without inter-robot communication, and a global mapping stage where global alignment and optimisation strategies are applied. In consideration of this framework, this research investigates how algorithms can be developed to produce fundamentally distributed solutions, designed to minimise computational complexity whilst maintaining excellent performance, and designed to operate effectively in the long term. Therefore, three primary objectives are sought aligning with these three stages

Cartographie dense basée sur une représentation compacte RGB-D dédiée à la navigation autonome

Our aim is concentrated around building ego-centric topometric maps represented as a graph of keyframe nodes which can be efficiently used by autonomous agents. The keyframe nodes which combines a spherical image and a depth map (augmented visual sphere) synthesises information collected in a local area of space by an embedded acquisition system. The representation of the global environment consists of a collection of augmented visual spheres that provide the necessary coverage of an operational area. A "pose" graph that links these spheres together in six degrees of freedom, also defines the domain potentially exploitable for navigation tasks in real time. As part of this research, an approach to map-based representation has been proposed by considering the following issues : how to robustly apply visual odometry by making the most of both photometric and ; geometric information available from our augmented spherical database ; how to determine the quantity and optimal placement of these augmented spheres to cover an environment completely ; how tomodel sensor uncertainties and update the dense infomation of the augmented spheres ; how to compactly represent the information contained in the augmented sphere to ensure robustness, accuracy and stability along an explored trajectory by making use of saliency maps.Dans ce travail, nous proposons une représentation efficace de l’environnement adaptée à la problématique de la navigation autonome. Cette représentation topométrique est constituée d’un graphe de sphères de vision augmentées d’informations de profondeur. Localement la sphère de vision augmentée constitue une représentation égocentrée complète de l’environnement proche. Le graphe de sphères permet de couvrir un environnement de grande taille et d’en assurer la représentation. Les "poses" à 6 degrés de liberté calculées entre sphères sont facilement exploitables par des tâches de navigation en temps réel. Dans cette thèse, les problématiques suivantes ont été considérées : Comment intégrer des informations géométriques et photométriques dans une approche d’odométrie visuelle robuste ; comment déterminer le nombre et le placement des sphères augmentées pour représenter un environnement de façon complète ; comment modéliser les incertitudes pour fusionner les observations dans le but d’augmenter la précision de la représentation ; comment utiliser des cartes de saillances pour augmenter la précision et la stabilité du processus d’odométrie visuelle

Analysis of Visual-Inertial Odometry Algorithms for Outdoor Drone Applications

Visual-inertial odometry (VIO) and visual-inertial simultaneous localisation and mapping (VISLAM) enables mobile robots to localise without relying on global navigation satellite systems (GNSS) or heavy sensors. They enable mobile robots, especially payload critical robots, such as drones, to perform autonomous tasks with limited resources. Localisation of drones for outdoor applications using visual and inertial sensor fusion is of particular interest, since it widens the use cases and reliability of autonomous drones in different flying conditions and environments. The goal of this thesis is to identify suitable VIO/VISLAM algorithms, and to develop a platform for localising a drone for outdoor applications. A stereo camera and IMU sensor suite was developed to collect visual-inertial data, since suitable off-the-shelf systems were not available. Three state-of-the-art VIO/VISLAM algorithms, FLVIS, ORB-SLAM3 and VINS-Fusion, were evaluated with outdoor drone datasets of varying flight altitudes of 40, 60, 80 and 100 m and speeds of 2, 3 and 4 m/s. The estimation results were compared with the ground truth and were quantitatively evaluated. VINS-Fusion estimated the trajectories most accurately among the three algorithms with an absolute trajectory error of 2.186 m and a relative rotation error of 0.862 deg at an altitude of 60 m for a trajectory of length 800 m. System configurations, algorithm parameters, external conditions, and scene content impacted the estimation results. These factors, further developments and future scopes are discussed along with the obtained results

Monocular 3D Scene Reconstruction for an Autonomous Unmanned Aerial Vehicle

Rekonstrukce 3D modelu prostředí je klíčovou částí autonomního letu bezpilotní helikoptéry (UAV). Kombinace inerciální měřicí jednotky (IMU) a kamery je běžnou a dostupnou senzorovou sadou, jež je schopna získat informaci o měřítku prostředí. Tato práce si klade za cíl vyvinout algoritmus řešící problém 3D rekostrukce pro tyto senzory za využití existujících metod vizuálně-inerciální lokalizace (VINS). V práci jsou navrženy dva algoritmy, odlišené způsobem, jakým extrahují korespondence mezi snímky: párovací algoritmus se širokou bází a algoritmus založený na trackingu s malou bází. Také je implementována metoda vylepšující výslednou 3D strukturu po letu. Algoritmy jsou otestovány na veřejně dostupné datové sadě. Navíc jsou otestovány v simulátoru a je proveden experiment v reálném prostředí. Výsledky ukazují, že algoritmus založený na trackingu dosahuje výrazně lepších výsledků. Navíc testy na datech a experimenty v reálném prostředí ukazují, že algoritmus může být nasazen v praktických aplikačních situacích.The real-time 3D reconstruction of the surrounding scene is a key part in the pipeline of the autonomous flight of unmanned aerial vehicle (UAV). The combination of an inertial measurement unit (IMU) and a monocular camera is a common and inexpensive sensor setup that can be used to recover the scale of the environment. This thesis aims to develop an algorithm solving this problem for this particular setup by leveraging the existing visual-inertial navigation system (VINS) odometry algorithms for localisation. Two algorithms are developed, wide-baseline matching-based and small-baseline tracking-based. Also, an offline post-processing structure-refinement step is implemented to further improve the resulting structure. The algorithms and the refinement step are then evaluated on publicly available datasets. Furthermore, they are tested in a simulator, and a real-world experiment is conducted. The results show that the tracking-based algorithm is significantly more performant. Importantly, tests on the datasets and the real-world experiments suggest that this algorithm can be practically employed in application scenarios

Robust dense visual SLAM using sensor fusion and motion segmentation

Visual simultaneous localisation and mapping (SLAM) is an important technique for enabling mobile robots to navigate autonomously within their environments. Using cameras, robots reconstruct a representation of their environment and simultaneously localise themselves within it. A dense visual SLAM system produces a high-resolution and detailed reconstruction of the environment which can be used for obstacle avoidance or semantic reasoning. State-of-the-art dense visual SLAM systems demonstrate robust performance and impressive accuracy in ideal conditions. However, these techniques are based on requirements which limit the extent to which they can be deployed in real applications. Fundamentally, they require constant scene illumination, smooth camera motion and no moving objects being present in the scene. Overcoming these requirements is not trivial and significant effort is needed to make dense visual SLAM approaches more robust to real-world conditions. The objective of this thesis is to develop dense visual SLAM systems which are more robust to real-world visually challenging conditions. For this, we leverage sensor fusion and motion segmentation for situations where camera data is unsuitable. The first contribution is a visual SLAM system for the NASA Valkyrie humanoid robot which is robust to the robot’s operation. It is based on a sensor fusion approach which combines visual SLAM and leg odometry to demonstrate increased robustness to illumination changes and fast camera motion. Second, we research methods for robust visual odometry in the presence of moving objects. We propose a formulation for joint visual odometry and motion segmentation that demonstrates increased robustness in scenes with moving objects compared to state-of-the-art approaches. We then extend this method using inertial information from a gyroscope to compare the contributions of motion segmentation and motion prior integration for robustness to scene dynamics. As part of this study we provide a dataset recorded in scenes with different numbers of moving objects. In conclusion, we find that both motion segmentation and motion prior integration are necessary for achieving significantly better results in real-world conditions. While motion priors increase robustness, motion segmentation increases the accuracy of the reconstruction results through filtering of moving objects.Edinburgh Centre for RoboticsEngineering and Physical Sciences Research Council (EPSRC