Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age

Simultaneous Localization and Mapping (SLAM)consists in the concurrent construction of a model of the environment (the map), and the estimation of the state of the robot moving within it. The SLAM community has made astonishing progress over the last 30 years, enabling large-scale real-world applications, and witnessing a steady transition of this technology to industry. We survey the current state of SLAM. We start by presenting what is now the de-facto standard formulation for SLAM. We then review related work, covering a broad set of topics including robustness and scalability in long-term mapping, metric and semantic representations for mapping, theoretical performance guarantees, active SLAM and exploration, and other new frontiers. This paper simultaneously serves as a position paper and tutorial to those who are users of SLAM. By looking at the published research with a critical eye, we delineate open challenges and new research issues, that still deserve careful scientific investigation. The paper also contains the authors' take on two questions that often animate discussions during robotics conferences: Do robots need SLAM? and Is SLAM solved

Robust convex optimisation techniques for autonomous vehicle vision-based navigation

This thesis investigates new convex optimisation techniques for motion and pose estimation. Numerous computer vision problems can be formulated as optimisation problems. These optimisation problems are generally solved via linear techniques using the singular value decomposition or iterative methods under an L2 norm minimisation. Linear techniques have the advantage of offering a closed-form solution that is simple to implement. The quantity being minimised is, however, not geometrically or statistically meaningful. Conversely, L2 algorithms rely on iterative estimation, where a cost function is minimised using algorithms such as Levenberg-Marquardt, Gauss-Newton, gradient descent or conjugate gradient. The cost functions involved are geometrically interpretable and can statistically be optimal under an assumption of Gaussian noise. However, in addition to their sensitivity to initial conditions, these algorithms are often slow and bear a high probability of getting trapped in a local minimum or producing infeasible solutions, even for small noise levels. In light of the above, in this thesis we focus on developing new techniques for finding solutions via a convex optimisation framework that are globally optimal. Presently convex optimisation techniques in motion estimation have revealed enormous advantages. Indeed, convex optimisation ensures getting a global minimum, and the cost function is geometrically meaningful. Moreover, robust optimisation is a recent approach for optimisation under uncertain data. In recent years the need to cope with uncertain data has become especially acute, particularly where real-world applications are concerned. In such circumstances, robust optimisation aims to recover an optimal solution whose feasibility must be guaranteed for any realisation of the uncertain data. Although many researchers avoid uncertainty due to the added complexity in constructing a robust optimisation model and to lack of knowledge as to the nature of these uncertainties, and especially their propagation, in this thesis robust convex optimisation, while estimating the uncertainties at every step is investigated for the motion estimation problem. First, a solution using convex optimisation coupled to the recursive least squares (RLS) algorithm and the robust H filter is developed for motion estimation. In another solution, uncertainties and their propagation are incorporated in a robust L convex optimisation framework for monocular visual motion estimation. In this solution, robust least squares is combined with a second order cone program (SOCP). A technique to improve the accuracy and the robustness of the fundamental matrix is also investigated in this thesis. This technique uses the covariance intersection approach to fuse feature location uncertainties, which leads to more consistent motion estimates. Loop-closure detection is crucial in improving the robustness of navigation algorithms. In practice, after long navigation in an unknown environment, detecting that a vehicle is in a location it has previously visited gives the opportunity to increase the accuracy and consistency of the estimate. In this context, we have developed an efficient appearance-based method for visual loop-closure detection based on the combination of a Gaussian mixture model with the KD-tree data structure. Deploying this technique for loop-closure detection, a robust L convex posegraph optimisation solution for unmanned aerial vehicle (UAVs) monocular motion estimation is introduced as well. In the literature, most proposed solutions formulate the pose-graph optimisation as a least-squares problem by minimising a cost function using iterative methods. In this work, robust convex optimisation under the L norm is adopted, which efficiently corrects the UAV’s pose after loop-closure detection. To round out the work in this thesis, a system for cooperative monocular visual motion estimation with multiple aerial vehicles is proposed. The cooperative motion estimation employs state-of-the-art approaches for optimisation, individual motion estimation and registration. Three-view geometry algorithms in a convex optimisation framework are deployed on board the monocular vision system for each vehicle. In addition, vehicle-to-vehicle relative pose estimation is performed with a novel robust registration solution in a global optimisation framework. In parallel, and as a complementary solution for the relative pose, a robust non-linear H solution is designed as well to fuse measurements from the UAVs’ on-board inertial sensors with the visual estimates. The suggested contributions have been exhaustively evaluated over a number of real-image data experiments in the laboratory using monocular vision systems and range imaging devices. In this thesis, we propose several solutions towards the goal of robust visual motion estimation using convex optimisation. We show that the convex optimisation framework may be extended to include uncertainty information, to achieve robust and optimal solutions. We observed that convex optimisation is a practical and very appealing alternative to linear techniques and iterative methods

Object-Aware Tracking and Mapping

Reasoning about geometric properties of digital cameras and optical physics enabled researchers to build methods that localise cameras in 3D space from a video stream, while – often simultaneously – constructing a model of the environment. Related techniques have evolved substantially since the 1980s, leading to increasingly accurate estimations. Traditionally, however, the quality of results is strongly affected by the presence of moving objects, incomplete data, or difficult surfaces – i.e. surfaces that are not Lambertian or lack texture. One insight of this work is that these problems can be addressed by going beyond geometrical and optical constraints, in favour of object level and semantic constraints. Incorporating specific types of prior knowledge in the inference process, such as motion or shape priors, leads to approaches with distinct advantages and disadvantages. After introducing relevant concepts in Chapter 1 and Chapter 2, methods for building object-centric maps in dynamic environments using motion priors are investigated in Chapter 5. Chapter 6 addresses the same problem as Chapter 5, but presents an approach which relies on semantic priors rather than motion cues. To fully exploit semantic information, Chapter 7 discusses the conditioning of shape representations on prior knowledge and the practical application to monocular, object-aware reconstruction systems

Visual SLAM muuttuvissa ympäristöissä

This thesis investigates the problem of Visual Simultaneous Localization and Mapping (vSLAM) in changing environments. The vSLAM problem is to sequentially estimate the pose of a device with mounted cameras in a map generated based on images taken with those cameras. vSLAM algorithms face two main challenges in changing environments: moving objects and temporal appearance changes. Moving objects cause problems in pose estimation if they are mistaken for static objects. Moving objects also cause problems for loop closure detection (LCD), which is the problem of detecting whether a previously visited place has been revisited. A same moving object observed in two different places may cause false loop closures to be detected. Temporal appearance changes such as those brought about by time of day or weather changes cause long-term data association errors for LCD. These cause difficulties in recognizing previously visited places after they have undergone appearance changes. Focus is placed on LCD, which turns out to be the part of vSLAM that changing environment affects the most. In addition, several techniques and algorithms for Visual Place Recognition (VPR) in challenging conditions that could be used in the context of LCD are surveyed and the performance of two state-of-the-art modern VPR algorithms in changing environments is assessed in an experiment in order to measure their applicability for LCD. The most severe performance degrading appearance changes are found to be those caused by change in season and illumination. Several algorithms and techniques that perform well in loop closure related tasks in specific environmental conditions are identified as a result of the survey. Finally, a limited experiment on the Nordland dataset implies that the tested VPR algorithms are usable as is or can be modified for use in long-term LCD. As a part of the experiment, a new simple neighborhood consistency check was also developed, evaluated, and found to be effective at reducing false positives output by the tested VPR algorithms

Vision Based Collaborative Localization and Path Planning for Micro Aerial Vehicles

Autonomous micro aerial vehicles (MAV) have gained immense popularity in both the commercial and research worlds over the last few years. Due to their small size and agility, MAVs are considered to have great potential for civil and industrial tasks such as photography, search and rescue, exploration, inspection and surveillance. Autonomy on MAVs usually involves solving the major problems of localization and path planning. While GPS is a popular choice for localization for many MAV platforms today, it suffers from issues such as inaccurate estimation around large structures, and complete unavailability in remote areas/indoor scenarios. From the alternative sensing mechanisms, cameras arise as an attractive choice to be an onboard sensor due to the richness of information captured, along with small size and inexpensiveness. Another consideration that comes into picture for micro aerial vehicles is the fact that these small platforms suffer from inability to fly for long amounts of time or carry heavy payload, scenarios that can be solved by allocating a group, or a swarm of MAVs to perform a task than just one. Collaboration between multiple vehicles allows for better accuracy of estimation, task distribution and mission efficiency. Combining these rationales, this dissertation presents collaborative vision based localization and path planning frameworks. Although these were created as two separate steps, the ideal application would contain both of them as a loosely coupled localization and planning algorithm. A forward-facing monocular camera onboard each MAV is considered as the sole sensor for computing pose estimates. With this minimal setup, this dissertation first investigates methods to perform feature-based localization, with the possibility of fusing two types of localization data: one that is computed onboard each MAV, and the other that comes from relative measurements between the vehicles. Feature based methods were preferred over direct methods for vision because of the relative ease with which tangible data packets can be transferred between vehicles, and because feature data allows for minimal data transfer compared to large images. Inspired by techniques from multiple view geometry and structure from motion, this localization algorithm presents a decentralized full 6-degree of freedom pose estimation method complete with a consistent fusion methodology to obtain robust estimates only at discrete instants, thus not requiring constant communication between vehicles. This method was validated on image data obtained from high fidelity simulations as well as real life MAV tests. These vision based collaborative constraints were also applied to the problem of path planning with a focus on performing uncertainty-aware planning, where the algorithm is responsible for generating not only a valid, collision-free path, but also making sure that this path allows for successful localization throughout. As joint multi-robot planning can be a computationally intractable problem, planning was divided into two steps from a vision-aware perspective. As the first step for improving localization performance is having access to a better map of features, a next-best-multi-view algorithm was developed which can compute the best viewpoints for multiple vehicles that can improve an existing sparse reconstruction. This algorithm contains a cost function containing vision-based heuristics that determines the quality of expected images from any set of viewpoints; which is minimized through an efficient evolutionary strategy known as Covariance Matrix Adaption (CMA-ES) that can handle very high dimensional sample spaces. In the second step, a sampling based planner called Vision-Aware RRT* (VA-RRT*) was developed which includes similar vision heuristics in an information gain based framework in order to drive individual vehicles towards areas that can benefit feature tracking and thus localization. Both steps of the planning framework were tested and validated using results from simulation

SLAM for drones : simultaneous localization and mapping for autonomous flying robots

The main objective of this thesis is to be a reference in SLAM for future work in robotics. It goes from almost a zero-point for a non-expert in the field until a revision of the SoA methods. It has been carefully divided into four parts: - The first one is a compilation of the basis in computer vision. If you are new into the field, it is recommended to read it carefully to really understand the most important concepts that will be applied in further sections. - The second part will be a full revision from zero of SLAM techniques, focusing on the award winning KinectFusion and other SoA methods. - The third part goes from a general flying robots overview in history until the mechanical model of a quadrotor. It has been intended to be completely apart from section two, for the case it has been determined to only focus on the vision part of this thesis. - The fourth part is a pro-cons overview of the SLAM methods described, applied into flying robots. We will finish with the conclusions and future work of this MSc research. ____________________________________________________________________________________________________________________El objetivo del proyecto es realizar documento que ordene, clasifique y explique desde un nivel básico hasta las técnicas más punteras, todo lo que el acrónimo SLAM engloba. Además nos focalizaremos en concreto en resolver el problema para robots voladores no tripulados. El documento original se divide en cuatro bloques principales precedidos por agradecimientos, una definición de los objetivos de la tesis, e introducción. Éstos cuatro bloques son: 1: Primera parte: Conceptos básicos de visión por computador 2: Segunda parte: S.L.A.M. 3: Tercera parte: cuadrotores 4: Cuarta parte: SLAM para robots voladores Por último incluye un apartado de trabajo futuro y conclusiones.Ingeniería Industria

Micro Aerial Vehicles (MAV) Assured Navigation in Search and Rescue Missions Robust Localization, Mapping and Detection

This Master's Thesis describes the developments on robust localization, mapping and detection algorithms for Micro Aerial Vehicles (MAVs). The localization method proposes a seamless indoor-outdoor multi-sensor architecture. This algorithm is capable of using all or a subset of its sensor inputs to determine a platform's position, velocity and attitude (PVA). It relies on the Inertial Measurement Unit as the core sensor and monitors the status and observability of the secondary sensors to select the most optimum estimator strategy for each situation. Furthermore, it ensures a smooth transition between filters structures. This document also describes the integration mechanism for a set of common sensors such as GNSS receivers, laser scanners and stereo and mono cameras. The mapping algorithm provides a fully automated fast aerial mapping pipeline. It speeds up the process by pre-selecting the images using the flight plan and the onboard localization. Furthermore, it relies on Structure from Motion (SfM) techniques to produce an optimized 3D reconstruction of camera locations and sparse scene geometry. These outputs are used to compute the perspective transformations that project the raw images on the ground and produce a geo-referenced map. Finally, these maps are fused with other domains in a collaborative UGV and UAV mapping algorithms. The real-time aerial detection of victims is based on a thermal camera. The algorithm is composed by three steps. Firstly, a normalization of the image is performed to get rid of the background and to extract the regions of interest. Later the victim detection and tracking steps produce the real-time geo-referenced locations of the detections. The thesis also proposes the concept of a MAV Copilot, a payload composed by a set of sensors and algorithm the enhances the capabilities of any commercial MAV. To develop and validate these contributions, a prototype of a search and rescue MAV and the Copilot has been developed. These developments have been validated in three large-scale demonstrations of search and rescue operations in the context of the European project ICARUS: a shipwreck in Lisbon (Portugal), an earthquake in Marche (Belgium), and the Fukushima nuclear disaster in the euRathlon 2015 competition in Piombino (Italy)

Probabilistic Outlier Removal for Stereo Visual Odometry

Thesis (MEng)--Stellenbosch University, 2017.ENGLISH ABSTRACT: The field of autonomous navigation is currently receiving significant attention from researchers in both academia and industry. With an end goal of fully autonomous vehicle systems, an increased effort is being made to develop systems that are more efficient, reliable and safe than human-controlled vehicles. Furthermore, the low cost and compact nature of cameras have led to an increased interest in vision-based navigation techniques. Despite their popularity, measurements obtained from cameras are often noisy and contaminated with outliers. A critical requirement for consistent and reliable autonomous navigation is the ability to identify and remove these outliers when measurements are highly uncertain. The focus of the research presented in this thesis is therefore on effective and efficient outlier removal. Many existing outlier removal methods are limited in their ability to handle datasets that are contaminated by a significant number of outliers in real-time. Furthermore, many of the current techniques perform inconsistently in the presence of high measurement noise. This thesis proposes methods for probabilistic outlier removal in a robust, real-time visual odometry framework. No assumptions are made about the vehicle motion or the environment, thereby keeping the research in a general form and allowing it to be applied to a wide variety of applications. The first part of this thesis details the modelling of sensor measurements obtained from a camera pair. The mapping process from 3D space to image space is described mathematically and the concept of triangulating matched image features is presented. Stereo measurements are modelled as random variables that are assumed to be normally distributed in image coordinates. Two techniques used for uncertainty propagation, linearisation and the unscented transform, are investigated. The results of experiments, performed on synthetic datasets, are presented and show that the unscented transform outperforms linearisation when used to approximate the distributions of reconstructed, 3D features. The second part of this thesis presents the development of a novel outlier removal technique, which is reliable and efficient. Instead of performing outlier removal with the standard hypothesise-and-verify approach of RANSAC, a novel mechanism is developed that uses a probabilistic measure of shape similarity to identify sets of points containing outliers. The measure of shape similarity is based on inherent spatial constraints, and is combined with an adaptive sampling approach to determine the probability of individual points being outliers. This novel approach is compared against a state-of-the-art RANSAC technique, where experiments indicate that the proposed method is more efficient and leads to more consistent motion estimation results. The novel outlier removal approach is also incorporated into a robust visual odometry pipeline that is tested on both synthetic and practical datasets. The results obtained from visual odometry experiments indicate that the proposed method is significantly faster than RANSAC, making it viable for real-time applications, and reliable for outlier removal even when measurements are highly uncertain.AFRIKAANSE OPSOMMING: Die area van outonome navigasie kry tans vele aandag van navorsers in akademie en in die bedryf. Met ’n einddoel van volledige outonome navigasie voertuigstelsels, word ’n verhoogde poging gemaak om stelsels te ontwerp wat meer effektief, betroubaar en veiliger is as menslik beheerde voertuie. Verder, die lae prys en kompakte struktuur van kameras het gelei tot ’n verhoogde belangstelling in visie gebaseerde navigasie tegnieke. Ten spyte van hierdie gewildheid, is kamera metings gewoonlik ruiserig en besoedel met uitskieters. ’n Kritiese vereiste vir konsekwente en betroubare outonome navigasie is die vermoë om uitskieters te kan identifiseer en verwyder as die metings hoogs onseker is. Die fokus van die navorsing wat in hierdie tesis aangebied sal word is dus op effektiewe en doeltreffende uitskieterverwydering. Talle bestaande uitskieterverwydermetodes is beperk in hulle vermoë om datastelle besoedel met vele uitskieters intyds te kan hanteer. Verder, talle van die huidige tegnieke tree inkonsekwent in die teenwoordigheid van hoë ruis op. Hierdie tesis stel metodes voor vir waarskynliksheid-verwydering van uitskieters in ’n kragtige, intydse, visuele verplasingsmeter raamwerk. Geen aannames word gemaak oor die voertuig se beweging of die omgewing nie. Die navorsing word dus algemeen gehou en laat toe om toegepas te word op verskillende toepassings. Die eerste gedeelte van hierdie tesis verduidelik die modellering van sensor metings geneem van ’n kamera paar. Die karteringsproses van 3D ruimte na beeld ruimte word wiskundig verduidelik en die konsep van triangulasie van ooreenstemmende beeldkenmerke word aangebied. Stereometings word gebruik as toevalsveranderlikes wat aanvaar word as normaal versprei in die beeld koördinate. Twee tegnieke wat gebruik word vir onsekerheid vooruitskatting, ’n lineariseringsmetode en die sigmapunt-transformasie, word ondersoek. Die resultate van eksperimente wat uitgevoer is op sintetiese datastelle word aangebied, en dit wys dat die sigmapunt-transformasie beter funksioneer as die lineariseringsmetode wanneer dit gebruik word om die verspreiding van gerekonstrueerde, 3D kenmerke te benader. Die tweede gedeelte van hierdie tesis bied die ontwikkeling van ’n nuwe uitskieterverwyderingsmetode, wat betroubaar en doeltreffend is aan. In plaas van uitskieters te verwyder met RANSAC se standaard tegniek van hipotetiseer-en-verifieer, word ’n nuwe meganisme ontwikkel wat vorm ooreenkoms meet om stelle punte wat uitskieters bevat te identifiseer. Die meting van vorm ooreenkoms is gebaseer op ingebore ruimtelike beperkings en word gekombineer met aanpasbare monstering om die waarskynlikheid van sekere punte om uitskieters te wees te bepaal. Hierdie nuwe benadering word vergelyk teen RANSAC waar eksperimente wys dat die voorgestelde metode meer doeltreffend is en lei tot meer konsekwente resultate. Die nuwe uitskieterverwyderingsmetode is ook opgeneem in ’n kragtige visuele verplasingsmeter wat getoets is met beide sintetiese en praktiese datastelle. Die resultate wat behaal is van die visuele verplasingsmeter eksperimente dui aan dat die voorgestelde metode aansienlik vinniger is as RANSAC, wat dit haalbaar maak vir intydse toepassings, en betroubaar is vir uitskieterverywydering al is die metings hoogs onseker