Monocular Visual-Inertial Slam and Self Calibration for Long Term Autonomy

This thesis is concerned with real-time monocular visual-inertial simultaneous localization and mapping (VI-SLAM) with application to Long Term Autonomy. Given a sensor rig capable of making visual and inertial measurements, accurate real-time estimation of its position and orientation (pose) as well as the creation of a scale-correct map of the surrounding environment is desired. This estimation task requires accurate calibration of both intrinsic and extrinsic properties of the visual and inertial sensors. As such, the continuous estimation of these calibration parameters is also desired. Three novel methods are presented, covering real-time VI-SLAM, self calibration, and change detection. Together they form a basis for long term localization and mapping robust to changes in calibration.The VI-slam methodology is motivated by the requirement to produce a scale-correct visual map, in an optimization framework that is able to incorporate relocalization and loop closure constraints. Special attention is paid to achieve robustness to many real world difficulties, including degenerate motions and unobservablity. A variety of helpful techniques are used, including: a relative manifold representation, a minimal-state inverse depth parameterization, and robust non- metric initialization and tracking. Also presented is an extensible framework for real-time self-calibration of cameras in the SLAM setting. The system is demonstrated to calibrate both pinhole and fish-eye camera models from unknown initial parameters while seamlessly solving the maximum likelihood online SLAM problem in real-time. Self-calibration is performed by tracking image features, and requires no predetermined calibration target. By automatically identifying and using only those portions of the sequence that contain useful information for the purpose of calibration the system achieves accurate results incrementally and in constant-time vs. the number of images. Finally, a framework for online SLAM and self-calibration is presented which can detect and handle significant change in the calibration parameters. A novel technique is presented to detect the probability that a significant change is present in the calibration parameters. The system is then able to re-calibrate. Maximum likelihood trajectory and map estimates are computed using an asynchronous and adaptive optimization. The system requires no prior information and is able to initialize without any special motions or routines, or in the case where observability over calibration parameters is delayed. Both self-calibration frameworks are extensible and able to cover any calibration parameters which can be estimated from the measurements. The contributions are individually evaluated in a number of experiments with real data. Specific focus is placed on accuracy and real-time performance

Attention and Anticipation in Fast Visual-Inertial Navigation

We study a Visual-Inertial Navigation (VIN) problem in which a robot needs to estimate its state using an on-board camera and an inertial sensor, without any prior knowledge of the external environment. We consider the case in which the robot can allocate limited resources to VIN, due to tight computational constraints. Therefore, we answer the following question: under limited resources, what are the most relevant visual cues to maximize the performance of visual-inertial navigation? Our approach has four key ingredients. First, it is task-driven, in that the selection of the visual cues is guided by a metric quantifying the VIN performance. Second, it exploits the notion of anticipation, since it uses a simplified model for forward-simulation of robot dynamics, predicting the utility of a set of visual cues over a future time horizon. Third, it is efficient and easy to implement, since it leads to a greedy algorithm for the selection of the most relevant visual cues. Fourth, it provides formal performance guarantees: we leverage submodularity to prove that the greedy selection cannot be far from the optimal (combinatorial) selection. Simulations and real experiments on agile drones show that our approach ensures state-of-the-art VIN performance while maintaining a lean processing time. In the easy scenarios, our approach outperforms appearance-based feature selection in terms of localization errors. In the most challenging scenarios, it enables accurate visual-inertial navigation while appearance-based feature selection fails to track robot's motion during aggressive maneuvers.Comment: 20 pages, 7 figures, 2 table

Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach

Autonomous navigation of miniaturized robots (e.g., nano/pico aerial vehicles) is currently a grand challenge for robotics research, due to the need of processing a large amount of sensor data (e.g., camera frames) with limited on-board computational resources. In this paper we focus on the design of a visual-inertial odometry (VIO) system in which the robot estimates its ego-motion (and a landmark-based map) from on- board camera and IMU data. We argue that scaling down VIO to miniaturized platforms (without sacrificing performance) requires a paradigm shift in the design of perception algorithms, and we advocate a co-design approach in which algorithmic and hardware design choices are tightly coupled. Our contribution is four-fold. First, we discuss the VIO co-design problem, in which one tries to attain a desired resource-performance trade-off, by making suitable design choices (in terms of hardware, algorithms, implementation, and parameters). Second, we characterize the design space, by discussing how a relevant set of design choices affects the resource-performance trade-off in VIO. Third, we provide a systematic experiment-driven way to explore the design space, towards a design that meets the desired trade-off. Fourth, we demonstrate the result of the co-design process by providing a VIO implementation on specialized hardware and showing that such implementation has the same accuracy and speed of a desktop implementation, while requiring a fraction of the power.United States. Air Force Office of Scientific Research. Young Investigator Program (FA9550-16-1-0228)National Science Foundation (U.S.) (NSF CAREER 1350685

Mixed Reality and Remote Sensing Application of Unmanned Aerial Vehicle in Fire and Smoke Detection

This paper proposes the development of a system incorporating inertial measurement unit (IMU), a consumer-grade digital camera and a fire detection algorithm simultaneously with a nano Unmanned Aerial Vehicle (UAV) for inspection purposes. The video streams are collected through the monocular camera and navigation relied on the state-of-the-art indoor/outdoor Simultaneous Localisation and Mapping (SLAM) system. It implements the robotic operating system (ROS) and computer vision algorithm to provide a robust, accurate and unique inter-frame motion estimation. The collected onboard data are communicated to the ground station and used the SLAM system to generate a map of the environment. A robust and efficient re-localization was performed to recover from tracking failure, motion blur, and frame lost in the data received. The fire detection algorithm was deployed based on the colour, movement attributes, temporal variation of fire intensity and its accumulation around a point. The cumulative time derivative matrix was utilized to analyze the frame-by-frame changes and to detect areas with high-frequency luminance flicker (random characteristic). Colour, surface coarseness, boundary roughness, and skewness features were perceived as the quadrotor flew autonomously within the clutter and congested area. Mixed Reality system was adopted to visualize and test the proposed system in a physical environment, and the virtual simulation was conducted through the Unity game engine. The results showed that the UAV could successfully detect fire and flame, autonomously fly towards and hover around it, communicate with the ground station and simultaneously generate a map of the environment. There was a slight error between the real and virtual UAV calibration due to the ground truth data and the correlation complexity of tracking real and virtual camera coordinate frames

FutureMapping 2: Gaussian Belief Propagation for Spatial AI

We argue the case for Gaussian Belief Propagation (GBP) as a strong algorithmic framework for the distributed, generic and incremental probabilistic estimation we need in Spatial AI as we aim at high performance smart robots and devices which operate within the constraints of real products. Processor hardware is changing rapidly, and GBP has the right character to take advantage of highly distributed processing and storage while estimating global quantities, as well as great flexibility. We present a detailed tutorial on GBP, relating to the standard factor graph formulation used in robotics and computer vision, and give several simulation examples with code which demonstrate its properties

Towards Robust Dense Visual Simultaneous Localization and Mapping (SLAM)

Long-term autonomy is the dream of many roboticists – and if a robotic system can be split into three main categories: perception, planning and control – then the biggest challenges to achieve this dream are undoubtedly faced in perception. Large scale environments that change with time – due to normal operations or even lighting changes – are typical situations the robot would encounter. As such, a Simultaneous Localization and Mapping (SLAM) system that is robust enough to handle many of these conditions is desired.The objective of this dissertation is to present components that would lead to a robust dense visual SLAM system. It starts by exploring 3D reconstruction algorithms, showing distinctions between local and global methods and presenting an incremental and adaptive global method designed to create depth maps as a robot navigates in space.It then introduces the concept of sensor fusion, where multiple sensors are joined to provide a higher degree of tracking accuracy. It compares different visual SLAM systems – dense and semi-dense – and shows how the inclusion of an Inertial Measurement Unit (IMU) aids considerably in tracking. It then uses this localization framework in a large scale volumetric mapping system, and shows results for both indoor and outdoor environments using real world datasets.Finally, it explores different error metrics used in direct photometric optimization – the foundation of dense tracking systems. It introduces the Normalized Information Distance (NID), an entropy based metric that is shown to achieve high localization success rate and accuracy even in the face of extreme lighting differences