Keyframe-based visual–inertial odometry using nonlinear optimization

Combining visual and inertial measurements has become popular in mobile robotics, since the two sensing modalities offer complementary characteristics that make them the ideal choice for accurate visual–inertial odometry or simultaneous localization and mapping (SLAM). While historically the problem has been addressed with filtering, advancements in visual estimation suggest that nonlinear optimization offers superior accuracy, while still tractable in complexity thanks to the sparsity of the underlying problem. Taking inspiration from these findings, we formulate a rigorously probabilistic cost function that combines reprojection errors of landmarks and inertial terms. The problem is kept tractable and thus ensuring real-time operation by limiting the optimization to a bounded window of keyframes through marginalization. Keyframes may be spaced in time by arbitrary intervals, while still related by linearized inertial terms. We present evaluation results on complementary datasets recorded with our custom-built stereo visual–inertial hardware that accurately synchronizes accelerometer and gyroscope measurements with imagery. A comparison of both a stereo and monocular version of our algorithm with and without online extrinsics estimation is shown with respect to ground truth. Furthermore, we compare the performance to an implementation of a state-of-the-art stochastic cloning sliding-window filter. This competitive reference implementation performs tightly coupled filtering-based visual–inertial odometry. While our approach declaredly demands more computation, we show its superior performance in terms of accuracy

KEYFRAME-BASED VISUAL-INERTIAL SLAM USING NONLINEAR OPTIMIZATION

Abstract—The fusion of visual and inertial cues has become popular in robotics due to the complementary nature of the two sensing modalities. While most fusion strategies to date rely on filtering schemes, the visual robotics community has recently turned to non-linear optimization approaches for tasks such as visual Simultaneous Localization And Mapping (SLAM), following the discovery that this comes with significant advantages in quality of performance and computational complexity. Following this trend, we present a novel approach to tightly integrate visual measurements with readings from an Inertial Measurement Unit (IMU) in SLAM. An IMU error term is integrated with the landmark reprojection error in a fully probabilistic manner, resulting to a joint non-linear cost function to be optimized. Employing the powerful concept of ‘keyframes ’ we partially marginalize old states to maintain a bounded-sized optimization window, ensuring real-time operation. Comparing against both vision-only and loosely-coupled visual-inertial algorithms, our experiments confirm the benefits of tight fusion in terms of accuracy and robustness. I

3D Visual Perception for Self-Driving Cars using a Multi-Camera System: Calibration, Mapping, Localization, and Obstacle Detection

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction

Continuous-Time Estimation of Attitude Using B-Splines on Lie Groups

Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/140656/1/1.g001149.pd

A mission control architecture for robotic lunar sample return as field tested in an analogue deployment to the Sudbury impact structure

A Mission Control Architecture is presented for a Robotic Lunar Sample Return Mission which builds upon the experience of the landed missions of the NASA Mars Exploration Program. This architecture consists of four separate processes working in parallel at Mission Control and achieving buy-in for plans sequentially instead of simultaneously from all members of the team. These four processes were: Science Processing, Science Interpretation, Planning and Mission Evaluation. Science Processing was responsible for creating products from data downlinked from the field and is organized by instrument. Science Interpretation was responsible for determining whether or not science goals are being met and what measurements need to be taken to satisfy these goals. The Planning process, responsible for scheduling and sequencing observations, and the Evaluation process that fostered inter-process communications, reporting and documentation assisted these processes. This organization is advantageous for its flexibility as shown by the ability of the structure to produce plans for the rover every two hours, for the rapidity with which Mission Control team members may be trained and for the relatively small size of each individual team. This architecture was tested in an analogue mission to the Sudbury impact structure from June 6-17, 2011. A rover was used which was capable of developing a network of locations that could be revisited using a teach and repeat method. This allowed the science team to process several different outcrops in parallel, downselecting at each stage to ensure that the samples selected for caching were the most representative of the site. Over the course of 10 days, 18 rock samples were collected from 5 different outcrops, 182 individual field activities - such as roving or acquiring an image mosaic or other data product - were completed within 43 command cycles, and the rover travelled over 2,200 m. Data transfer from communications passes were filled to 74%. Sample triage was simulated to allow down-selection to 1kg of material for return to Earth

Image features for visual teach-and-repeat navigation in changing environments

We present an evaluation of standard image features in the context of long-term visual teach-and-repeat navigation of mobile robots, where the environment exhibits significant changes in appearance caused by seasonal weather variations and daily illumination changes. We argue that for long-term autonomous navigation, the viewpoint-, scale- and rotation- invariance of the standard feature extractors is less important than their robustness to the mid- and long-term environment appearance changes. Therefore, we focus our evaluation on the robustness of image registration to variable lighting and naturally-occurring seasonal changes. We combine detection and description components of different image extractors and evaluate their performance on five datasets collected by mobile vehicles in three different outdoor environments over the course of one year. Moreover, we propose a trainable feature descriptor based on a combination of evolutionary algorithms and Binary Robust Independent Elementary Features, which we call GRIEF (Generated BRIEF). In terms of robustness to seasonal changes, the most promising results were achieved by the SpG/CNN and the STAR/GRIEF feature, which was slightly less robust, but faster to calculate

Mixed Reality and Remote Sensing Application of Unmanned Aerial Vehicle in Fire and Smoke Detection

This paper proposes the development of a system incorporating inertial measurement unit (IMU), a consumer-grade digital camera and a fire detection algorithm simultaneously with a nano Unmanned Aerial Vehicle (UAV) for inspection purposes. The video streams are collected through the monocular camera and navigation relied on the state-of-the-art indoor/outdoor Simultaneous Localisation and Mapping (SLAM) system. It implements the robotic operating system (ROS) and computer vision algorithm to provide a robust, accurate and unique inter-frame motion estimation. The collected onboard data are communicated to the ground station and used the SLAM system to generate a map of the environment. A robust and efficient re-localization was performed to recover from tracking failure, motion blur, and frame lost in the data received. The fire detection algorithm was deployed based on the colour, movement attributes, temporal variation of fire intensity and its accumulation around a point. The cumulative time derivative matrix was utilized to analyze the frame-by-frame changes and to detect areas with high-frequency luminance flicker (random characteristic). Colour, surface coarseness, boundary roughness, and skewness features were perceived as the quadrotor flew autonomously within the clutter and congested area. Mixed Reality system was adopted to visualize and test the proposed system in a physical environment, and the virtual simulation was conducted through the Unity game engine. The results showed that the UAV could successfully detect fire and flame, autonomously fly towards and hover around it, communicate with the ground station and simultaneously generate a map of the environment. There was a slight error between the real and virtual UAV calibration due to the ground truth data and the correlation complexity of tracking real and virtual camera coordinate frames

Extensions to the Visual Odometry Pipeline for the Exploration of Planetary Surfaces

Mars represents one of the most important targets for space exploration in the next 10 to 30 years, particularly because of evidence of liquid water in the planet's past. Current environmental conditions dictate that any existing water reserves will be in the form of ice; finding and sampling these ice deposits would further the study of the planet's climate history, further the search for evidence of life, and facilitate in-situ resource utilization during future manned exploration missions. This thesis presents a suite of algorithms to help enable a robotic ice-prospecting mission to Mars. Starting from visual odometry---the estimation of a rover's motion using a stereo camera as the primary sensor---we develop the following extensions: (i) a coupled surface/subsurface modelling system that provides novel data products to scientists working remotely, (ii) an autonomous retrotraverse system that allows a rover to return to previously visited places along a route for sampling, or to return a sample to an ascent vehicle, and (iii) the extension of the appearance-based visual odometry pipeline to an actively illuminated light detection and ranging sensor that provides data similar to a stereo camera but is not reliant on consistent ambient lighting, thereby enabling appearance-based vision techniques to be used in environments that are not conducive to passive cameras, such as underground mines or permanently shadowed craters on the moon. All algorithms are evaluated on real data collected using our field robot at the University of Toronto Institute for Aerospace Studies, or at a planetary analogue site on Devon Island, in the Canadian High Arctic.Ph