106 research outputs found

    Cross-View Visual Geo-Localization for Outdoor Augmented Reality

    Full text link
    Precise estimation of global orientation and location is critical to ensure a compelling outdoor Augmented Reality (AR) experience. We address the problem of geo-pose estimation by cross-view matching of query ground images to a geo-referenced aerial satellite image database. Recently, neural network-based methods have shown state-of-the-art performance in cross-view matching. However, most of the prior works focus only on location estimation, ignoring orientation, which cannot meet the requirements in outdoor AR applications. We propose a new transformer neural network-based model and a modified triplet ranking loss for joint location and orientation estimation. Experiments on several benchmark cross-view geo-localization datasets show that our model achieves state-of-the-art performance. Furthermore, we present an approach to extend the single image query-based geo-localization approach by utilizing temporal information from a navigation pipeline for robust continuous geo-localization. Experimentation on several large-scale real-world video sequences demonstrates that our approach enables high-precision and stable AR insertion.Comment: IEEE VR 202

    Robust multispectral image-based localisation solutions for autonomous systems

    Get PDF
    With the recent increase of interest in multispectral imaging, new image-based localisation solutions have emerged. However, its application to visual odometry remains overlooked. Most localisation techniques are still being developed with visible cameras only, because the portability they can offer and the wide variety of cameras available. Yet, other modalities have great potentials for navigation purposes. Infrared imaging for example, provides different information about the scene and is already used to enhance visible images. This is especially the case of far-infrared cameras which can produce images at night and see hot objects like other cars, animals or pedestrians. Therefore, the aim of this thesis is to tackle the lack of research in multispectral localisation and to explore new ways of performing visual odometry accurately with visible and thermal images. First, a new calibration pattern made of LED lights is presented in Chapter 3. Emitting both visible and thermal radiations, it can easily be seen by infrared and visible cameras. Due to its peculiar shape, the whole pattern can be moved around the cameras and automatically identified in the different images recorded. Monocular and stereo calibration are then performed to precisely estimate the camera parameters. Then, a multispectral monocular visual odometry algorithm is proposed in Chapter 4. This generic technique is able to operate in infrared and visible modalities, regardless of the nature of the images. Incoming images are processed at a high frame rate based on a 2D-to-2D unscaled motion estimation method. However, specific keyframes are carefully selected to avoid degenerate cases and a bundle adjustment optimisation is performed on a sliding window to refine the initial estimation. The advantage of visible-thermal odometry is shown on a scenario with extreme illumination conditions, where the limitation of each modality is reached. The simultaneous combination of visible and thermal images for visual odometry is also explored. In Chapter 5, two feature matching techniques are presented and tested in a multispectral stereo visual odometry framework. One method matches features between stereo pairs independently while the other estimates unscaled motion first, before matching the features altogether. Even though these techniques require more processing power to overcome the dissimilarities between V multimodal images, they have the benefit of estimating scaled transformations. Finally, the camera pose estimates obtained with multispectral stereo odometry are fused with inertial data to create a robustified localisation solution which is detailed in Chapter 6. The full state of the system is estimated, including position, velocity, orientation and IMU biases. It is shown that multispectral visual odometry can correct drifting IMU measurements effectively. Furthermore, it is demonstrated that such multi-sensors setups can be beneficial in challenging situations where features cannot be extracted or tracked. In that case, inertial data can be integrated to provide a state estimate while visual odometry cannot

    Advanced Location-Based Technologies and Services

    Get PDF
    Since the publication of the first edition in 2004, advances in mobile devices, positioning sensors, WiFi fingerprinting, and wireless communications, among others, have paved the way for developing new and advanced location-based services (LBSs). This second edition provides up-to-date information on LBSs, including WiFi fingerprinting, mobile computing, geospatial clouds, geospatial data mining, location privacy, and location-based social networking. It also includes new chapters on application areas such as LBSs for public health, indoor navigation, and advertising. In addition, the chapter on remote sensing has been revised to address advancements

    A Survey on Odometry for Autonomous Navigation Systems

    Get PDF
    The development of a navigation system is one of the major challenges in building a fully autonomous platform. Full autonomy requires a dependable navigation capability not only in a perfect situation with clear GPS signals but also in situations, where the GPS is unreliable. Therefore, self-contained odometry systems have attracted much attention recently. This paper provides a general and comprehensive overview of the state of the art in the field of self-contained, i.e., GPS denied odometry systems, and identifies the out-coming challenges that demand further research in future. Self-contained odometry methods are categorized into five main types, i.e., wheel, inertial, laser, radar, and visual, where such categorization is based on the type of the sensor data being used for the odometry. Most of the research in the field is focused on analyzing the sensor data exhaustively or partially to extract the vehicle pose. Different combinations and fusions of sensor data in a tightly/loosely coupled manner and with filtering or optimizing fusion method have been investigated. We analyze the advantages and weaknesses of each approach in terms of different evaluation metrics, such as performance, response time, energy efficiency, and accuracy, which can be a useful guideline for researchers and engineers in the field. In the end, some future research challenges in the field are discussed

    Adaptive Localization and Mapping for Planetary Rovers

    Get PDF
    Future rovers will be equipped with substantial onboard autonomy as space agencies and industry proceed with missions studies and technology development in preparation for the next planetary exploration missions. Simultaneous Localization and Mapping (SLAM) is a fundamental part of autonomous capabilities and has close connections to robot perception, planning and control. SLAM positively affects rover operations and mission success. The SLAM community has made great progress in the last decade by enabling real world solutions in terrestrial applications and is nowadays addressing important challenges in robust performance, scalability, high-level understanding, resources awareness and domain adaptation. In this thesis, an adaptive SLAM system is proposed in order to improve rover navigation performance and demand. This research presents a novel localization and mapping solution following a bottom-up approach. It starts with an Attitude and Heading Reference System (AHRS), continues with a 3D odometry dead reckoning solution and builds up to a full graph optimization scheme which uses visual odometry and takes into account rover traction performance, bringing scalability to modern SLAM solutions. A design procedure is presented in order to incorporate inertial sensors into the AHRS. The procedure follows three steps: error characterization, model derivation and filter design. A complete kinematics model of the rover locomotion subsystem is developed in order to improve the wheel odometry solution. Consequently, the parametric model predicts delta poses by solving a system of equations with weighed least squares. In addition, an odometry error model is learned using Gaussian processes (GPs) in order to predict non-systematic errors induced by poor traction of the rover with the terrain. The odometry error model complements the parametric solution by adding an estimation of the error. The gained information serves to adapt the localization and mapping solution to the current navigation demands (domain adaptation). The adaptivity strategy is designed to adjust the visual odometry computational load (active perception) and to influence the optimization back-end by including highly informative keyframes in the graph (adaptive information gain). Following this strategy, the solution is adapted to the navigation demands, providing an adaptive SLAM system driven by the navigation performance and conditions of the interaction with the terrain. The proposed methodology is experimentally verified on a representative planetary rover under realistic field test scenarios. This thesis introduces a modern SLAM system which adapts the estimated pose and map to the predicted error. The system maintains accuracy with fewer nodes, taking the best of both wheel and visual methods in a consistent graph-based smoothing approach

    LiDAR Place Recognition with Image Retrieval

    Get PDF
    This thesis is about LiDAR place recognition. Place recognition is the problem of being able to recognize already seen or visited places ā€“ an important sub-problem of robot navigation. LiDAR sensors oļ¬€er accurate and cost-eļ¬€ective range and reļ¬‚ec-tivity data that can replace or complement RGB cameras. Place recognition has been studied with diļ¬€erent sensors and methods for many years. Traditional methods use handcrafted features to match images in order to recognize places. In recent years, the surge of deep learning has made learned features the main approach. In this work LiDAR place recognition is studied with exported 2D pixel images and deep learning models. Place recognition is posed as an image retrieval problem, where a model is trained to learn a feature space in such a way that the similarity of images can be conveniently compared. With a trained model, one can use an image to search for other similar images, and thus recognize places. The key ļ¬nding of the thesis publications is that place recognition with image retrieval using exported pixel images from LiDAR scans is a well performing method, as evidenced by achieving about 80% recall@1 with 5 meter test distance in urban outdoors and 1 meter indoors. The other key ļ¬ndings are: Loop points in the route are detectable with image retrieval type methods. LiDAR is a competitive modality versus RGB. LiDAR depth maps are more robust to change than intensity maps. Generalized mean is a good pooling method for place recognition. Simulated data is beneļ¬cial when mixed in with real-world data at a suitable ratio. Dataset quality is very important in regards to ground truth position and LiDAR resolution

    Multimodal Navigation for Accurate Space Rendezvous Missions

    Get PDF
    Ā© Cranfield University 2021. All rights reserved. No part of this publication may be reproduced without the written permission of the copyright ownerRelative navigation is paramount in space missions that involve rendezvousing between two spacecraft. It demands accurate and continuous estimation of the six degree-of-freedom relative pose, as this stage involves close-proximity-fast-reaction operations that can last up to five orbits. This has been routinely achieved thanks to active sensors such as lidar, but their large size, cost, power and limited operational range remain a stumbling block for en masse on-board integration. With the onset of faster processing units, lighter and cheaper passive optical sensors are emerging as the suitable alternative for autonomous rendezvous in combination with computer vision algorithms. Current vision-based solutions, however, are limited by adverse illumination conditions such as solar glare, shadowing, and eclipse. These effects are exacerbated when the target does not hold cooperative markers to accommodate the estimation process and is incapable of controlling its rotational state. This thesis explores novel model-based methods that exploit sequences of monoc ular images acquired by an on-board camera to accurately carry out spacecraft relative pose estimation for non-cooperative close-range rendezvous with a known artificial target. The proposed solutions tackle the current challenges of imaging in the visible spectrum and investigate the contribution of the long wavelength infrared (or ā€œthermalā€) band towards a combined multimodal approach. As part of the research, a visible-thermal synthetic dataset of a rendezvous approach with the defunct satellite Envisat is generated from the ground up using a realistic orbital camera simulator. From the rendered trajectories, the performance of several state-of-the-art feature detectors and descriptors is first evaluated for both modalities in a tailored scenario for short and wide baseline image processing transforms. Multiple combinations, including the pairing of algorithms with their non-native counterparts, are tested. Computational runtimes are assessed in an embedded hardware board. From the insight gained, a method to estimate the pose on the visible band is derived from minimising geometric constraints between online local point and edge contour features matched to keyframes generated offline from a 3D model of the target. The combination of both feature types is demonstrated to achieve a pose solution for a tumbling target using a sparse set of training images, bypassing the need for hardware-accelerated real-time renderings of the model. The proposed algorithm is then augmented with an extended Kalman filter which processes each feature-induced minimisation output as individual pseudo measurements, fusing them to estimate the relative pose and velocity states at each time-step. Both the minimisation and filtering are established using Lie group formalisms, allowing for the covariance of the solution computed by the former to be automatically incorporated as measurement noise in the latter, providing an automatic weighing of each feature type directly related to the quality of the matches. The predicted states are then used to search for new feature matches in the subsequent time-step. Furthermore, a method to derive a coarse viewpoint estimate to initialise the nominal algorithm is developed based on probabilistic modelling of the targetā€™s shape. The robustness of the complete approach is demonstrated for several synthetic and laboratory test cases involving two types of target undergoing extreme illumination conditions. Lastly, an innovative deep learning-based framework is developed by processing the features extracted by a convolutional front-end with long short-term memory cells, thus proposing the first deep recurrent convolutional neural network for spacecraft pose estimation. The framework is used to compare the performance achieved by visible-only and multimodal input sequences, where the addition of the thermal band is shown to greatly improve the performance during sunlit sequences. Potential limitations of this modality are also identified, such as when the targetā€™s thermal signature is comparable to Earthā€™s during eclipse.PH

    Spacecraft/Rover Hybrids for the Exploration of Small Solar System Bodies

    Get PDF
    This study investigated a mission architecture that allows the systematic and affordable in-situ exploration of small solar system bodies, such as asteroids, comets, and Martian moons (Figure 1). The architecture relies on the novel concept of spacecraft/rover hybrids,which are surface mobility platforms capable of achieving large surface coverage (by attitude controlled hops, akin to spacecraft flight), fine mobility (by tumbling), and coarse instrument pointing (by changing orientation relative to the ground) in the low-gravity environments(micro-g to milli-g) of small bodies. The actuation of the hybrids relies on spinning three internal flywheels. Using a combination of torques, the three flywheel motors can produce a reaction torque in any orientation without additional moving parts. This mobility concept allows all subsystems to be packaged in one sealed enclosure and enables the platforms to be minimalistic. The hybrids would be deployed from a mother spacecraft, which would act as a communication relay to Earth and would aid the in-situ assets with tasks such as localization and navigation (Figure 1). The hybrids are expected to be more capable and affordable than wheeled or legged rovers, due to their multiple modes of mobility (both hopping and tumbling), and have simpler environmental sealing and thermal management (since all components are sealed in one enclosure, assuming non-deployable science instruments). In summary, this NIAC Phase II study has significantly increased the TRL (Technology Readiness Level) of the mobility and autonomy subsystems of spacecraft/rover hybrids, and characterized system engineering aspects in the context of a reference mission to Phobos. Future studies should focus on improving the robustness of the autonomy module and further refine system engineering aspects, in view of opportunities for technology infusion
    • ā€¦
    corecore