Visual localization in challenging environments

Visual localization, the method of self-localization based on camera images, has established as an additional, GNSS-free technology that is investigated in increasingly real and challenging applications. Particularly demanding is the self-localization of first responders in unstructured and unknown environments, for which visual localization can substantially contribute to increase the situational awareness and safety of first responders. Challenges arise from the operation under adverse conditions on computationally restricted platforms in the presence of dynamic objects. Current solutions are quickly pushed to their limits and the development of more robust approaches is of high demand. This thesis investigates the application of visual localization in dynamic, adverse environments to identify challenges and accordingly to increase the robustness, on the example of a dedicated visual-inertial navigation system. The methodical contributions of this work relate to the introduction of semantic understanding, improvements in error propagation and the development of a digital twin. The geometric visual odometry component is extended to a hybrid approach that includes a deep neural network for semantic segmentation to ignore distracting image areas of certain object classes. A Sensor-AI approach complements this method by directly training the network to segment image areas that are critical for the considered visual odometry system. Another improvement results from analyses and modifications of the existing error propagation in visual odometry. Furthermore, a digital twin is presented that closely replicates geometric and radiometric properties of the real sensor system in simulation in order to multiply experimental possibilities. The experiments are based on datasets from inspections that are used to motivate three first responder scenarios, namely indoor rescue, flood disaster and wildfire. The datasets were recorded in corridor, mall, coast, river and fumarole environments and aim to analyze the influence of the dynamic elements person, water and smoke. Each investigation starts with extensive in-depth analyses in simulation based on created synthetic video clones of the respective dynamic environments. Specifically, a combined sensitivity analysis allows to jointly consider environment, system design, sensor property and calibration error parameters to account for adverse conditions. All investigations are verified with experiments based on the real system. The results show the susceptibility of geometric approaches to dynamic objects in challenging scenarios. The introduction of the segmentation aid within the hybrid system contributes well in terms of robustness by preventing significant failures, but understandably it cannot compensate for a lack of visible static backgrounds. As a consequence, future visual localization systems require both the ability of semantic understanding and its integration into a complementary multi-sensor system

GPGM-SLAM: a Robust SLAM System for Unstructured Planetary Environments with Gaussian Process Gradient Maps

Simultaneous Localization and Mapping (SLAM) techniques play a key role towards long-term autonomy of mobile robots due to the ability to correct localization errors and produce consistent maps of an environment over time. Contrarily to urban or man-made environments, where the presence of unique objects and structures offer unique cues for localization, the apperance of unstructured natural environments is often ambiguous and self-similar, hindering the performances of loop closure detection. In this paper, we present an approach to improve the robustness of place recognition in the context of a submap-based stereo SLAM based on Gaussian Process Gradient Maps (GPGMaps). GPGMaps embed a continuous representation of the gradients of the local terrain elevation by means of Gaussian Process regression and Structured Kernel Interpolation, given solely noisy elevation measurements. We leverage the imagelike structure of GPGMaps to detect loop closures using traditional visual features and Bag of Words. GPGMap matching is performed as an SE(2) alignment to establish loop closure constraints within a pose graph. We evaluate the proposed pipeline on a variety of datasets recorded on Mt. Etna, Sicily and in the Morocco desert, respectively Moon- and Mars-like environments, and we compare the localization performances with state-of-the-art approaches for visual SLAM and visual loop closure detection

Collaborative Localization and Mapping for Autonomous Planetary Exploration : Distributed Stereo Vision-Based 6D SLAM in GNSS-Denied Environments

Mobile robots are a crucial element of present and future scientific missions to explore the surfaces of foreign celestial bodies such as Moon and Mars. The deployment of teams of robots allows to improve efficiency and robustness in such challenging environments. As long communication round-trip times to Earth render the teleoperation of robotic systems inefficient to impossible, on-board autonomy is a key to success. The robots operate in Global Navigation Satellite System (GNSS)-denied environments and thus have to rely on space-suitable on-board sensors such as stereo camera systems. They need to be able to localize themselves online, to model their surroundings, as well as to share information about the environment and their position therein. These capabilities constitute the basis for the local autonomy of each system as well as for any coordinated joint action within the team, such as collaborative autonomous exploration. In this thesis, we present a novel approach for stereo vision-based on-board and online Simultaneous Localization and Mapping (SLAM) for multi-robot teams given the challenges imposed by planetary exploration missions. We combine distributed local and decentralized global estimation methods to get the best of both worlds: A local reference filter on each robot provides real-time local state estimates required for robot control and fast reactive behaviors. We designed a novel graph topology to incorporate these state estimates into an online incremental graph optimization to compute global pose and map estimates that serve as input to higher-level autonomy functions. In order to model the 3D geometry of the environment, we generate dense 3D point cloud and probabilistic voxel-grid maps from noisy stereo data. We distribute the computational load and reduce the required communication bandwidth between robots by locally aggregating high-bandwidth vision data into partial maps that are then exchanged between robots and composed into global models of the environment. We developed methods for intra- and inter-robot map matching to recognize previously visited locations in semi- and unstructured environments based on their estimated local geometry, which is mostly invariant to light conditions as well as different sensors and viewpoints in heterogeneous multi-robot teams. A decoupling of observable and unobservable states in the local filter allows us to introduce a novel optimization: Enforcing all submaps to be gravity-aligned, we can reduce the dimensionality of the map matching from 6D to 4D. In addition to map matches, the robots use visual fiducial markers to detect each other. In this context, we present a novel method for modeling the errors of the loop closure transformations that are estimated from these detections. We demonstrate the robustness of our methods by integrating them on a total of five different ground-based and aerial mobile robots that were deployed in a total of 31 real-world experiments for quantitative evaluations in semi- and unstructured indoor and outdoor settings. In addition, we validated our SLAM framework through several different demonstrations at four public events in Moon and Mars-like environments. These include, among others, autonomous multi-robot exploration tests at a Moon-analogue site on top of the volcano Mt. Etna, Italy, as well as the collaborative mapping of a Mars-like environment with a heterogeneous robotic team of flying and driving robots in more than 35 public demonstration runs

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available

Mars Science Laboratory Mission and Science Investigation

Scheduled to land in August of 2012, the Mars Science Laboratory (MSL) Mission was initiated to explore the habitability of Mars. This includes both modern environments as well as ancient environments recorded by the stratigraphic rock record preserved at the Gale crater landing site. The Curiosity rover has a designed lifetime of at least one Mars year (∼23 months), and drive capability of at least 20 km. Curiosity’s science payload was specifically assembled to assess habitability and includes a gas chromatograph-mass spectrometer and gas analyzer that will search for organic carbon in rocks, regolith fines, and the atmosphere (SAM instrument); an x-ray diffractometer that will determine mineralogical diversity (CheMin instrument); focusable cameras that can image landscapes and rock/regolith textures in natural color (MAHLI, MARDI, and Mastcam instruments); an alpha-particle x-ray spectrometer for in situ determination of rock and soil chemistry (APXS instrument); a laser-induced breakdown spectrometer to remotely sense the chemical composition of rocks and minerals (ChemCam instrument); an active neutron spectrometer designed to search for water in rocks/regolith (DAN instrument); a weather station to measure modern-day environmental variables (REMS instrument); and a sensor designed for continuous monitoring of background solar and cosmic radiation (RAD instrument). The various payload elements will work together to detect and study potential sampling targets with remote and in situ measurements; to acquire samples of rock, soil, and atmosphere and analyze them in onboard analytical instruments; and to observe the environment around the rover. The 155-km diameter Gale crater was chosen as Curiosity’s field site based on several attributes: an interior mountain of ancient flat-lying strata extending almost 5 km above the elevation of the landing site; the lower few hundred meters of the mountain show a progression with relative age from clay-bearing to sulfate-bearing strata, separated by an unconformity from overlying likely anhydrous strata; the landing ellipse is characterized by a mixture of alluvial fan and high thermal inertia/high albedo stratified deposits; and a number of stratigraphically/geomorphically distinct fluvial features. Samples of the crater wall and rim rock, and more recent to currently active surface materials also may be studied. Gale has a well-defined regional context and strong evidence for a progression through multiple potentially habitable environments. These environments are represented by a stratigraphic record of extraordinary extent, and insure preservation of a rich record of the environmental history of early Mars. The interior mountain of Gale Crater has been informally designated at Mount Sharp, in honor of the pioneering planetary scientist Robert Sharp. The major subsystems of the MSL Project consist of a single rover (with science payload), a Multi-Mission Radioisotope Thermoelectric Generator, an Earth-Mars cruise stage, an entry, descent, and landing system, a launch vehicle, and the mission operations and ground data systems. The primary communication path for downlink is relay through the Mars Reconnaissance Orbiter. The primary path for uplink to the rover is Direct-from-Earth. The secondary paths for downlink are Direct-to-Earth and relay through the Mars Odyssey orbiter. Curiosity is a scaled version of the 6-wheel drive, 4-wheel steering, rocker bogie system from the Mars Exploration Rovers (MER) Spirit and Opportunity and the Mars Pathfinder Sojourner. Like Spirit and Opportunity, Curiosity offers three primary modes of navigation: blind-drive, visual odometry, and visual odometry with hazard avoidance. Creation of terrain maps based on HiRISE (High Resolution Imaging Science Experiment) and other remote sensing data were used to conduct simulated driving with Curiosity in these various modes, and allowed selection of the Gale crater landing site which requires climbing the base of a mountain to achieve its primary science goals. The Sample Acquisition, Processing, and Handling (SA/SPaH) subsystem is responsible for the acquisition of rock and soil samples from the Martian surface and the processing of these samples into fine particles that are then distributed to the analytical science instruments. The SA/SPaH subsystem is also responsible for the placement of the two contact instruments (APXS, MAHLI) on rock and soil targets. SA/SPaH consists of a robotic arm and turret-mounted devices on the end of the arm, which include a drill, brush, soil scoop, sample processing device, and the mechanical and electrical interfaces to the two contact science instruments. SA/SPaH also includes drill bit boxes, the organic check material, and an observation tray, which are all mounted on the front of the rover, and inlet cover mechanisms that are placed over the SAM and CheMin solid sample inlet tubes on the rover top deck

Visual control of multi-rotor UAVs

Recent miniaturization of computer hardware, MEMs sensors, and high energy density batteries have enabled highly capable mobile robots to become available at low cost. This has driven the rapid expansion of interest in multi-rotor unmanned aerial vehicles. Another area which has expanded simultaneously is small powerful computers, in the form of smartphones, which nearly always have a camera attached, many of which now contain a OpenCL compatible graphics processing units. By combining the results of those two developments a low-cost multi-rotor UAV can be produced with a low-power onboard computer capable of real-time computer vision. The system should also use general purpose computer vision software to facilitate a variety of experiments. To demonstrate this I have built a quadrotor UAV based on control hardware from the Pixhawk project, and paired it with an ARM based single board computer, similar those in high-end smartphones. The quadrotor weights 980 g and has a flight time of 10 minutes. The onboard computer capable of running a pose estimation algorithm above the 10 Hz requirement for stable visual control of a quadrotor. A feature tracking algorithm was developed for efficient pose estimation, which relaxed the requirement for outlier rejection during matching. Compared with a RANSAC- only algorithm the pose estimates were less variable with a Z-axis standard deviation 0.2 cm compared with 2.4 cm for RANSAC. Processing time per frame was also faster with tracking, with 95 % confidence that tracking would process the frame within 50 ms, while for RANSAC the 95 % confidence time was 73 ms. The onboard computer ran the algorithm with a total system load of less than 25 %. All computer vision software uses the OpenCV library for common computer vision algorithms, fulfilling the requirement for running general purpose software. The tracking algorithm was used to demonstrate the capability of the system by per- forming visual servoing of the quadrotor (after manual takeoff). Response to external perturbations was poor however, requiring manual intervention to avoid crashing. This was due to poor visual controller tuning, and to variations in image acquisition and attitude estimate timing due to using free running image acquisition. The system, and the tracking algorithm, serve as proof of concept that visual control of a quadrotor is possible using small low-power computers and general purpose computer vision software