Multimodal learning from visual and remotely sensed data

Autonomous vehicles are often deployed to perform exploration and monitoring missions in unseen environments. In such applications, there is often a compromise between the information richness and the acquisition cost of different sensor modalities. Visual data is usually very information-rich, but requires in-situ acquisition with the robot. In contrast, remotely sensed data has a larger range and footprint, and may be available prior to a mission. In order to effectively and efficiently explore and monitor the environment, it is critical to make use of all of the sensory information available to the robot. One important application is the use of an Autonomous Underwater Vehicle (AUV) to survey the ocean floor. AUVs can take high resolution in-situ photographs of the sea floor, which can be used to classify different regions into various habitat classes that summarise the observed physical and biological properties. This is known as benthic habitat mapping. However, since AUVs can only image a tiny fraction of the ocean floor, habitat mapping is usually performed with remotely sensed bathymetry (ocean depth) data, obtained from shipborne multibeam sonar. With the recent surge in unsupervised feature learning and deep learning techniques, a number of previous techniques have investigated the concept of multimodal learning: capturing the relationship between different sensor modalities in order to perform classification and other inference tasks. This thesis proposes related techniques for visual and remotely sensed data, applied to the task of autonomous exploration and monitoring with an AUV. Doing so enables more accurate classification of the benthic environment, and also assists autonomous survey planning. The first contribution of this thesis is to apply unsupervised feature learning techniques to marine data. The proposed techniques are used to extract features from image and bathymetric data separately, and the performance is compared to that with more traditionally used features for each sensor modality. The second contribution is the development of a multimodal learning architecture that captures the relationship between the two modalities. The model is robust to missing modalities, which means it can extract better features for large-scale benthic habitat mapping, where only bathymetry is available. The model is used to perform classification with various combinations of modalities, demonstrating that multimodal learning provides a large performance improvement over the baseline case. The third contribution is an extension of the standard learning architecture using a gated feature learning model, which enables the model to better capture the ‘one-to-many’ relationship between visual and bathymetric data. This opens up further inference capabilities, with the ability to predict visual features from bathymetric data, which allows image-based queries. Such queries are useful for AUV survey planning, especially when supervised labels are unavailable. The final contribution is the novel derivation of a number of information-theoretic measures to aid survey planning. The proposed measures predict the utility of unobserved areas, in terms of the amount of expected additional visual information. As such, they are able to produce utility maps over a large region that can be used by the AUV to determine the most informative locations from a set of candidate missions. The models proposed in this thesis are validated through extensive experiments on real marine data. Furthermore, the introduced techniques have applications in various other areas within robotics. As such, this thesis concludes with a discussion on the broader implications of these contributions, and the future research directions that arise as a result of this work

Combining Exploration and Exploitation in Active Learning

This thesis investigates the active learning in the presence of model bias. State of the art approaches advocate combining exploration and exploitation in active learning. However, they suffer from premature exploitation or unnecessary exploration in the later stages of learning. We propose to combine exploration and exploitation in active learning by discarding instances outside a sampling window that is centered around the estimated decision boundary and uniformly draw sample from this window. Initially the window spans the entire feature space and is gradually constricted, where the rate of constriction models the exploration-exploitation tradeoff. The desired effect of this approach (CExp) is that we get an increasing sampling density in informative regions as active learning progresses, resulting in a continuous and natural transition from exploration to exploitation, limiting both premature exploitation and unnecessary exploration. We show that our approach outperforms state of the art on real world multiclass datasets. We also extend our approach to spatial mapping problems where the standard active learning assumption of uniform costs is violated. We show that we can take advantage of \emph{spatial continuity} in the data by geographically partitioning the instances in the sampling window and choosing a single partition (region) for sampling, as opposed to taking a random sample from the entire window, resulting in a novel spatial active learning algorithm that combines exploration and exploitation. We demonstrate that our approach (CExp-Spatial) can generate cost-effective sampling trajectories over baseline sampling methods. Finally, we present the real world problem of mapping benthic habitats where bathymetry derived features are typically not strong enough to discriminate the fine details between classes identified from high-resolution imagery, increasing the possiblity of model bias in active learning. We demonstrate, under such conditions, that CExp outperforms state of the art and that CExp-Spatial can generate more cost-effective sampling trajectories for an Autonomous Underwater Vehicle in contrast to baseline sampling strategies

Combining Exploration and Exploitation in Active Learning

This thesis investigates the active learning in the presence of model bias. State of the art approaches advocate combining exploration and exploitation in active learning. However, they suffer from premature exploitation or unnecessary exploration in the later stages of learning. We propose to combine exploration and exploitation in active learning by discarding instances outside a sampling window that is centered around the estimated decision boundary and uniformly draw sample from this window. Initially the window spans the entire feature space and is gradually constricted, where the rate of constriction models the exploration-exploitation tradeoff. The desired effect of this approach (CExp) is that we get an increasing sampling density in informative regions as active learning progresses, resulting in a continuous and natural transition from exploration to exploitation, limiting both premature exploitation and unnecessary exploration. We show that our approach outperforms state of the art on real world multiclass datasets. We also extend our approach to spatial mapping problems where the standard active learning assumption of uniform costs is violated. We show that we can take advantage of \emph{spatial continuity} in the data by geographically partitioning the instances in the sampling window and choosing a single partition (region) for sampling, as opposed to taking a random sample from the entire window, resulting in a novel spatial active learning algorithm that combines exploration and exploitation. We demonstrate that our approach (CExp-Spatial) can generate cost-effective sampling trajectories over baseline sampling methods. Finally, we present the real world problem of mapping benthic habitats where bathymetry derived features are typically not strong enough to discriminate the fine details between classes identified from high-resolution imagery, increasing the possiblity of model bias in active learning. We demonstrate, under such conditions, that CExp outperforms state of the art and that CExp-Spatial can generate more cost-effective sampling trajectories for an Autonomous Underwater Vehicle in contrast to baseline sampling strategies

A revised ocean glider concept to realize Stommel's vision and supplement Argo floats

This paper revisits Stommel's vision for a global glider network and the Argo design specification. A concept of floats with wings, so-called slow underwater gliders, is explored. An analysis of the energy or power consumption shows that, by operating gliders with half the vehicle volume at half the speed compared to present gliders, the energy requirements for long-duration missions can be met with available battery capacities. Simulation experiments of slow gliders are conducted using the horizontal current fields from an eddy-permitting ocean reanalysis product. By employing a semi-Lagrangian, streamwise navigation whereby the glider steers at right angles to ocean currents, we show that the concept is feasible. The simulated glider tracks demonstrate the potential for efficient coverage of key oceanographic features and variability.publishedVersio

Underwater Exploration and Mapping

This paper analyzes the open challenges of exploring and mapping in the underwater realm with the goal of identifying research opportunities that will enable an Autonomous Underwater Vehicle (AUV) to robustly explore different environments. A taxonomy of environments based on their 3D structure is presented together with an analysis on how that influences the camera placement. The difference between exploration and coverage is presented and how they dictate different motion strategies. Loop closure, while critical for the accuracy of the resulting map, proves to be particularly challenging due to the limited field of view and the sensitivity to viewing direction. Experimental results of enforcing loop closures in underwater caves demonstrate a novel navigation strategy. Dense 3D mapping, both online and offline, as well as other sensor configurations are discussed following the presented taxonomy. Experimental results from field trials illustrate the above analysis.acceptedVersio

Adaptive Sampling For Efficient Online Modelling

This thesis examines methods enabling autonomous systems to make active sampling and planning decisions in real time. Gaussian Process (GP) regression is chosen as a framework for its non-parametric approach allowing flexibility in unknown environments. The first part of the thesis focuses on depth constrained full coverage bathymetric surveys in unknown environments. Algorithms are developed to find and follow a depth contour, modelled with a GP, and produce a depth constrained boundary. An extension to the Boustrophedon Cellular Decomposition, Discrete Monotone Polygonal Partitioning is developed allowing efficient planning for coverage within this boundary. Efficient computational methods such as incremental Cholesky updates are implemented to allow online Hyper Parameter optimisation and fitting of the GP's. This is demonstrated in simulation and the field on a platform built for the purpose. The second part of this thesis focuses on modelling the surface salinity profiles of estuarine tidal fronts. The standard GP model assumes evenly distributed noise, which does not always hold. This can be handled with Heteroscedastic noise. An efficient new method, Parametric Heteroscedastic Gaussian Process regression, is proposed. This is applied to active sample selection on stationary fronts and adaptive planning on moving fronts where a number of information theoretic methods are compared. The use of a mean function is shown to increase the accuracy of predictions whilst reducing optimisation time. These algorithms are validated in simulation. Algorithmic development is focused on efficient methods allowing deployment on platforms with constrained computational resources. Whilst the application of this thesis is Autonomous Surface Vessels, it is hoped the issues discussed and solutions provided have relevance to other applications in robotics and wider fields such as spatial statistics and machine learning in general

Towards basin-scale in-situ characterization of sea-ice using an Autonomous Underwater Glider

Submitted in partial fulfillment of the requirements for the degree of Master of Science in Mechanical Engineering at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution September 2020.This thesis presents an Autonomous Underwater Glider (AUG) architecture that is intended for basin-scale unattended survey of Arctic sea-ice. The distinguishing challenge for AUG operations in the Arctic environment is the presence of year-round sea-ice cover which prevents vehicle surfacing for localization updates and shore-side communication. Due to the high cost of operating support vessels in the Arctic, the proposed AUG architecture minimizes external infrastructure requirements to brief and infrequent satellite updates on the order of once per day. This is possible by employing onboard acoustic sensing for sea-ice observation and navigation, along with intelligent management of onboard resources. To enable unattended survey of Arctic sea-ice with an AUG, this thesis proposes a hierarchical acoustics-based sea-ice characterization scheme to perform science data collection and assess environment risk, a multi-factor terrain-aided navigation method that leverages bathymetric features and active ocean current sensing to limit localization error, and a set of energy-optimal propulsive and hotel policies that react to evolving environmental conditions to improve AUG endurance. These methods are evaluated with respect to laboratory experiments and preliminary field data, and future Arctic sea-ice survey mission concepts are discussed.Support for this research was provided through the National Science Foundation Navigating the New Arctic Grant #1839063 and the NASA PSTAR Grant #NNX16AL08G. Additionally, this research was supported by the Walter A. Rosenblith Presidential Fellowship

Autonomous Exploration of Large-Scale Natural Environments

This thesis addresses issues which arise when using robotic platforms to explore large-scale, natural environments. Two main problems are identified: the volume of data collected by autonomous platforms and the complexity of planning surveys in large environments. Autonomous platforms are able to rapidly accumulate large data sets. The volume of data that must be processed is often too large for human experts to analyse exhaustively in a practical amount of time or in a cost-effective manner. This burden can create a bottleneck in the process of converting observations into scientifically relevant data. Although autonomous platforms can collect precisely navigated, high-resolution data, they are typically limited by finite battery capacities, data storage and computational resources. Deployments are also limited by project budgets and time frames. These constraints make it impractical to sample large environments exhaustively. To use the limited resources effectively, trajectories which maximise the amount of information gathered from the environment must be designed. This thesis addresses these problems. Three primary contributions are presented: a new classifier designed to accept probabilistic training targets rather than discrete training targets; a semi-autonomous pipeline for creating models of the environment; and an offline method for autonomously planning surveys. These contributions allow large data sets to be processed with minimal human intervention and promote efficient allocation of resources. In this thesis environmental models are established by learning the correlation between data extracted from a digital elevation model (DEM) of the seafloor and habitat categories derived from in-situ images. The DEM of the seafloor is collected using ship-borne multibeam sonar and the in-situ images are collected using an autonomous underwater vehicle (AUV). While the thesis specifically focuses on mapping and exploring marine habitats with an AUV, the research applies equally to other applications such as aerial and terrestrial environmental monitoring and planetary exploration