Advanced perception, navigation and planning for autonomous in-water ship hull inspection

Inspection of ship hulls and marine structures using autonomous underwater vehicles has emerged as a unique and challenging application of robotics. The problem poses rich questions in physical design and operation, perception and navigation, and planning, driven by difficulties arising from the acoustic environment, poor water quality and the highly complex structures to be inspected. In this paper, we develop and apply algorithms for the central navigation and planning problems on ship hulls. These divide into two classes, suitable for the open, forward parts of a typical monohull, and for the complex areas around the shafting, propellers and rudders. On the open hull, we have integrated acoustic and visual mapping processes to achieve closed-loop control relative to features such as weld-lines and biofouling. In the complex area, we implemented new large-scale planning routines so as to achieve full imaging coverage of all the structures, at a high resolution. We demonstrate our approaches in recent operations on naval ships.United States. Office of Naval Research (Grant N00014-06-10043)United States. Office of Naval Research (Grant N00014-07-1-0791

Advances in Simultaneous Localization and Mapping in Confined Underwater Environments Using Sonar and Optical Imaging.

This thesis reports on the incorporation of surface information into a probabilistic simultaneous localization and mapping (SLAM) framework used on an autonomous underwater vehicle (AUV) designed for underwater inspection. AUVs operating in cluttered underwater environments, such as ship hulls or dams, are commonly equipped with Doppler-based sensors, which---in addition to navigation---provide a sparse representation of the environment in the form of a three-dimensional (3D) point cloud. The goal of this thesis is to develop perceptual algorithms that take full advantage of these sparse observations for correcting navigational drift and building a model of the environment. In particular, we focus on three objectives. First, we introduce a novel representation of this 3D point cloud as collections of planar features arranged in a factor graph. This factor graph representation probabalistically infers the spatial arrangement of each planar segment and can effectively model smooth surfaces (such as a ship hull). Second, we show how this technique can produce 3D models that serve as input to our pipeline that produces the first-ever 3D photomosaics using a two-dimensional (2D) imaging sonar. Finally, we propose a model-assisted bundle adjustment (BA) framework that allows for robust registration between surfaces observed from a Doppler sensor and visual features detected from optical images. Throughout this thesis, we show methods that produce 3D photomosaics using a combination of triangular meshes (derived from our SLAM framework or given a-priori), optical images, and sonar images. Overall, the contributions of this thesis greatly increase the accuracy, reliability, and utility of in-water ship hull inspection with AUVs despite the challenges they face in underwater environments. We provide results using the Hovering Autonomous Underwater Vehicle (HAUV) for autonomous ship hull inspection, which serves as the primary testbed for the algorithms presented in this thesis. The sensor payload of the HAUV consists primarily of: a Doppler velocity log (DVL) for underwater navigation and ranging, monocular and stereo cameras, and---for some applications---an imaging sonar.PhDElectrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120750/1/paulozog_1.pd

Enhanced fish bending model for automatic tuna sizing using computer vision

[EN] This paper presents a non-invasive fully automatic procedure to obtain highly accurate fish length estimation in adult Bluefin Tuna, based on a stereoscopic vision system and a deformable model of the fish ventral silhouette. The present work takes a geometric tuna model, which was previously developed by the same authors to discriminate fish in 2D images, and proposes new models to enhance the capabilities of the automatic procedure, from fish discrimination to accurate 3D length estimation. Fish length information is an important indicator of the health of wild fish stocks and for predicting biomass using length-weight relations. The proposal pays special attention to parts of the fish silhouette that have special relevance for accurate length estimation. The models have been designed to best fit the rear part of the fish, in particular the caudal peduncle, and a width parameter has been added to better fit the silhouette. Moreover, algorithms have been developed to extract snout tip and caudal peduncle features, allowing better initialization of model parameters. Snout Fork Length (SFL) measurements using the different models are extracted from images recorded with a stereoscopic vision system in a sea cage containing 312 adult Atlantic Bluefin Tuna. The automatic measurements are compared with two ground truths: one configured with semiautomatic measurements of favourable selected samples and one with real SFL measurements of the tuna stock collected at harvesting. Comparison with the semiautomatic measurements demonstrates that the combination of improved geometric models and feature extraction algorithms delivers good results in terms of fish length estimation error (up to 90% of the samples bounded in a 3% error margin) and number of automatic measurements (up to 950 samples out of 1000). When compared with real SFL measurements of the tuna stock, the system provides a high number of automatic detections (up to 6706 in a video of 135¿min duration, i.e., 50 automatic measurements per minute of recording) and highly accurate length measurements, obtaining no statistically significant difference between automatic and real SFL frequency distributions. This procedure could be extended to other species to assess the size distribution of stocks, as discussed in the paper.This work was supported by funding from ACUSTUNA project ref. CTM2015-70446-R (MINECO/ERDF, EU). This project has been possible thanks to the collaboration of IEO (Spanish Oceanographic Institute). We acknowledge the assistance provided by the Spanish company Grup Balfego S.L. in supplying boats and divers to acquire underwater video in the Mediterranean Sea.Muñoz-Benavent, P.; Andreu García, G.; Valiente González, JM.; Atienza-Vanacloig, V.; Puig Pons, V.; Espinosa Roselló, V. (2018). Enhanced fish bending model for automatic tuna sizing using computer vision. Computers and Electronics in Agriculture. 150:52-61. https://doi.org/10.1016/j.compag.2018.04.005S526115

Internet of Underwater Things and Big Marine Data Analytics -- A Comprehensive Survey

The Internet of Underwater Things (IoUT) is an emerging communication ecosystem developed for connecting underwater objects in maritime and underwater environments. The IoUT technology is intricately linked with intelligent boats and ships, smart shores and oceans, automatic marine transportations, positioning and navigation, underwater exploration, disaster prediction and prevention, as well as with intelligent monitoring and security. The IoUT has an influence at various scales ranging from a small scientific observatory, to a midsized harbor, and to covering global oceanic trade. The network architecture of IoUT is intrinsically heterogeneous and should be sufficiently resilient to operate in harsh environments. This creates major challenges in terms of underwater communications, whilst relying on limited energy resources. Additionally, the volume, velocity, and variety of data produced by sensors, hydrophones, and cameras in IoUT is enormous, giving rise to the concept of Big Marine Data (BMD), which has its own processing challenges. Hence, conventional data processing techniques will falter, and bespoke Machine Learning (ML) solutions have to be employed for automatically learning the specific BMD behavior and features facilitating knowledge extraction and decision support. The motivation of this paper is to comprehensively survey the IoUT, BMD, and their synthesis. It also aims for exploring the nexus of BMD with ML. We set out from underwater data collection and then discuss the family of IoUT data communication techniques with an emphasis on the state-of-the-art research challenges. We then review the suite of ML solutions suitable for BMD handling and analytics. We treat the subject deductively from an educational perspective, critically appraising the material surveyed.Comment: 54 pages, 11 figures, 19 tables, IEEE Communications Surveys & Tutorials, peer-reviewed academic journa

Toward autonomous underwater mapping in partially structured 3D environments

Submitted in partial fulfillment of the requirements for the degree of Master of Science at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution February 2014Motivated by inspection of complex underwater environments, we have developed a system for multi-sensor SLAM utilizing both structured and unstructured environmental features. We present a system for deriving planar constraints from sonar data, and jointly optimizing the vehicle and plane positions as nodes in a factor graph. We also present a system for outlier rejection and smoothing of 3D sonar data, and for generating loop closure constraints based on the alignment of smoothed submaps. Our factor graph SLAM backend combines loop closure constraints from sonar data with detections of visual fiducial markers from camera imagery, and produces an online estimate of the full vehicle trajectory and landmark positions. We evaluate our technique on an inspection of a decomissioned aircraft carrier, as well as synthetic data and controlled indoor experiments, demonstrating improved trajectory estimates and reduced reprojection error in the final 3D map

Automated mapping of oblique imagery collected with unmanned vehicles in coastal and marine environments

Recent technological advances in unmanned observational platforms, including remotely operated vehicles (ROVs) and small unmanned aerial systems (sUAS), have made them highly effective tools for research and monitoring within marine and coastal environments. One of the primary types of data collected by these systems is video imagery, which is often captured at an angle oblique to the Earth’s surface, rather than normal to it (e.g., downward looking). This thesis presents a newly developed suite of tools designed to digitally map oblique imagery data collected with ROV and sUAS in coastal and marine environments and quantitatively evaluates the accuracy of the resultant maps. Results indicate that maps generated from oblique imagery collected with unmanned vehicles have highly variable accuracy relative to maps generated with imagery data collected with conventional mapping platforms. These results suggest that resultant maps have the potential to match or even surpass the accuracy of maps generated with imagery data collected with conventional mapping platforms but realizing that potential is largely dependent upon careful survey design

Toward autonomous harbor surveillance

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2010.Includes bibliographical references (p. 105-113).In this thesis we address the problem of drift-free navigation for underwater vehicles performing harbor surveillance and ship hull inspection. Maintaining accurate localization for the duration of a mission is important for a variety of tasks, such as planning the vehicle trajectory and ensuring coverage of the area to be inspected. Our approach uses only onboard sensors in a simultaneous localization and mapping setting and removes the need for any external infrastructure like acoustic beacons. We extract dense features from a forward-looking imaging sonar and apply pair-wise registration between sonar frames. The registrations are combined with onboard velocity, attitude and acceleration sensors to obtain an improved estimate of the vehicle trajectory. In addition, an architecture for a persistent mapping is proposed. With the intention of handling long term operations and repetitive surveillance tasks. The proposed architecture is flexible and supports different types of vehicles and mapping methods. The design of the system is demonstrated with an implementation of some of the key features of the system. In addition, methods for re-localization are considered. Finally, results from several experiments that demonstrate drift-free navigation in various underwater environments are presented.by Hordur Johannsson.S.M