Large-area visually augmented navigation for autonomous underwater vehicles

Submitted to the Joint Program in Applied Ocean Science & Engineering in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution June 2005This thesis describes a vision-based, large-area, simultaneous localization and mapping (SLAM) algorithm that respects the low-overlap imagery constraints typical of autonomous underwater vehicles (AUVs) while exploiting the inertial sensor information that is routinely available on such platforms. We adopt a systems-level approach exploiting the complementary aspects of inertial sensing and visual perception from a calibrated pose-instrumented platform. This systems-level strategy yields a robust solution to underwater imaging that overcomes many of the unique challenges of a marine environment (e.g., unstructured terrain, low-overlap imagery, moving light source). Our large-area SLAM algorithm recursively incorporates relative-pose constraints using a view-based representation that exploits exact sparsity in the Gaussian canonical form. This sparsity allows for efficient O(n) update complexity in the number of images composing the view-based map by utilizing recent multilevel relaxation techniques. We show that our algorithmic formulation is inherently sparse unlike other feature-based canonical SLAM algorithms, which impose sparseness via pruning approximations. In particular, we investigate the sparsification methodology employed by sparse extended information filters (SEIFs) and offer new insight as to why, and how, its approximation can lead to inconsistencies in the estimated state errors. Lastly, we present a novel algorithm for efficiently extracting consistent marginal covariances useful for data association from the information matrix. In summary, this thesis advances the current state-of-the-art in underwater visual navigation by demonstrating end-to-end automatic processing of the largest visually navigated dataset to date using data collected from a survey of the RMS Titanic (path length over 3 km and 3100 m2 of mapped area). This accomplishment embodies the summed contributions of this thesis to several current SLAM research issues including scalability, 6 degree of freedom motion, unstructured environments, and visual perception.This work was funded in part by the CenSSIS ERC of the National Science Foundation under grant EEC-9986821, in part by the Woods Hole Oceanographic Institution through a grant from the Penzance Foundation, and in part by a NDSEG Fellowship awarded through the Department of Defense

Videos in Context for Telecommunication and Spatial Browsing

The research presented in this thesis explores the use of videos embedded in panoramic imagery to transmit spatial and temporal information describing remote environments and their dynamics. Virtual environments (VEs) through which users can explore remote locations are rapidly emerging as a popular medium of presence and remote collaboration. However, capturing visual representation of locations to be used in VEs is usually a tedious process that requires either manual modelling of environments or the employment of specific hardware. Capturing environment dynamics is not straightforward either, and it is usually performed through specific tracking hardware. Similarly, browsing large unstructured video-collections with available tools is difficult, as the abundance of spatial and temporal information makes them hard to comprehend. At the same time, on a spectrum between 3D VEs and 2D images, panoramas lie in between, as they offer the same 2D images accessibility while preserving 3D virtual environments surrounding representation. For this reason, panoramas are an attractive basis for videoconferencing and browsing tools as they can relate several videos temporally and spatially. This research explores methods to acquire, fuse, render and stream data coming from heterogeneous cameras, with the help of panoramic imagery. Three distinct but interrelated questions are addressed. First, the thesis considers how spatially localised video can be used to increase the spatial information transmitted during video mediated communication, and if this improves quality of communication. Second, the research asks whether videos in panoramic context can be used to convey spatial and temporal information of a remote place and the dynamics within, and if this improves users' performance in tasks that require spatio-temporal thinking. Finally, the thesis considers whether there is an impact of display type on reasoning about events within videos in panoramic context. These research questions were investigated over three experiments, covering scenarios common to computer-supported cooperative work and video browsing. To support the investigation, two distinct video+context systems were developed. The first telecommunication experiment compared our videos in context interface with fully-panoramic video and conventional webcam video conferencing in an object placement scenario. The second experiment investigated the impact of videos in panoramic context on quality of spatio-temporal thinking during localization tasks. To support the experiment, a novel interface to video-collection in panoramic context was developed and compared with common video-browsing tools. The final experimental study investigated the impact of display type on reasoning about events. The study explored three adaptations of our video-collection interface to three display types. The overall conclusion is that videos in panoramic context offer a valid solution to spatio-temporal exploration of remote locations. Our approach presents a richer visual representation in terms of space and time than standard tools, showing that providing panoramic contexts to video collections makes spatio-temporal tasks easier. To this end, videos in context are suitable alternative to more difficult, and often expensive solutions. These findings are beneficial to many applications, including teleconferencing, virtual tourism and remote assistance

Characterising biodiversity patterns associated with marine oil and gas infrastructure using environmental DNA

Environmental DNA metabarcoding is under-utilised at marine infrastructure, which can develop extensive biotic communities. To optimise environmental DNA at these challenging environments, I trialled and applied novel collection methods and analysed biodiversity at eight oil and gas platforms in the Gulf of Thailand. This data was used to inform impacts of potential decommissioning. This research demonstrates the enormous potential of eDNA as an ecosystem-wide monitoring approach as well as an informative tool for marine management

Nearshore hydrodynamics and morphology derived from video imagery

Tese de doutoramento, Geologia (Geodinâmica Externa), Universidade de Lisboa, Faculdade de Ciências, 2018The coastal zone is the dynamic interface between the land and the ocean. Natural processes, including wave action, flooding and coastal erosion, often endanger human occupation and the use of the littoral. It is therefore essential to improve our understanding of the physical processes occurring at the coast, particularly those related with coastal morphodynamics. Due to the complexity of the coastal environment, littoral studies should be as comprehensive as possible, covering both hydrodynamic forcing and morphological response. However, conventional in-situ survey methods involve the use of instrumentation which, due to the logistical commitments, do not provide the required time-space scales. Remote sensing methods emerge in this context as an interesting alternative solution to yield simultaneous high temporal frequency and high spatial resolution observations of the nearshore processes. Among others, shore-based video remote sensing systems have been proved, over the last three decades, as a cost-efficient and high-quality tool to support coastal scientists and managers. Video monitoring installations offer excellent spatio-temporal resolutions, in combination with cost-efficient long-term data acquisition. This dissertation aims to present new conceptual models and video imagery tools to assess nearshore morphodynamics. This objective was accomplished through the development of a set of efficient computational tools to extract synoptic hydrodynamic and morphology information from video images. Data used in this work were acquired at five different study sites located worldwide. At three sites, video data were collected from dedicated video systems installed for scientific purpose. Two more additional video data sets were derived from the acquisition of online-streaming surfcams, which are camera infrastructures installed at the coast to provide remote visual information of sea state to surf users. A stand-alone set of algorithm was built to process and to geo-reference the acquired video sequence using already existing software. In addition, the automated processing is set to produce special images, namely Timex Variance and Timestack. A first video-based technique exploited the pixel intensity variation of Timestack images to characterize nearshore hydrodynamics. The standard deviation of pixel intensity was successfully related to the spatial distribution of wave transformation domains. Therefore, shoaling, surf and swash zones could be clearly identified in the nearshore profile covered by the image. This technique provides a new tool to study the nearshore dynamics, as the extent of wave domains can be related with distinctive morphodynamic behaviour. The method can be also directly applied to Variance images, hence it offers the possibility of extending such studies to the alongshore dimension. A second methodology developed in the scope of the present work exploited the use of pixel intensity average of Timestack images to estimate wave breaking height. Breakpoint locations and pixel intensity profiles were used to define the cross-shore breaking pattern length visible on a time-averaged image, here defined as the parameter. A first approach coupled to the available bathymetry to solve a simple conceptual model for finding breaker height. Wave breaking height estimates yield a Normalized Root Mean Square Error (NRMSE) of 14% when compared to numerical model results, for offshore wave heights ranging from 1.6 m to 3.5 m. A second approach proposed the relationship /24 to replace water depth parameter on the simplest wave height calculation formula, which multiplies water depth by the breaker index. The technique can be directly applied on Timex, therefore images from four different sites were used to test its validity, obtaining an NRMSE of about 22% for a wide range of wave heights. A third methodology aimed to investigate the possibility of combining two shorebased remote sensing techniques, 2D terrestrial LiDAR and video imagery to perform detailed beach intertidal topography. 2D LiDAR provided precise shoreline elevation along a cross-shore beach transect, while shoreline contour was detected on Timex images in the alongshore dimension. The dataset from both instruments were complemented to perform 3D beach intertidal topography mapping with a Root Mean Square Error (RMSE) of approximately 0.12 m. Finally, a method to assess nearshore bathymetry was developed. The method is based on a depth inversion technique, where wave celerity was estimated using wave trajectories visible on Timestacks. The procedure differentiates the waves in the shoaling and breaking zones and then estimates local depth from shallow or intermediate water equations. In the test case, bathymetry was mapped till a depth of 11 m with relative short time observations (5 hours), registering a RMSE of about 0.46 m when compared to ground truth data. The techniques herein developed allow to extract from video images some of the key drivers of nearshore morphodynamics, such as wave breaking height and wave period, as well as the main morphological features, namely subtidal bathymetry and intertidal beach topography. The combination of the methodologies presented in this thesis provides a comprehensive coverage of nearshore processes, enabling a synoptic representation of hydrodynamics and morphology. These methodologies may foster the implementation of new video-based operational systems and support the quasi-real time determination of coastal indicators and early warning systems for coastal hazards.Fundação para a Ciência e a Tecnologia (FCT), SFRH/BD/52558/201

Investigating the build-up of precedence effect using reflection masking

The auditory processing level involved in the build‐up of precedence [Freyman et al., J. Acoust. Soc. Am. 90, 874–884 (1991)] has been investigated here by employing reflection masked threshold (RMT) techniques. Given that RMT techniques are generally assumed to address lower levels of the auditory signal processing, such an approach represents a bottom‐up approach to the buildup of precedence. Three conditioner configurations measuring a possible buildup of reflection suppression were compared to the baseline RMT for four reflection delays ranging from 2.5–15 ms. No buildup of reflection suppression was observed for any of the conditioner configurations. Buildup of template (decrease in RMT for two of the conditioners), on the other hand, was found to be delay dependent. For five of six listeners, with reflection delay=2.5 and 15 ms, RMT decreased relative to the baseline. For 5‐ and 10‐ms delay, no change in threshold was observed. It is concluded that the low‐level auditory processing involved in RMT is not sufficient to realize a buildup of reflection suppression. This confirms suggestions that higher level processing is involved in PE buildup. The observed enhancement of reflection detection (RMT) may contribute to active suppression at higher processing levels

Exploration of the Southern California Borderland

E/V Nautilus cruise NA075 returned to the Southern California Continental Borderland, an area that remains largely unexplored. Part of the broader North America-Pacific plate boundary, this region extends ~300 km west of the San Andreas Fault and displays an unusually rugged physiography. During the cruise, the multibeam sonar mapped ~5,200 km2 of seafloor, and ROVs Hercules and Argus were deployed for 16 dives to explore geological and biological targets (Figure 1) and collect samples

\u3cem\u3eNautilus\u3c/em\u3e Sample 2016: New Techniques and Partnerships

In 2016, E/V Nautilus and the ROV Hercules collected 549 geological, biological, and water samples (2,022 subsamples) to characterize several US West Coast national marine sanctuaries, the Cascadia margin, and offshore southern California. Most samples are archived at partnering repositories: geological samples to the Marine Geological Samples Lab at the University of Rhode Island and biological samples to Harvard University’s Museum of Comparative Zoology. The national marine sanctuary samples were split between these repositories and the California Academy of Sciences. During this field season, we experimented with new sampling methods to improve exploration efficiency and robustness