UW-MARL: Multi-Agent Reinforcement Learning for Underwater Adaptive Sampling using Autonomous Vehicles

Near-real-time water-quality monitoring in uncertain environments such as rivers, lakes, and water reservoirs of different variables is critical to protect the aquatic life and to prevent further propagation of the potential pollution in the water. In order to measure the physical values in a region of interest, adaptive sampling is helpful as an energy- and time-efficient technique since an exhaustive search of an area is not feasible with a single vehicle. We propose an adaptive sampling algorithm using multiple autonomous vehicles, which are well-trained, as agents, in a Multi-Agent Reinforcement Learning (MARL) framework to make efficient sequence of decisions on the adaptive sampling procedure. The proposed solution is evaluated using experimental data, which is fed into a simulation framework. Experiments were conducted in the Raritan River, Somerset and in Carnegie Lake, Princeton, NJ during July 2019

A Distributed Adaptive Sampling Soluting using Autonomous Underwater Vehicles

To achieve efficient and cost-effective sensing coverage of the vast under-sampled 3D aquatic volume, intelligent adaptive sampling strategies involving a team of Autonomous Underwater Vehicles (AUVs) endowed with underwater wireless communication capabilities become essential. Given a 3D field of interest to sample, the AUVs should coordinate to take measurements using minimal resources (time or energy) in order to reconstruct the field at an onshore station with admissible error. A novel distributed adaptive sampling solution that can minimize the sampling cost (in terms of time or energy expenditure) is proposed along with underwater acoustic communication protocols that facilitate the coordination of the vehicles. The proposed solution operates in two distinct phases in which it employs random compressive sensing (Phase I) and adaptive sampling (Phase II). Phase I captures the spatial distribution of the field of interest while Phase II tracks the temporal variation of the same. A distributed framework for multi-vehicle adaptive sampling that facilitates the movement of data between AUVs and enables compute intensive adaptive sampling algorithms is proposed. Simulation results on real data traces show that the proposed adaptive sampling solution significantly outperforms existing solutions in terms of reconstruction accuracy and energy expenditure

Intelligent and Secure Underwater Acoustic Communication Networks

Underwater acoustic (UWA) communication networks are promising techniques for medium- to long-range wireless information transfer in aquatic applications. The harsh and dynamic water environment poses grand challenges to the design of UWA networks. This dissertation leverages the advances in machine learning and signal processing to develop intelligent and secure UWA communication networks. Three research topics are studied: 1) reinforcement learning (RL)-based adaptive transmission in UWA channels; 2) reinforcement learning-based adaptive trajectory planning for autonomous underwater vehicles (AUVs) in under-ice environments; 3) signal alignment to secure underwater coordinated multipoint (CoMP) transmissions. First, a RL-based algorithm is developed for adaptive transmission in long-term operating UWA point-to-point communication systems. The UWA channel dynamics are learned and exploited to trade off energy consumption with information delivery latency. The adaptive transmission problem is formulated as a partially observable Markov decision process (POMDP) which is solved by a Monte Carlo sampling-based approach, and an expectation-maximization-type of algorithm is developed to recursively estimate the channel model parameters. The experimental data processing reveals that the proposed algorithm achieves a good balance between energy efficiency and information delivery latency. Secondly, an online learning-based algorithm is developed for adaptive trajectory planning of multiple AUVs in under-ice environments to reconstruct a water parameter field of interest. The field knowledge is learned online to guide the trajectories of AUVs for collection of informative water parameter samples in the near future. The trajectory planning problem is formulated as a Markov decision process (MDP) which is solved by an actor-critic algorithm, where the field knowledge is estimated online using the Gaussian process regression. The simulation results show that the proposed algorithm achieves the performance close to a benchmark method that assumes perfect field knowledge. Thirdly, the dissertation presents a signal alignment method to secure underwater CoMP transmissions of geographically distributed antenna elements (DAEs) against eavesdropping. Exploiting the low sound speed in water and the spatial diversity of DAEs, the signal alignment method is developed such that useful signals will collide at the eavesdropper while stay collision-free at the legitimate user. The signal alignment mechanism is formulated as a mixed integer and nonlinear optimization problem which is solved through a combination of the simulated annealing method and the linear programming. Taking the orthogonal frequency-division multiplexing (OFDM) as the modulation technique, simulation and emulated experimental results demonstrate that the proposed method significantly degrades the eavesdropper\u27s interception capability

Framework for multi-vehicle adaptive sampling of jets and plumes in coastal zones

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 125-130).This thesis presents a framework for the sampling of thermal and effluent jets and plumes using multiple autonomous surface vehicles. The framework was developed with the goal of achieving rapid and accurate in-situ measurement and characterization of these features. The framework is presented as a collection of simulation, estimation and field tools for use within the Mission Oriented Operations Suite (MOOS) and a novel Acoustic Doppler Current Profiling system that is capable of reorientation and real-time feedback. Key features developed within MOOS include a multi-parameter model of thermal and effluent jet and plume fields, online parameter estimation and sensor fusion. Using these tools, a collaborative adaptive sampling strategy is implemented to efficiently sample an industrial jet and plume. The capabilities of this strategy are demonstrated in realistic mission simulations and in field trials using a fleet of autonomous kayaks equipped with environmental sensors.by Matthew Lee Gildner.S.M