Reinforcement learning-based multi-AUV adaptive trajectory planning for under-ice field estimation

This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center. We model the water parameter field of interest as a Gaussian process with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the access points relay the observed field samples from all the AUVs to the fusion center, which computes the posterior distribution of the field based on the Gaussian process regression and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to maximize a long-term reward that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning-based online learning algorithm is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters

Intelligent and Secure Underwater Acoustic Communication Networks

Underwater acoustic (UWA) communication networks are promising techniques for medium- to long-range wireless information transfer in aquatic applications. The harsh and dynamic water environment poses grand challenges to the design of UWA networks. This dissertation leverages the advances in machine learning and signal processing to develop intelligent and secure UWA communication networks. Three research topics are studied: 1) reinforcement learning (RL)-based adaptive transmission in UWA channels; 2) reinforcement learning-based adaptive trajectory planning for autonomous underwater vehicles (AUVs) in under-ice environments; 3) signal alignment to secure underwater coordinated multipoint (CoMP) transmissions. First, a RL-based algorithm is developed for adaptive transmission in long-term operating UWA point-to-point communication systems. The UWA channel dynamics are learned and exploited to trade off energy consumption with information delivery latency. The adaptive transmission problem is formulated as a partially observable Markov decision process (POMDP) which is solved by a Monte Carlo sampling-based approach, and an expectation-maximization-type of algorithm is developed to recursively estimate the channel model parameters. The experimental data processing reveals that the proposed algorithm achieves a good balance between energy efficiency and information delivery latency. Secondly, an online learning-based algorithm is developed for adaptive trajectory planning of multiple AUVs in under-ice environments to reconstruct a water parameter field of interest. The field knowledge is learned online to guide the trajectories of AUVs for collection of informative water parameter samples in the near future. The trajectory planning problem is formulated as a Markov decision process (MDP) which is solved by an actor-critic algorithm, where the field knowledge is estimated online using the Gaussian process regression. The simulation results show that the proposed algorithm achieves the performance close to a benchmark method that assumes perfect field knowledge. Thirdly, the dissertation presents a signal alignment method to secure underwater CoMP transmissions of geographically distributed antenna elements (DAEs) against eavesdropping. Exploiting the low sound speed in water and the spatial diversity of DAEs, the signal alignment method is developed such that useful signals will collide at the eavesdropper while stay collision-free at the legitimate user. The signal alignment mechanism is formulated as a mixed integer and nonlinear optimization problem which is solved through a combination of the simulated annealing method and the linear programming. Taking the orthogonal frequency-division multiplexing (OFDM) as the modulation technique, simulation and emulated experimental results demonstrate that the proposed method significantly degrades the eavesdropper\u27s interception capability

UW-MARL: Multi-Agent Reinforcement Learning for Underwater Adaptive Sampling using Autonomous Vehicles

Near-real-time water-quality monitoring in uncertain environments such as rivers, lakes, and water reservoirs of different variables is critical to protect the aquatic life and to prevent further propagation of the potential pollution in the water. In order to measure the physical values in a region of interest, adaptive sampling is helpful as an energy- and time-efficient technique since an exhaustive search of an area is not feasible with a single vehicle. We propose an adaptive sampling algorithm using multiple autonomous vehicles, which are well-trained, as agents, in a Multi-Agent Reinforcement Learning (MARL) framework to make efficient sequence of decisions on the adaptive sampling procedure. The proposed solution is evaluated using experimental data, which is fed into a simulation framework. Experiments were conducted in the Raritan River, Somerset and in Carnegie Lake, Princeton, NJ during July 2019

Risk analysis and decision making for autonomous underwater vehicles

Risk analysis for autonomous underwater vehicles (AUVs) is essential to enable AUVs to explore extreme and dynamic environments. This research aims to augment existing risk analysis methods for AUVs, and it proposes a suite of methods to quantify mission risks and to support the implementation of safety-based decision making strategies for AUVs in harsh marine environments. This research firstly provides a systematic review of past progress of risk analysis research for AUV operations. The review answers key questions including fundamental concepts and evolving methods in the domain of risk analysis for AUVs, and it highlights future research trends to bridge existing gaps. Based on the state-of-the-art research, a copula-based approach is proposed for predicting the risk of AUV loss in underwater environments. The developed copula Bayesian network (CBN) aims to handle non-linear dependencies among environmental variables and inherent technical failures for AUVs, and therefore achieve accurate risk estimation for vehicle loss given various environmental observations. Furthermore, path planning for AUVs is an effective decision making strategy for mitigating risks and ensuring safer routing. A further study presents an offboard risk-based path planning approach for AUVs, considering a challenging environment with oil spill scenarios incorporated. The proposed global Risk-A* planner combines a Bayesian-based risk model for probabilistic risk reasoning and an A*-based algorithm for path searching. However, global path planning designed for static environments cannot handle the unpredictable situations that may emerge, and real-time replanned solutions are required to account for dynamic environmental observations. Therefore, a hybrid risk-aware decision making strategy is investigated for AUVs to combine static global planning with dynamic local re-planning. A dynamic risk analysis model based on the system theoretic process analysis (STPA) and BN is applied for generating a real-time risk map in target mission areas. The dynamic window algorithm (DWA) serves for local path planning to avoid moving obstacles. The proposed hybrid risk-aware decisionmaking architecture is essential for the real-life implementation of AUVs, leading eventually to a real-time adaptive path planning process onboard the AUV

An energy-aware architecture : a practical implementation for autonomous underwater vehicles

Energy awareness, fault tolerance and performance estimation are important aspects for extending the autonomy levels of today’s autonomous vehicles. Those are related to the concepts of survivability and reliability, two important factors that often limit the trust of end users in conducting large-scale deployments of such vehicles. With the aim of preparing the way for persistent autonomous operations this work focuses its efforts on investigating those effects on underwater vehicles capable of long-term missions. A novel energy-aware architecture for autonomous underwater vehicles (AUVs) is presented. This, by monitoring at runtime the vehicle’s energy usage, is capable of detecting and mitigating failures in the propulsion subsystem, one of the most common sources of mission-time problems. Furthermore it estimates the vehicle’s performance when operating in unknown environments and in the presence of external disturbances. These capabilities are a great contribution for reducing the operational uncertainty that most underwater platforms face during their deployment. Using knowledge collected while conducting real missions the proposed architecture allows the optimisation of on-board resource usage. This improves the vehicle’s effectiveness when operating in unknown stochastic scenarios or when facing the problem of resource scarcity. The architecture has been implemented on a real vehicle, Nessie AUV, used for real sea experiments as part of multiple research projects. These gave the opportunity of evaluating the improvements of the proposed system when considering more complex autonomous tasks. Together with Nessie AUV, the commercial platform IVER3 AUV has been involved in the evaluating the feasibility of this approach. Results and operational experience, gathered both in real sea scenarios and in controlled environment experiments, are discussed in detail showing the benefits and the operational constraints of the introduced architecture, alongside suggestions for future research directions

Docking control of an autonomous underwater vehicle using reinforcement learning

To achieve persistent systems in the future, autonomous underwater vehicles (AUVs) will need to autonomously dock onto a charging station. Here, reinforcement learning strategies were applied for the first time to control the docking of an AUV onto a fixed platform in a simulation environment. Two reinforcement learning schemes were investigated: one with continuous state and action spaces, deep deterministic policy gradient (DDPG), and one with continuous state but discrete action spaces, deep Q network (DQN). For DQN, the discrete actions were selected as step changes in the control input signals. The performance of the reinforcement learning strategies was compared with classical and optimal control techniques. The control actions selected by DDPG suffer from chattering effects due to a hyperbolic tangent layer in the actor. Conversely, DQN presents the best compromise between short docking time and low control effort, whilst meeting the docking requirements. Whereas the reinforcement learning algorithms present a very high computational cost at training time, they are five orders of magnitude faster than optimal control at deployment time, thus enabling an on-line implementation. Therefore, reinforcement learning achieves a performance similar to optimal control at a much lower computational cost at deployment, whilst also presenting a more general framework

A future for intelligent autonomous ocean observing systems

Ocean scientists have dreamed of and recently started to realize an ocean observing revolution with autonomous observing platforms and sensors. Critical questions to be answered by such autonomous systems are where, when, and what to sample for optimal information, and how to optimally reach the sampling locations. Definitions, concepts, and progress towards answering these questions using quantitative predictions and fundamental principles are presented. Results in reachability and path planning, adaptive sampling, machine learning, and teaming machines with scientists are overviewed. The integrated use of differential equations and theory from varied disciplines is emphasized. The results provide an inference engine and knowledge base for expert autonomous observing systems. They are showcased using a set of recent at-sea campaigns and realistic simulations. Real-time experiments with identical autonomous underwater vehicles (AUVs) in the Buzzards Bay and Vineyard Sound region first show that our predicted time-optimal paths were faster than shortest distance paths. Deterministic and probabilistic reachability and path forecasts issued and validated for gliders and floats in the northern Arabian Sea are then presented. Novel Bayesian adaptive sampling for hypothesis testing and optimal learning are finally shown to forecast the observations most informative to estimate the accuracy of model formulations, the values of ecosystem parameters and dynamic fields, and the presence of Lagrangian Coherent Structures