Improving the energy efficiency of autonomous underwater vehicles by learning to model disturbances

Energy efficiency is one of the main challenges for long-term autonomy of AUVs (Autonomous Underwater Vehicles). We propose a novel approach for improving the energy efficiency of AUV controllers based on the ability to learn which external disturbances can safely be ignored. The proposed learning approach uses adaptive oscillators that are able to learn online the frequency, amplitude and phase of zero-mean periodic external disturbances. Such disturbances occur naturally in open water due to waves, currents, and gravity, but also can be caused by the dynamics and hydrodynamics of the AUV itself. We formulate the theoretical basis of the approach, and demonstrate its abilities on a number of input signals. Further experimental evaluation is conducted using a dynamic model of the Girona 500 AUV in simulation on two important underwater scenarios: hovering and trajectory tracking. The proposed approach shows significant energy-saving capabilities while at the same time maintaining high controller gains. The approach is generic and applicable not only for AUV control, but also for other type of control where periodic disturbances exist and could be accounted for by the controller. © 2013 IEEE

Intelligent and Secure Underwater Acoustic Communication Networks

Underwater acoustic (UWA) communication networks are promising techniques for medium- to long-range wireless information transfer in aquatic applications. The harsh and dynamic water environment poses grand challenges to the design of UWA networks. This dissertation leverages the advances in machine learning and signal processing to develop intelligent and secure UWA communication networks. Three research topics are studied: 1) reinforcement learning (RL)-based adaptive transmission in UWA channels; 2) reinforcement learning-based adaptive trajectory planning for autonomous underwater vehicles (AUVs) in under-ice environments; 3) signal alignment to secure underwater coordinated multipoint (CoMP) transmissions. First, a RL-based algorithm is developed for adaptive transmission in long-term operating UWA point-to-point communication systems. The UWA channel dynamics are learned and exploited to trade off energy consumption with information delivery latency. The adaptive transmission problem is formulated as a partially observable Markov decision process (POMDP) which is solved by a Monte Carlo sampling-based approach, and an expectation-maximization-type of algorithm is developed to recursively estimate the channel model parameters. The experimental data processing reveals that the proposed algorithm achieves a good balance between energy efficiency and information delivery latency. Secondly, an online learning-based algorithm is developed for adaptive trajectory planning of multiple AUVs in under-ice environments to reconstruct a water parameter field of interest. The field knowledge is learned online to guide the trajectories of AUVs for collection of informative water parameter samples in the near future. The trajectory planning problem is formulated as a Markov decision process (MDP) which is solved by an actor-critic algorithm, where the field knowledge is estimated online using the Gaussian process regression. The simulation results show that the proposed algorithm achieves the performance close to a benchmark method that assumes perfect field knowledge. Thirdly, the dissertation presents a signal alignment method to secure underwater CoMP transmissions of geographically distributed antenna elements (DAEs) against eavesdropping. Exploiting the low sound speed in water and the spatial diversity of DAEs, the signal alignment method is developed such that useful signals will collide at the eavesdropper while stay collision-free at the legitimate user. The signal alignment mechanism is formulated as a mixed integer and nonlinear optimization problem which is solved through a combination of the simulated annealing method and the linear programming. Taking the orthogonal frequency-division multiplexing (OFDM) as the modulation technique, simulation and emulated experimental results demonstrate that the proposed method significantly degrades the eavesdropper\u27s interception capability

Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

Low-level control of autonomous underwater vehicles (AUVs) has been extensively addressed by classical control techniques. However, the variable operating conditions and hostile environments faced by AUVs have driven researchers towards the formulation of adaptive control approaches. The reinforcement learning (RL) paradigm is a powerful framework which has been applied in different formulations of adaptive control strategies for AUVs. However, the limitations of RL approaches have lead towards the emergence of deep reinforcement learning which has become an attractive and promising framework for developing real adaptive control strategies to solve complex control problems for autonomous systems. However, most of the existing applications of deep RL use video images to train the decision making artificial agent but obtaining camera images only for an AUV control purpose could be costly in terms of energy consumption. Moreover, the rewards are not easily obtained directly from the video frames. In this work we develop a deep RL framework for adaptive control applications of AUVs based on an actor-critic goal-oriented deep RL architecture, which takes the available raw sensory information as input and as output the continuous control actions which are the low-level commands for the AUV's thrusters. Experiments on a real AUV demonstrate the applicability of the stated deep RL approach for an autonomous robot control problem.Fil: Carlucho, Ignacio. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; ArgentinaFil: de Paula, Mariano. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; ArgentinaFil: Wang, Sen. Heriot-Watt University; Reino UnidoFil: Petillot, Yvan. Heriot-Watt University; Reino UnidoFil: Acosta, Gerardo Gabriel. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; Argentin

Docking control of an autonomous underwater vehicle using reinforcement learning

To achieve persistent systems in the future, autonomous underwater vehicles (AUVs) will need to autonomously dock onto a charging station. Here, reinforcement learning strategies were applied for the first time to control the docking of an AUV onto a fixed platform in a simulation environment. Two reinforcement learning schemes were investigated: one with continuous state and action spaces, deep deterministic policy gradient (DDPG), and one with continuous state but discrete action spaces, deep Q network (DQN). For DQN, the discrete actions were selected as step changes in the control input signals. The performance of the reinforcement learning strategies was compared with classical and optimal control techniques. The control actions selected by DDPG suffer from chattering effects due to a hyperbolic tangent layer in the actor. Conversely, DQN presents the best compromise between short docking time and low control effort, whilst meeting the docking requirements. Whereas the reinforcement learning algorithms present a very high computational cost at training time, they are five orders of magnitude faster than optimal control at deployment time, thus enabling an on-line implementation. Therefore, reinforcement learning achieves a performance similar to optimal control at a much lower computational cost at deployment, whilst also presenting a more general framework

Oceanic Challenges to Technological Solutions : A Review of Autonomous Underwater Vehicle Path Technologies in Biomimicry, Control, Navigation, and Sensing

Autonomous Underwater Vehicles (AUVs) epitomize a revolutionary stride in underwater exploration, seamlessly assuming tasks once exclusive to manned vehicles. Their collaborative prowess within joint missions has inaugurated a new epoch of intricate applications in underwater domains. This study’s primary aim is to scrutinize recent technological advancements in AUVs and their role in navigating the complexities of underwater environments. Through a meticulous review of literature and empirical studies, this review synthesizes recent technological strides, spotlighting developments in biomimicry models, cutting-edge control systems, adaptive navigation algorithms, and pivotal sensor arrays crucial for exploring and mapping the ocean floor. The article meticulously delineates the profound impact of AUVs on underwater robotics, offering a comprehensive panorama of advancements and illustrating their far-reaching implications for underwater exploration and mapping. This review furnishes a holistic comprehension of the current landscape of AUV technology. This condensed overview furnishes a swift comparative analysis, aiding in discerning the focal points of each study while spotlighting gaps and intersections within the existing body of knowledge. It efficiently steers researchers toward complementary sources, enabling a focused examination and judicious allocation of time to the most pertinent studies. Furthermore, it functions as a blueprint for comprehensive studies within the AUV domain, pinpointing areas where amalgamating multiple sources would yield a more comprehensive understanding. By elucidating the purpose, employing a robust methodology, and anticipating comprehensive results, this study endeavors to serve as a cornerstone resource that not only encapsulates recent technological strides but also provides actionable insights and directions for advancing the field of underwater robotics.© 2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/fi=vertaisarvioitu|en=peerReviewed