Cooperative Navigation for Low-bandwidth Mobile Acoustic Networks.

This thesis reports on the design and validation of estimation and planning algorithms for underwater vehicle cooperative localization. While attitude and depth are easily instrumented with bounded-error, autonomous underwater vehicles (AUVs) have no internal sensor that directly observes XY position. The global positioning system (GPS) and other radio-based navigation techniques are not available because of the strong attenuation of electromagnetic signals in seawater. The navigation algorithms presented herein fuse local body-frame rate and attitude measurements with range observations between vehicles within a decentralized architecture. The acoustic communication channel is both unreliable and low bandwidth, precluding many state-of-the-art terrestrial cooperative navigation algorithms. We exploit the underlying structure of a post-process centralized estimator in order to derive two real-time decentralized estimation frameworks. First, the origin state method enables a client vehicle to exactly reproduce the corresponding centralized estimate within a server-to-client vehicle network. Second, a graph-based navigation framework produces an approximate reconstruction of the centralized estimate onboard each vehicle. Finally, we present a method to plan a locally optimal server path to localize a client vehicle along a desired nominal trajectory. The planning algorithm introduces a probabilistic channel model into prior Gaussian belief space planning frameworks. In summary, cooperative localization reduces XY position error growth within underwater vehicle networks. Moreover, these methods remove the reliance on static beacon networks, which do not scale to large vehicle networks and limit the range of operations. Each proposed localization algorithm was validated in full-scale AUV field trials. The planning framework was evaluated through numerical simulation.PhDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113428/1/jmwalls_1.pd

Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimization

Contemporary autopilot systems for unmanned aerial vehicles (UAVs) are far more limited in their flight envelope as compared to experienced human pilots, thereby restricting the conditions UAVs can operate in and the types of missions they can accomplish autonomously. This paper proposes a deep reinforcement learning (DRL) controller to handle the nonlinear attitude control problem, enabling extended flight envelopes for fixed-wing UAVs. A proof-of-concept controller using the proximal policy optimization (PPO) algorithm is developed, and is shown to be capable of stabilizing a fixed-wing UAV from a large set of initial conditions to reference roll, pitch and airspeed values. The training process is outlined and key factors for its progression rate are considered, with the most important factor found to be limiting the number of variables in the observation vector, and including values for several previous time steps for these variables. The trained reinforcement learning (RL) controller is compared to a proportional-integral-derivative (PID) controller, and is found to converge in more cases than the PID controller, with comparable performance. Furthermore, the RL controller is shown to generalize well to unseen disturbances in the form of wind and turbulence, even in severe disturbance conditions.Comment: 11 pages, 3 figures, 2019 International Conference on Unmanned Aircraft Systems (ICUAS

A COLLISION AVOIDANCE SYSTEM FOR AUTONOMOUS UNDERWATER VEHICLES

The work in this thesis is concerned with the development of a novel and practical collision avoidance system for autonomous underwater vehicles (AUVs). Synergistically, advanced stochastic motion planning methods, dynamics quantisation approaches, multivariable tracking controller designs, sonar data processing and workspace representation, are combined to enhance significantly the survivability of modern AUVs. The recent proliferation of autonomous AUV deployments for various missions such as seafloor surveying, scientific data gathering and mine hunting has demanded a substantial increase in vehicle autonomy. One matching requirement of such missions is to allow all the AUV to navigate safely in a dynamic and unstructured environment. Therefore, it is vital that a robust and effective collision avoidance system should be forthcoming in order to preserve the structural integrity of the vehicle whilst simultaneously increasing its autonomy. This thesis not only provides a holistic framework but also an arsenal of computational techniques in the design of a collision avoidance system for AUVs. The design of an obstacle avoidance system is first addressed. The core paradigm is the application of the Rapidly-exploring Random Tree (RRT) algorithm and the newly developed version for use as a motion planning tool. Later, this technique is merged with the Manoeuvre Automaton (MA) representation to address the inherent disadvantages of the RRT. A novel multi-node version which can also address time varying final state is suggested. Clearly, the reference trajectory generated by the aforementioned embedded planner must be tracked. Hence, the feasibility of employing the linear quadratic regulator (LQG) and the nonlinear kinematic based state-dependent Ricatti equation (SDRE) controller as trajectory trackers are explored. The obstacle detection module, which comprises of sonar processing and workspace representation submodules, is developed and tested on actual sonar data acquired in a sea-trial via a prototype forward looking sonar (AT500). The sonar processing techniques applied are fundamentally derived from the image processing perspective. Likewise, a novel occupancy grid using nonlinear function is proposed for the workspace representation of the AUV. Results are presented that demonstrate the ability of an AUV to navigate a complex environment. To the author's knowledge, it is the first time the above newly developed methodologies have been applied to an A UV collision avoidance system, and, therefore, it is considered that the work constitutes a contribution of knowledge in this area of work.J&S MARINE LT

Actor-critic reinforcement learning algorithms for yaw control of an Autonomous Underwater Vehicle

An Autonomous Underwater Vehicle (AUV) poses unique challenges that must be solved in order to achieve persistent autonomy. The requirement of persistent autonomy entails that a control solution must be capable of controlling a vehicle that is operating in an environment with complex non-linear dynamics and adapt to changes in those dynamics. In essence, artificial intelligence is required so that the vehicle can learn from its experience operating in the domain. In this thesis, reinforcement learning is the chosen machine learning mechanism. This learning paradigm is investigated by applying multiple actor-critic temporal difference learning algorithms to the yaw degree-of-freedom of a simulated model and the physical hardware of the Nessie VII AUV in a closed-loop feedback control problem. Additionally, results are also presented for path planning and path optimisation problems. These control problems are solved by modelling the AUV’s interaction with its environment as an optimal decision-making problem using a Markov Decision Process (MDP). Two novel actor-critic temporal difference learning algorithms called Linear True Online Continuous Learning Automation (Linear TOCLA) and Non-linear True Online Continuous Learning Automation (Non-linear TOCLA) are also presented and serve as new contributions to the reinforcement learning research community. These algorithms have been applied to the real Nessie vehicle and its simulated model. The proposed algorithms hold theoretical and practical advantages over previous state-of-the-art temporal difference learning algorithms. A new genetic algorithm is also presented and developed specifically for the optimisation of the continuous-valued reinforcement learning algorithms’. This genetic algorithm is used to find the optimal hyperparameters for four actor-critic algorithms in the well-known continuous-valued mountain car reinforcement learning benchmark problem. The results of this benchmark show that the Non-linear TOCLA algorithm achieves a similar performance to the state-of-the-art forward actor-critic algorithm it extends while significantly reducing the sensitivity of the hyperparameter selection. This reduction in hyperparameter sensitivity is shown using the distribution of optimal hyperparameters from ten separate optimisation runs. The actor learning rate of the forward actor-critic algorithm had a standard deviation of 0.00088, while the Non-linear TOCLA algorithm demonstrated a standard deviation of 0.00186. An even greater improvement is observed in the multi-step target weight, λ, which increased from a standard deviation of 0.036 for the forward actor-critic to 0.266 for the Non-linear TOCLA algorithm. All of the sourcecode used to generate the results in this thesis has been made available as open-source software.ARchaeological RObot systems for the Worlds Seas (ARROWS) EU FP7 project under grant agreement ID 30872

Path Planning and Control of UAV using Machine Learning and Deep Reinforcement Learning Techniques

Uncrewed Aerial Vehicles (UAVs) are playing an increasingly signifcant role in modern life. In the past decades, lots of commercial and scientifc communities all over the world have been developing autonomous techniques of UAV for a broad range of applications, such as forest fre monitoring, parcel delivery, disaster rescue, natural resource exploration, and surveillance. This brings a large number of opportunities and challenges for UAVs to improve their abilities in path planning, motion control and fault-tolerant control (FTC) directions. Meanwhile, due to the powerful decisionmaking, adaptive learning and pattern recognition capabilities of machine learning (ML) and deep reinforcement learning (DRL), the use of ML and DRL have been developing rapidly and obtain major achievement in a variety of applications. However, there is not many researches on the ML and DRl in the feld of motion control and real-time path planning of UAVs. This thesis focuses on the development of ML and DRL in the path planning, motion control and FTC of UAVs. A number of ontributions pertaining to the state space defnition, reward function design and training method improvement have been made in this thesis, which improve the effectiveness and efciency of applying DRL in UAV motion control problems. In addition to the control problems, this thesis also presents real-time path planning contributions, including relative state space defnition and human pedestrian inspired reward function, which provide a reliable and effective solution of the real-time path planning in a complex environment

Aeolus Ocean -- A simulation environment for the autonomous COLREG-compliant navigation of Unmanned Surface Vehicles using Deep Reinforcement Learning and Maritime Object Detection

Heading towards navigational autonomy in unmanned surface vehicles (USVs) in the maritime sector can fundamentally lead towards safer waters as well as reduced operating costs, while also providing a range of exciting new capabilities for oceanic research, exploration and monitoring. However, achieving such a goal is challenging. USV control systems must, safely and reliably, be able to adhere to the international regulations for preventing collisions at sea (COLREGs) in encounters with other vessels as they navigate to a given waypoint while being affected by realistic weather conditions, either during the day or at night. To deal with the multitude of possible scenarios, it is critical to have a virtual environment that is able to replicate the realistic operating conditions USVs will encounter, before they can be implemented in the real world. Such "digital twins" form the foundations upon which Deep Reinforcement Learning (DRL) and Computer Vision (CV) algorithms can be used to develop and guide USV control systems. In this paper we describe the novel development of a COLREG-compliant DRL-based collision avoidant navigational system with CV-based awareness in a realistic ocean simulation environment. The performance of the trained autonomous Agents resulting from this approach is evaluated in several successful navigations to set waypoints in both open sea and coastal encounters with other vessels. A binary executable version of the simulator with trained agents is available at https://github.com/aavek/Aeolus-OceanComment: 22 pages, last blank page, 17 figures, 1 table, color, high resolution figure