Actor-critic reinforcement learning algorithms for yaw control of an Autonomous Underwater Vehicle

An Autonomous Underwater Vehicle (AUV) poses unique challenges that must be solved in order to achieve persistent autonomy. The requirement of persistent autonomy entails that a control solution must be capable of controlling a vehicle that is operating in an environment with complex non-linear dynamics and adapt to changes in those dynamics. In essence, artificial intelligence is required so that the vehicle can learn from its experience operating in the domain. In this thesis, reinforcement learning is the chosen machine learning mechanism. This learning paradigm is investigated by applying multiple actor-critic temporal difference learning algorithms to the yaw degree-of-freedom of a simulated model and the physical hardware of the Nessie VII AUV in a closed-loop feedback control problem. Additionally, results are also presented for path planning and path optimisation problems. These control problems are solved by modelling the AUV’s interaction with its environment as an optimal decision-making problem using a Markov Decision Process (MDP). Two novel actor-critic temporal difference learning algorithms called Linear True Online Continuous Learning Automation (Linear TOCLA) and Non-linear True Online Continuous Learning Automation (Non-linear TOCLA) are also presented and serve as new contributions to the reinforcement learning research community. These algorithms have been applied to the real Nessie vehicle and its simulated model. The proposed algorithms hold theoretical and practical advantages over previous state-of-the-art temporal difference learning algorithms. A new genetic algorithm is also presented and developed specifically for the optimisation of the continuous-valued reinforcement learning algorithms’. This genetic algorithm is used to find the optimal hyperparameters for four actor-critic algorithms in the well-known continuous-valued mountain car reinforcement learning benchmark problem. The results of this benchmark show that the Non-linear TOCLA algorithm achieves a similar performance to the state-of-the-art forward actor-critic algorithm it extends while significantly reducing the sensitivity of the hyperparameter selection. This reduction in hyperparameter sensitivity is shown using the distribution of optimal hyperparameters from ten separate optimisation runs. The actor learning rate of the forward actor-critic algorithm had a standard deviation of 0.00088, while the Non-linear TOCLA algorithm demonstrated a standard deviation of 0.00186. An even greater improvement is observed in the multi-step target weight, λ, which increased from a standard deviation of 0.036 for the forward actor-critic to 0.266 for the Non-linear TOCLA algorithm. All of the sourcecode used to generate the results in this thesis has been made available as open-source software.ARchaeological RObot systems for the Worlds Seas (ARROWS) EU FP7 project under grant agreement ID 30872

Development of Robust Control Strategies for Autonomous Underwater Vehicles

The resources of the energy and chemical balance in the ocean sustain mankind in many ways. Therefore, ocean exploration is an essential task that is accomplished by deploying Underwater Vehicles. An Underwater Vehicle with autonomy feature for its navigation and control is called Autonomous Underwater Vehicle (AUV). Among the task handled by an AUV, accurately positioning itself at a desired position with respect to the reference objects is called set-point control. Similarly, tracking of the reference trajectory is also another important task. Battery recharging of AUV, positioning with respect to underwater structure, cable, seabed, tracking of reference trajectory with desired accuracy and speed to avoid collision with the guiding vehicle in the last phase of docking are some significant applications where an AUV needs to perform the above tasks. Parametric uncertainties in AUV dynamics and actuator torque limitation necessitate to design robust control algorithms to achieve motion control objectives in the face of uncertainties. Sliding Mode Controller (SMC), H / μ synthesis, model based PID group controllers are some of the robust controllers which have been applied to AUV. But SMC suffers from less efficient tuning of its switching gains due to model parameters and noisy estimated acceleration states appearing in its control law. In addition, demand of high control effort due to high frequency chattering is another drawback of SMC. Furthermore, real-time implementation of H / μ synthesis controller based on its stability study is restricted due to use of linearly approximated dynamic model of an AUV, which hinders achieving robustness. Moreover, model based PID group controllers suffer from implementation complexities and exhibit poor transient and steady-state performances under parametric uncertainties. On the other hand model free Linear PID (LPID) has inherent problem of narrow convergence region, i.e.it can not ensure convergence of large initial error to zero. Additionally, it suffers from integrator-wind-up and subsequent saturation of actuator during the occurrence of large initial error. But LPID controller has inherent capability to cope up with the uncertainties. In view of addressing the above said problem, this work proposes wind-up free Nonlinear PID with Bounded Integral (BI) and Bounded Derivative (BD) for set-point control and combination of continuous SMC with Nonlinear PID with BI and BD namely SM-N-PID with BI and BD for trajectory tracking. Nonlinear functions are used for all P,I and D controllers (for both of set-point and tracking control) in addition to use of nonlinear tan hyperbolic function in SMC(for tracking only) such that torque demand from the controller can be kept within a limit. A direct Lyapunov analysis is pursued to prove stable motion of AUV. The efficacies of the proposed controllers are compared with other two controllers namely PD and N-PID without BI and BD for set-point control and PD plus Feedforward Compensation (FC) and SM-NPID without BI and BD for tracking control. Multiple AUVs cooperatively performing a mission offers several advantages over a single AUV in a non-cooperative manner; such as reliability and increased work efficiency, etc. Bandwidth limitation in acoustic medium possess challenges in designing cooperative motion control algorithm for multiple AUVs owing to the necessity of communication of sensors and actuator signals among AUVs. In literature, undirected graph based approach is used for control design under communication constraints and thus it is not suitable for large number of AUVs participating in a cooperative motion plan. Formation control is a popular cooperative motion control paradigm. This thesis models the formation as a minimally persistent directed graph and proposes control schemes for maintaining the distance constraints during the course of motion of entire formation. For formation control each AUV uses Sliding Mode Nonlinear PID controller with Bounded Integrator and Bounded Derivative. Direct Lyapunov stability analysis in the framework of input-to-state stability ensures the stable motion of formation while maintaining the desired distance constraints among the AUVs

Underwater Visual Inspection using Robotics

[EN] This article presents an industrial application of underwater robotics for visual inspection of the seabed. The system consists of an underwater robot to acquire visual images at a very short distance (1-2 meters) from the surface to be inspected and an underwater localization system installed in a boat or in a mooring buoy. This system estimates the robot absolute position during the inspection in Earth coordinates using a USBL acoustic locator, an inertial navigation system (INS) and a GPS. Moreover, the robot has its own navigation system onboard, based on EKF, which uses a velocity sensor based on Doppler effect and an INS with a fiber optic gyroscope (FOG). The control architecture of the robot allows the inspection to be in teleoperation, semi-autonomous or autonomous mode. The robot can perform the inspection in 2D vertical or horizontal surfaces. Once the images and all navigation data have been acquired, an offline process is executed for data fusion, and the processing of the images ends up with a 2D georeferenced image mosaic from the inspected area. The paper details the developed technologies and describes a campaign in the Mequinenza reservoir to detect zebra mussel populations.[ES] Este artículo presenta una aplicación industrial de robótica submarina que consiste en un sistema para realizar inspecciones visuales del fondo subacuático. El sistema consta de un robot submarino para adquirir imágenes visuales a poca distancia (1-2 metros) de la superficie a inspeccionar y un localizador submarino instalado en una embarcación o boya. Este localizador permite conocer la posición absoluta del robot durante la inspección y se basa en un sistema acústico de tipo USBL (Ultra Short Base Line), una unidad de navegación inercial (INS) y un GPS. Además, el robot tiene su propio sistema de navegación a bordo, basado en EKF, que utiliza un sensor de velocidad basado en efecto Doppler y una INS con un giroscopio de fibra óptica (FOG). La arquitectura de control del robot permite realizar la inspección de forma teleoperada, semi-autónoma o completamente autónoma. El robot puede realizar inspecciones de superficies 2D verticales y horizontales. Una vez adquiridas las imágenes y todos los datos de navegación y percepción, se realiza un proceso fuera de linea de fusión de los datos y procesado de las imágenes que concluye con la generación de un mosaico 2D georeferenciado de la superficie inspeccionada. El artículo detalla las tecnologías desarrolladas y describe una campaña realizada en el embalse de Mequinenza (Aragón) para detectar poblaciones de mejillón cebra.Este trabajo ha sido realizado gracias al apoyo del Ministerio de Educación y Ciencia (proyectos DPI2005-09001, CTM200404205, DPI2008-06548 y CTM2010-1521), al Centro de Innovacion y Desarrollo Empresarial de la Generalitat de Catalunya (Proyecto INSPECSUB) y a la colaboración con la empresa Ecohydros S.L.Carreras, M.; Ridao, P.; García, R.; Ribas, D.; Palomeras, N. (2012). Inspección visual subacuática mediante robótica submarina. Revista Iberoamericana de Automática e Informática industrial. 9(1):34-45. https://doi.org/10.1016/j.riai.2011.11.011OJS344591Arkin, R.C., 1998. Behavior-Based Robotics. The MIT Press, Cambridge, MA, USA.Bay, H., Tuytelaars, T., Gool, L.V., may 2006. SURF: Speeded up robust features. In: European Conference on Computer Vision. Graz, Austria, pp. 404-417.Brokloff, N.A., September 1994. Matrix algorithm for Doppler sonar navigation. In: Proceedings of the Oceans MTS/IEEE. Vol. 3. Brest, France, pp. 378-383.Carreras, M., Yuh, J., Batlle, J., & Ridao, P. (2007). Application of SONQL for real-time learning of robot behaviors. Robotics and Autonomous Systems, 55(8), 628-642. doi:10.1016/j.robot.2007.03.003Cote, J., Lavallee, J., 1995. Augmented reality graphic interface for upstream dam inspection. In: Proceedings of SPIE Telemanipulator and Telepresence Technologies II. Vol. 2590. Philadelphia, PA, pp. 33-39.Ferrer, J., Elibol, A., Delaunoy, O., Gracias, N., García, R., November 2007. Large-area photo-mosaics using global alignment and navigation data. In: Proceedings of the Oceans MTS/IEEE. Vancouber, Canada, pp. 1-9.Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus. Communications of the ACM, 24(6), 381-395. doi:10.1145/358669.358692Fossen, T.I., 1994. Guidance and Control of Ocean Vehicles. John Wiley & Sons Ltd.Kalman, R.E., March 1960. A new approach to linear filtering and prediction problems. Transactions of the ASME, Journal of Basic Engineering 82 (Series D), 35-45.Kazhdan, M.M., Hoppe, H., 2008. Streaming multigrid for gradient-domain operations on large images. ACM Trans. Graph. 27 (3).Leonard, J.J., Rikoski, R.J., 2001. Incorporation of delayed decision making into stochastic mapping. Vol. 271. Springer Verlag, pp. 533-542.Palomeras, N., Ridao, P., Carreras, M., Silvestre, C., 2009. Using petri nets to specify and execute missions for autonomous underwater vehicles. In: International Conference on Intelligent Robots and Systems. St. Louis, MO, pp. 4439-4444.Poupart, M., Benefice, P., Plutarque, M., November 2001. Subacuatic inspections of EDF (Electricite de France) dams. In: OCEANS, 2000. MTS/IEEE Conference and Exhibition. Vol. 2. pp. 939-942.Prados, R., May 2007. Image blending techniques and their application in underwater mosaicing. Master's thesis, University of Girona.Ribas, D., Palomer, N., Ridao, P., Carreras, M., Hern‘andez, E., April 2007. Ictineu AUV wins the first SAUC-E competition. In: Proceedings of the IEEE International Conference on Robotics and Automation. Roma, Italy, pp. 151-156.Ribas, D., Ridao, P., Neira, J., 2010. Underwater SLAM for Structured Environments Using an Imaging Sonar. No. 65 in Springer Tracts in Advanced Robotics. Springer Verlag.Ridao, P., Batlle, E., Ribas, D., Carreras, M., November 9-12 2004. NEPTUNE: A HIL simulator for multiple UUVs. In: Proceedings of the Oceans MTS/IEEE. Vol. 1. Kobe, Japan, pp. 524-531.Ridao, P., Carreras, M., Ribas, D., & Garcia, R. (2010). Visual inspection of hydroelectric dams using an autonomous underwater vehicle. Journal of Field Robotics, 27(6), 759-778. doi:10.1002/rob.20351Smith, R., Self, M., Cheeseman, P., 1990. Estimating uncertain spatial relationships in robotics. In: Autonomous robot vehicles. Springer-Verlag New York, Inc., New York, NY, USA, pp. 167-193.Tao, 2003. TAO Developer's Guide Version 1.3a. Vol. 2. Object Computing Inc. Thrun, S., Burgard, W., Fox, D., 2005. Probabilistic Robotics. The MIT Press