Control of wave energy converters using machine learning strategies

Wave energy converters are devices that are designed to extract power from ocean waves. Existing wave energy converter technologies are not financially viable yet. Control systems have been identified as one of the areas that can contribute the most towards the increase in energy absorption and reduction of loads acting on the structure, whilst incurring only minimal extra hardware costs. In this thesis, control schemes are developed for wave energy converters, with the focus on single isolated devices. Numerical models of increasing complexity are developed for the simulation of a point absorber, which is a type of wave energy converter with small dimensions with respect to the dominating wave length. After investigating state-of-the-art control schemes, the existing control strategies reported in the literature have been found to rely on the model of the system dynamics to determine the optimal control action. This is despite the fact that modelling errors can negatively affect the performance of the device, particularly in highly energetic waves when non-linear effects become more significant. Furthermore, the controller should be adaptive so that changes in the system dynamics, e.g. due to marine growth or non-critical subsystem failure, are accounted for. Hence, machine learning approaches have been investigated as an alternative, with a focus on neural networks and reinforcement learning for control applications. A time-averaged approach will be employed for the development of the control schemes to enable a practical implementation on WECs based on the standard in the industry at the moment. Neural networks are applied to the active control of a point absorber. They are used mainly for system identification, where the mean power is related to the current sea state and parameters of the power take-off unit. The developed control scheme presents a similar performance to optimal active control for the analysed simulations, which rely on linear hydrodynamics. Reinforcement learning is then applied to the passive and active control of a wave energy converter for the first time. The successful development of different control schemes is described in detail, focusing on the encountered challenges in the selection of states, actions and reward function. The performance of reinforcement learning is assessed against state-of-the-art control strategies. Reinforcement learning is shown to learn the optimal behaviour in a reasonable time frame, whilst recognizing each sea state without reliance on any models of the system dynamics. Additionally, the strategy is able to deal with model non-linearities. Furthermore, it is shown that the control scheme is able to adapt to changes in the device dynamics, as for instance due to marine growth

Docking control of an autonomous underwater vehicle using reinforcement learning

To achieve persistent systems in the future, autonomous underwater vehicles (AUVs) will need to autonomously dock onto a charging station. Here, reinforcement learning strategies were applied for the first time to control the docking of an AUV onto a fixed platform in a simulation environment. Two reinforcement learning schemes were investigated: one with continuous state and action spaces, deep deterministic policy gradient (DDPG), and one with continuous state but discrete action spaces, deep Q network (DQN). For DQN, the discrete actions were selected as step changes in the control input signals. The performance of the reinforcement learning strategies was compared with classical and optimal control techniques. The control actions selected by DDPG suffer from chattering effects due to a hyperbolic tangent layer in the actor. Conversely, DQN presents the best compromise between short docking time and low control effort, whilst meeting the docking requirements. Whereas the reinforcement learning algorithms present a very high computational cost at training time, they are five orders of magnitude faster than optimal control at deployment time, thus enabling an on-line implementation. Therefore, reinforcement learning achieves a performance similar to optimal control at a much lower computational cost at deployment, whilst also presenting a more general framework

Kinematic model of active extension across the Umbria-Marche Apennines from GPS measurements: fault slip-rates and interseismic coupling of the Alto Tiberina low-angle normal fault

The Umbria-Marche Apennines are characterized mainly by SW-NE oriented extensional deformation and most of major historical and instrumental earthquakes occurred mainly on the western side of chain, bounded by west-dipping buried high-angle normal faults. Recent studies about the northernmost part of Umbria-Marche region show seismic and tectonic activity on correspondence of the east-dipping Alto Tiberina (AT) low-angle normal fault (LANF), which is widely documented by geological data and deep seismic reflection profiles. In this area which of the known fault systems play a major role in accommodating the extension, and which are the modes (seismic VS aseismic deformation) this extension is taken up, is still a debated topic. During last years on Umbria-Marche Apennines close to Gubbio fault (GuF) a dense network of continuous GPS stations, belonging to the RING-INGV network, has been installed, improving significantly the spatial resolution of the detectable geodetic gradients. We used a self-consistent kinematic block modeling to study this sector of the Umbria-Marche Apennines, in order to understand which fault system is accommodating the tectonic extension. We found that both fault systems, i.e. the Alto Tiberina LANF and the antithetic high-angle normal faults, are needed to better reproduce the nearfield GPS velocities, obtaining kinematic agreement with geological slip-rates. Moreover we parameterized the ATF fault as a, more realistic, curved surface to infer the distribution of interseismic coupling (IC), which is validated by numerous resolution tests. The obtained IC distribution shows a correlation between relocated microseismicity and uncoupled patches attributed to aseismic creeping behavior, which could be explained by the presence of fluid overpressure. Otherwise this correlation has been verified with a very small quantity of events (almost 400) and it might be of interest to evaluate this correlation with future available data

Augmented Neural Lyapunov Control

Machine learning-based methodologies have recently been adapted to solve control problems. The Neural Lyapunov Control (NLC) method is one such example. This approach combines Artificial Neural Networks (ANNs) with Satisfiability Modulo Theories (SMT) solvers to synthesise stabilising control laws and to prove their formal correctness. The ANNs are trained over a dataset of state-space samples to generate candidate control and Lyapunov functions, while the SMT solvers are tasked with certifying the correctness of the Lyapunov function over a continuous domain or by returning a counterexample. Despite the approach’s attractiveness, issues can occur due to subsequent calls of the SMT module at times returning similar counterexamples, which can turn out to be uninformative and may lead to dataset overfitting. Additionally, the control network weights are usually initialised with pre-computed gains from state-feedback controllers, e.g. Linear-Quadratic Regulators. To properly perform the initialisation requires user time and control expertise. In this work, we present an Augmented NLC method that mitigates these drawbacks, removes the need for the control initialisation and further improves counterexample generation. As a result, the proposed method allows the synthesis of nonlinear (as well as linear) control laws with the sole requirement being the knowledge of the system dynamics. The ANLC is tested over challenging benchmarks such as the Lorenz attractor and outperformed existing methods in terms of successful synthesis rate. The developed framework is released open-source at: https://github.com/grande-dev/Augmented-Neural-Lyapunov-Control

Sliding mode control of a nonlinear wave energy converter model

The most accurate wave energy converter models for heaving point absorbers include nonlinearities, which increase as resonance is achieved to maximize the energy capture. Over the power production spectrum and within the physical limits of the devices, the efficiency of wave energy converters can be enhanced by employing a control scheme that accounts for these nonlinearities. This paper proposes a sliding mode control for a heaving point absorber that includes the nonlinear effects of the dynamic and static Froude‐Krylov forces. The sliding mode controller tracks a reference velocity that matches the phase of the excitation force to ensure higher energy absorption. This control algorithm is tested in regular linear waves and is compared to a complex‐conjugate control and a nonlinear variation of the complex‐conjugate control. The results show that the sliding mode control successfully tracks the reference and keeps the device displacement bounded while absorbing more energy than the other control strategies. Furthermore, due to the robustness of the control law, it can also accommodate disturbances and uncertainties in the dynamic model of the wave energy converter

Kinematic block modeling of GPS velocities in Italy and seismic potential

We use a dense GPS velocity field, from the analysis of >1000 continuous stations, and elastic block modeling to study the interseismic strain accumulation along the Alpine and Apennines active tectonic belts in Italy. We consider available fault catalogues, instrumental and historical seismicity to determine the blocks boundaries geometry, parameterized as uniformly slipping rectangular planes. We invert horizontal velocities to estimate Euler vectors of tectonic blocks together with slip-rates at block-bounding faults. When allowed by density of GPS data, we optimize faults dip and locking-depth by searching the parameters that provide the best fit to local GPS data. Overall we obtain a good fit of the horizontal velocities and geodetic slip rates that are kinematically consistent with available geological and seismotectonic information. We use the best-fit geometric and kinematic model parameters to compute the expected GPS velocities over a dense regular grid. Denser model velocities are used to estimate the velocity gradient field on a regular grid, made by cell elements of 0.25°x0.25°. Geodetic strain-rates at each cell are converted into seismic moment accumulation rates, following the Kostrov formulation, considering as seismogenic thickness values obtained from a crustal (EPcrust) model and earthquake hypocentral distribution. Geodetic moment accumulation rates are compared with seismic moment rates released by earthquakes, obtained from the analysis of a seismic catalogue realized by merging several instrumental and historical catalogues covering the 1600-2012 timespan, and uniformly defined moment magnitudes. The comparison between geodetic moment accumulation rates and seismic moment release rates highlights regions with significant moment deficits but also areas with a surplus of the seismic moment released, with important implications for seismic hazard evaluations and assumptions behind the approach used in this work

Control of a Realistic Wave Energy Converter Model using Least-Squares Policy Iteration

PublishedThis is the author accepted manuscript. The final version is available from Institute of Electrical and Electronics Engineers via the DOI in this record.An algorithm has been developed for the resistive control of a non-linear model of a wave energy converter using least-squares policy iteration, which incorporates function approximation, with tabular and radial basis functions being used as features. With this method, the controller learns the optimal PTO damping coefficient in each sea state for the maximization of the mean generated power. The performance of the algorithm is assessed against two on-line reinforcement learning schemes: Q-learning and SARSA. In both regular and irregular waves, least-squares policy iteration outperforms the other strategies, especially when starting from unfavourable conditions for learning. Similar performance is observed for both basis functions, with a smaller number of radial basis functions underfitting the Q-function. The shorter learning time is fundamental for a practical application on a real wave energy converter. Furthermore, this work shows that least-squares policy iteration is able to maximize the energy absorption of a wave energy converter despite strongly non-linear effects due to its model-free nature, which removes the influence of modelling errors. Additionally, the floater geometry has been changed during a simulation to show that reinforcement learning control is able to adapt to variations in the system dynamics.This work was supported partly by the Energy Technologies Institute and the Research Councils Energy Programme (grant EP/J500847/), partly by the Engineering and Physical Sciences Research Council (grant EP/J500847/1), and partly by Wave Energy Scotland

Radiation Force Modeling for a Wave Energy Converter Array

The motivation and focus of this work is to generate passive transfer function matrices that model the radiation forces for an array of WECs. Multivariable control design is often based on linear time-invariant (LTI) systems such as state-space or transfer function matrix models. The intended use is for designing real-time control strategies where knowledge of the model’s poles and zeros is helpful. This work presents a passivity-based approach to estimate radiation force transfer functions that accurately replace the convolution operation in the Cummins equation while preserving the physical properties of the radiation function. A two-stage numerical optimization approach is used, the first stage uses readily available algorithms for fitting a radiation damping transfer function matrix to the system’s radiation frequency response. The second stage enforces additional constraints on the form of the transfer function matrix to increase its passivity index. After introducing the passivity-based algorithm to estimate radiation force transfer functions for a single WEC, the algorithm was extended to a WEC array. The proposed approach ensures a high degree of match with the radiation function without degrading its passivity characteristics. The figures of merit that will be assessed are (i) the accuracy of the LTI systems in approximating the radiation function, as measured by the normalized root mean squared error (NRMSE), and (ii) the stability of the overall system, quantified by the input passivity index, , of the radiation force transfer function matrix

Control of a Point Absorber using Reinforcement Learning

This work presents the application of reinforcement learning for the optimal resistive control of a point absorber. The model-free Q-learning algorithm is selected in order to maximise energy absorption in each sea state. Step changes are made to the controller damping, observing the associated penalty, for excessive motions, or reward, i.e. gain in associated power. Due to the general periodicity of gravity waves, the absorbed power is averaged over a time horizon lasting several wave periods. The performance of the algorithm is assessed through the numerical simulation of a point absorber subject to motions in heave in both regular and irregular waves. The algorithm is found to converge towards the optimal controller damping in each sea state. Additionally, the model-free approach ensures the algorithm can adapt to changes to the device hydrodynamics over time and is unbiased by modelling errors.The authors would like to thank the Energy Technology Institute and the Research Council Energy Programme for funding this research as part of the IDCORE programme (grant EP/J500847) as well as the Engineering and Physical Sciences Research Council (grant EP/J500847/1). In addition, Mr. Anderlini would like to thank Wave Energy Scotland for sponsoring his Eng.D. research project

Machine learning in sustainable ship design and operation: a review

The shipping industry faces a large challenge as it needs to significantly lower the amounts of Green House Gas emissions. Traditionally, reducing the fuel consumption for ships has been achieved during the design stage and, after building a ship, through optimisation of ship operations. In recent years, ship efficiency improvements using Machine Learning (ML) methods are quickly progressing, facilitated by available data from remote sensing, experiments and high-fidelity simulations. The data have been successfully applied to extract intricate empirical rules that can reduce emissions thereby helping achieve green shipping. This article presents an overview of applying ML techniques to enhance ships’ sustainability. The work covers the ML fundamentals and applications in relevant areas: ship design, operational performance, and voyage planning. Suitable ML approaches are analysed and compared on a scenario basis, with their space for improvements also discussed. Meanwhile, a reminder is given that ML has many inherent uncertainties and hence should be used with caution