Neural networks and reinforcement learning in wind turbine control

[EN] Pitch control of wind turbines is complex due to the intrinsic non-linear behavior of these devices, and the external disturbances they are subjected to related to changing wind conditions and other meteorological phenomena. This difficulty is even higher in the case of floating offshore turbines, due to ocean currents and waves. Neural networks and other intelligent control techniques have been proven very useful for the modeling and control of these complex systems. Thus, this paper presents different intelligent control configurations applied to wind turbine pitch control. Direct pitch control based on neural networks and reinforcement learning, and some hybrid control configurations are described. The usefulness of neuro-estimators for the improvement of controllers is also presented. Finally, some of these techniques are used in an application example with a land wind turbine model.[ES] El control del ángulo de las palas de las turbinas eólicas es complejo debido al comportamiento no lineal de los aerogeneradores, y a las perturbaciones externas a las que están sometidas debido a las condiciones cambiantes del viento y otros fenómenos meteorológicos. Esta dificultad se agrava en el caso de las turbinas flotantes marinas, donde también les afectan las corrientes marinas y las olas. Las redes neuronales, y otras técnicas del control inteligente, han demostrado ser muy útiles para el modelado y control de estos sistemas. En este trabajo se presentan diferentes configuraciones de control inteligente, basadas principalmente en redes neuronales y aprendizaje por refuerzo, aplicadas al control de las turbinas eólicas. Se describe el control directo del ángulo de las palas del aerogenerador y algunas configuraciones híbridas de control. Se expone la utilidad de los neuro-estimadores para la mejora de los controladores. Finalmente, se muestra un ejemplo de aplicación de algunas de estas técnicas en un modelo de turbina terrestre.Ministerio de Ciencia, Innovación y Universidades: Proyecto MCI AEI/FEDER RTI2018- 094902-B-C21Sierra-García, JE.; Santos, M. (2021). Redes neuronales y aprendizaje por refuerzo en el control de turbinas eólicas. Revista Iberoamericana de Automática e Informática industrial. 18(4):327-335. https://doi.org/10.4995/riai.2021.16111327335184Abouheaf, M., Gueaieb, W., Sharaf, A. 2018. Model-free adaptive learning control scheme for wind turbines with doubly fed induction generators. IET Renewable Power Generation 12(14), 1675-1686. https://doi.org/10.1049/iet-rpg.2018.5353Alvarez-Ramos, C. M., Santos, M., López, V. 2010. Reinforcement learning vs. A* in a role playing game benchmark scenario. In Computational Intelligence: Foundations and Applications (pp. 644-650). https://doi.org/10.1142/9789814324700_0097Asghar, A. B., Liu, X. 2018a. Adaptive neuro-fuzzy algorithm to estimate effective wind speed and optimal rotor speed for variable-speed wind turbine. Neurocomputing 272, 495-504. https://doi.org/10.1016/j.neucom.2017.07.022Asghar, A. B., Liu, X. 2018b. Estimation of wind speed probability distribution and wind energy potential using adaptive neuro-fuzzy methodology. Neurocomputing, 287, 58-67. https://doi.org/10.1016/j.neucom.2018.01.077Casteleiro-Roca, J.L., Quintián, H., Calvo-Rolle, J.L, Méndez-Pérez, J. A., Perez-Castelo, F.J., Corchado, E. 2020. Lithium iron phosphate power cell fault detection system based on hybrid intelligent system. Logic Journal of the IGPL, 28(1), 71-82. https://doi.org/10.1093/jigpal/jzz072Chavero-Navarrete, E., Trejo-Perea, M., Jáuregui-Correa, J. C., Carrillo- Serrano, R. V., Ronquillo-Lomeli, G., Ríos-Moreno, J. G. 2020. Hierarchical pitch control for small wind turbines based on fuzzy logic and anticipated wind speed measurement. Applied Sciences, 10(13), 4592. https://doi.org/10.3390/app10134592Chen, P., Han, D., Tan, F., Wang, J. 2020. Reinforcement-based robust variable pitch control of wind turbines. IEEE Access 8, 20493-20502. https://doi.org/10.1109/ACCESS.2020.2968853Demirdelen, T., Tekin, P., Aksu, I. O., Ekinci, F. 2019. The prediction model of characteristics for wind turbines based on meteorological properties using neural network swarm intelligence. Sustainability, 11(17), 4803. https://doi.org/10.3390/su11174803Deng, X., Yang, J., Sun, Y., Song, D., Xiang, X., Ge, X., Joo, Y. H. 2019. Sensorless effective wind speed estimation method based on unknown input disturbance observer and extreme learning machine. Energy, 186, 115790. https://doi.org/10.1016/j.energy.2019.07.120Deng, X., Yang, J., Sun, Y., Song, D., Yang, Y., Joo, Y. H. 2020. An effective wind speed estimation based extended optimal torque control for maximum wind energy capture. IEEE Access, 8, 65959-65969. https://doi.org/10.1109/ACCESS.2020.2984654Du, J., Wang, B. 2020. Pitch Control of wind turbines based on BP neural network PI. In Journal of Physics: Conference Series (Vol. 1678, No. 1, p. 012060). IOP Publishing. https://doi.org/10.1088/1742-6596/1678/1/012060El Maati, Y. A., El Bahir, L. 2020. Optimal fault tolerant control of large-scale wind turbines in the case of the pitch actuator partial faults. Complexity. https://doi.org/10.1155/2020/6210407Fernandez-Gauna, B., Fernandez-Gamiz, U., Grana, M. 2017. Variable speed wind turbine controller adaptation by reinforcement learning. Integrated Computer-Aided Engineering 24(1), 27-39. https://doi.org/10.3233/ICA-160531Fernandez-Gauna, B., Osa, J. L., Graña, M. 2018. Experiments of conditioned reinforcement learning in continuous space control tasks. Neurocomputing 271, 38-47. https://doi.org/10.1016/j.neucom.2016.08.155Guo, C., Wang, D. 2019. Frequency regulation and coordinated control for complex wind power systems. Complexity, 2019. https://doi.org/10.1155/2019/8525397Hosseini, E., Aghadavoodi, E., Ramírez, L. M. F. 2020. Improving response of wind turbines by pitch angle controller based on gain-scheduled recurrent ANFIS type 2 with passive reinforcement learning. Renewable Energy, 157, 897-910. https://doi.org/10.1016/j.renene.2020.05.060IRENA. 2019. Future of wind: Deployment, investment, technology, grid integration and socio-economic aspects (A Global Energy Transformation paper), International Renewable Energy Agency, Abu Dhabi. https://www.irena.org/-/media/Files/IRENA/Agency/Publication/2019/Oct/IRENA_Future_of_wind_2019.pdfJeon, T., Paek, I. 2021. Design and verification of the LQR controller based on fuzzy logic for large wind turbine. Energies, 14(1), 230. https://doi.org/10.3390/en14010230Jie, W., Jingchun, C., Lin, Y., Wenliang, W., Jian, D. 2020. Pitch control of wind turbine based on deep neural network. In IOP Conference Series: Earth and Environmental Science (Vol. 619, No. 1, p. 012034). IOP Publishing https://doi.org/10.1088/1755-1315/619/1/012034Jove, E., Casteleiro‐Roca, J. L., Quintián, H., Méndez‐Pérez, J. A., Calvo‐Rolle, J. L. 2019. A fault detection system based on unsupervised techniques for industrial control loops. Expert Systems, 36(4), e12395. https://doi.org/10.1111/exsy.12395Jove, E., Casteleiro-Roca, J., Quintián, H., Méndez-Pérez, J. A., Calvo-Rolle, J. L. 2020. Detección de anomalías basada en técnicas inteligentes de una planta de obtención de material bicomponente empleado en la fabricación de palas de aerogenerador. Revista Iberoamericana de Automática e Informática industrial, 17(1), 84-93. https://doi.org/10.4995/riai.2019.11055Li, M., Wang, S. 2019. Dynamic fault monitoring of pitch system in wind turbines using selective ensemble small-world neural networks. Energies, 12(17), 3256. https://doi.org/10.3390/en12173256Mikati, M., Santos, M., Armenta, C. 2013. Electric grid dependence on the configuration of a small-scale wind and solar power hybrid system. Renewable energy, 57, 587-593. https://doi.org/10.1016/j.renene.2013.02.018Moodi, H., Bustan, D. 2019. Wind turbine control using TS systems with nonlinear consequent parts. Energy, 172, 922-931. https://doi.org/10.1016/j.energy.2019.01.133Naciones Unidas. 2021. https://sdgs.un.org/2030agenda. Accedido por última vez en 15/08/2021Ngo, Q. V., Chai, Y., Nguyen, T. T. 2020. The fuzzy-PID based-pitch angle controller for small-scale wind turbine. International Journal of Power Electronics and Drive Systems, 11(1), 135. https://doi.org/10.11591/ijpeds.v11.i1.pp135-142Our World in Data. 2021. https://ourworldindata.org/renewable-energy. Accedido por última vez en 15/08/2021.Phan, B. C., Lai, Y. C. 2019. Control strategy of a hybrid renewable energy system based on reinforcement learning approach for an isolated microgrid. Applied Sciences, 9(19), 4001. https://doi.org/10.3390/app9194001Ren, H., Hou, B., Zhou, G., Shen, L., Wei, C., Li, Q. 2020. Variable pitch active disturbance rejection control of wind turbines based on BP neural network PID. IEEE Access, 8, 71782-71797. https://doi.org/10.1109/ACCESS.2020.2987912Rubio, P. M., Quijano, J. F., López, P. Z., Lozano, J. J. F., Cerezo, A. G., Casanova, J. O. 2019. Control inteligente para mejorar el rendimiento de una plataforma semisumergible híbrida con aerogeneradores y convertidores de oleaje: sistema de control borroso para la turbina. Revista Iberoamericana de Automática e Informática industrial, 16(4), 480-491. https://doi.org/10.4995/riai.2019.10972Saénz-Aguirre, A., Zulueta, E., Fernández-Gamiz, U., Lozano, J., Lopez-Guede, J. M. 2019. Artificial neural network based reinforcement learning for wind turbine yaw control. Energies 12(3), 436. https://doi.org/10.3390/en12030436Saénz‐Aguirre, A., Zulueta, E., Fernandez‐Gamiz, U., Ulazia, A., Teso‐Fz‐Betono, D. 2020. Performance enhancement of the artificial neural network-based reinforcement learning for wind turbine yaw control. Wind Energy 23(3), 676-690. https://doi.org/10.1002/we.2451Santos, M. 2011. Un enfoque aplicado del control inteligente. Revista Iberoamericana de Automática e Informática Industrial RIAI 8(4), 283-296. https://doi.org/10.1016/j.riai.2011.09.016Sedighizadeh, M., Rezazadeh, A. 2008. Adaptive PID controller based on reinforcement learning for wind turbine control. In: Proc. World Academy of Science, Engineering and Technology 27, 257-262.Sierra, J. E., Santos, M. 2018. Modelling engineering systems using analytical and neural techniques: Hybridization. Neurocomputing, 271, 70-83. https://doi.org/10.1016/j.neucom.2016.11.099Sierra-García, J. E., Santos, M. 2020a. Performance analysis of a wind turbine pitch neurocontroller with unsupervised learning. Complexity, 2020. https://doi.org/10.1155/2020/4681767Sierra-García, J. E., Santos, M. 2020b. Exploring reward strategies for wind turbine pitch control by reinforcement learning. Applied Sciences, 10(21), 7462. https://doi.org/10.3390/app10217462Sierra-García, J. E., Santos, M. 2021a. Improving wind turbine pitch control by effective wind neuro-estimators. IEEE Access, 9, 10413-10425. https://doi.org/10.1109/ACCESS.2021.3051063Sierra-García, J. E., Santos, M. 2021b. Lookup table and neural network hybrid strategy for wind turbine pitch control. Sustainability, 13(6), 3235. https://doi.org/10.3390/su13063235Sierra-Garcia, J. E., Santos, M. 2021c. Deep learning and fuzzy logic to implement a hybrid wind turbine pitch control. Neural Computing and Applications, 1-15. https://doi.org/10.1007/s00521-021-06323-wSutton, R. S., Barto, A. G. 2015. Reinforcement learning an introduction-Second edition, in progress.Tomás-Rodríguez, M., Santos, M. 2019. Modelado y control de turbinas eólicas marinas flotantes. Revista Iberoamericana de Automática e Informática Industrial, 16(4), 381-390. https://doi.org/10.4995/riai.2019.11648Tomin, N., Kurbatsky, V., Guliyev, H. 2019. Intelligent control of a wind turbine based on reinforcement learning. In 16th Conf. on Electrical Machines, Drives and Power Systems ELMA, 1-6. IEEE. https://doi.org/10.1109/ELMA.2019.8771645Vives, J., Quiles, E., García, E. 2020. AI techniques applied to diagnosis of vibrations failures in wind turbines. IEEE Latin America Transactions, 18(08), 1478-1486. https://doi.org/10.1109/TLA.2020.9111685Zhao, H., Zhao, J., Qiu, J., Liang, G., Dong, Z. Y. 2020. Cooperative wind farm control with deep reinforcement learning and knowledge assisted learning. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/TII.2020.297403

Reinforcement learning-based structural control of floating wind turbines

The structural control of floating wind turbines using active tuned mass damper is investigated in this article. To our knowledge, this is for the first time that reinforcement learning-based control approach is employed to this type of application. Specifically, an adaptive dynamic programming (ADP) algorithm is used to derive the optimal control law based on the nonlinear structural dynamics, and the large-scale machine learning platform Tensorflow is employed for the design and implementation of the neural network (NN) structure. Three fully connected NNs, i.e., a plant network, a critic network, and an action network, are included in the proposed NN structure. Their training requires the gradient information flowing through the whole network, which is tackled by automatic differentiation, a popular technique for deriving the gradients of complex networks automatically. While to our knowledge, the network structures in the existing literature are rather simple and the training of the hidden layer is usually ignored. This allows their gradients to be derived analytically, which is infeasible with complex network structures. Thus, automatic differentiation greatly improves the employed ADP algorithm's ability in solving complex problems. The simulation results of structural control of floating wind turbines show that ADP controller performs very well in both normal and extreme conditions, with the standard deviation of the platform pitch displacement being reduced by around 40%. A clear advantage of ADP controllers over the H∞ controller is observed, especially in extreme conditions. Moreover, our design considers the tradeoff between the control performance and power consumption

Actor-critic continuous state reinforcement learning for wind-turbine control robust optimization

[EN] The control of Variable-Speed Wind-Turbines (VSWT) extracting electrical power from the wind kinetic energy are composed of subsystems that need to be controlled jointly, namely the blade pitch and the generator torque controllers. Previous state of the art approaches decompose the joint control problem into independent control subproblems, each with its own control subgoal, carrying out separately the design and tuning of a parameterized controller for each subproblem. Such approaches neglect interactions among subsystems which can introduce significant effects. This paper applies Actor-Critic Reinforcement Learning (ACRL) for the joint control problem as a whole, carrying out the simultaneous control parameter optimization of both subsystems without neglecting their interactions, aiming for a globally optimal control of the whole system. The innovative control architecture uses an augmented input space so that the parameters can be fine-tuned for each working condition. Validation results conducted on simulation experiments using the state-of-the-art OpenFAST simulator show a significant efficiency improvement relative to the best state of the art controllers used as benchmarks, up to a 22% improvement in the average power error performance after ACRL training.This work has been partially supported by FEDER funds through MINECO project TIN2017-85827-P, MCIN project PID2020-116346 GB-I00, and project KK-202000044 of the Elkartek 2020 funding program of the Basque Government

H-infinity Variable-Pitch Control for Wind Turbines Based on Takagi-Sugeno Fuzzy Theory

When the wind speed is above the rated value, the output power of the wind turbine should be maintained at the rated value in order to prevent the power generation system from overheating. In addition, the natural wind speed will fluctuate randomly in a large range of values, making the traditional control effect not ideal. This paper presents a novel H-infinity (H∞) pitch control strategy for Wind Turbine Generators (WTGs), which can make the rotor speed and output power constant when the wind speed changes in a large range. In order to shorten response time and reduce overshoot, in the specific solution, the control method combines the H∞ theory and the Takagi-Sugeno (T-S) fuzzy theory. Firstly, the linearized models of several operating points were obtained with the T-S fuzzy theory. Then, a robust controller was designed for each linear sub-system based on the H∞ control theory. Furthermore, the controllers of the sub-systems were superimposed into a global controller for the entire system through the membership function. Finally, modeling and simulation were carried out in MATLAB/SIMULINK. The simulation results show that when the wind speed changes above the rated speed, the rotor speed can be maintained at the rated value, and the output power also can be maintained at the rated value. Compared with the optimal control, the response speed of this method is faster and the overshoot is smaller. It provides a new idea for the pitch angle control of wind turbine

Actuator fault tolerant offshore wind turbine load mitigation control

Offshore wind turbine (OWT) rotors have large diameters with flexible blade structures which are subject to asymmetrical loads caused by blade flapping and turbulent or unsteady wind flow. Rotor imbalance inevitably leads to enhanced fatigue of blade rotor hub and tower structures. Hence, to enhance the life of the OWT and maintain good power conversion the unbalanced loading requires a reliable mitigation strategy, typically using a combination of Individual Pitch Control (IPC) and Collective Pitch Control (CPC). Increased pitch motion resulting from IPC activity can increase the possibility of pitch actuator faults and the resulting load imbalance results in loss of power and enhanced fatigue. This has accelerated the emergence of new research areas combining IPC with the fault tolerant control (FTC)-based fault compensation, a so-called FTC and IPC “co-design” system. A related research challenge is the clear need to enhance the robustness of the FTC IPC “co-design” to some dynamic uncertainty and unwanted disturbance. In this work a Bayesian optimization-based pitch controller using Proportional–Integral (PI) control is proposed to improve pitch control robustness. This is achieved using a systematic search for optimal controller coefficients by evaluating a Gaussian process model between the designed objective function and the coefficients. The pitch actuator faults are estimated and compensated using a robust unknown input observer (UIO)-based FTC scheme. The robustness and effectiveness of this “co-design” scheme are verified using Monte Carlo simulations applied to the 5MW NREL FAST WT benchmark system. The results show clearly (a) the effectiveness of the load mitigation control for a wide range of wind loading conditions, (b) the effect of actuator faults on the load mitigation performance and (c) the recovery to normal load mitigation, subject to FTC action

Development of a contra-rotating tidal current turbine and analysis of performance

A contra-rotating marine current turbine has a number of attractive features: nearzero reactive torque on the support structure, near-zero swirl in the wake, and high relative inter-rotor rotational speeds. Modified blade element modelling theory has been used to design and predict the characteristics of such a turbine, and a model turbine and test rig have been constructed. Tests in a towing tank demonstrated the feasibility of the concept. Power coefficients were high for such a small model and in excellent agreement with predictions, confirming the accuracy of the computational modelling procedures. Highfrequency blade loading data were obtained in the course of the experiments. These show the anticipated dynamic components for a contra-rotating machine. Flow visualization of the wake verified the lack of swirl behind the turbine. A larger machine is presently under construction for sea trials

Hybrid pitch angle controller approaches for stable wind turbine power under variable wind speed

The production of maximum wind energy requires controlling various parts of medium to large-scale wind turbines (WTs). This paper presents a robust pitch angle control system for the rated wind turbine power at a wide range of simulated wind speeds by means of a proportional–integral–derivative (PID) controller. In addition, ant colony optimization (ACO), particle swarm optimization (PSO), and classical Ziegler–Nichols (Z-N) algorithms have been used for tuning the PID controller parameters to obtain within rated stable output power of WTs from fluctuating wind speeds. The proposed system is simulated under fast wind speed variation, and its results are compared with those of the PID-ZN controller and PID-PSO to verify its effeteness. The proposed approach contains several benefits including simple implementation, as well as tolerance of turbine parameters and several nonparametric uncertainties. Robust control of the generator output power with wind-speed variations can also be considered a significant advantage of this strategy. Theoretical analyses, as well as simulation results, indicate that the proposed controller can perform better in a wide range of wind speed compared with the PID-ZN and PID-PSO controllers. The WT model and hybrid controllers (PID-ACO and PID-PSO) have been developed in MATLAB/Simulink with validated controller models. The hybrid PID-ACO controller was found to be the most suitable in comparison to the PID-PSO and conventional PID. The root mean square (RMS) error calculated between the desired power and the WT’s output power with PID-ACO is found to be 0.00036, which is the smallest result among the studied controllers