Value Function Estimation in Optimal Control via Takagi-Sugeno Models and Linear Programming

[ES] La presente Tesis emplea técnicas de programación dinámica y aprendizaje por refuerzo para el control de sistemas no lineales en espacios discretos y continuos. Inicialmente se realiza una revisión de los conceptos básicos de programación dinámica y aprendizaje por refuerzo para sistemas con un número finito de estados. Se analiza la extensión de estas técnicas mediante el uso de funciones de aproximación que permiten ampliar su aplicabilidad a sistemas con un gran número de estados o sistemas continuos. Las contribuciones de la Tesis son: -Se presenta una metodología que combina identificación y ajuste de la función Q, que incluye la identificación de un modelo Takagi-Sugeno, el cálculo de controladores subóptimos a partir de desigualdades matriciales lineales y el consiguiente ajuste basado en datos de la función Q a través de una optimización monotónica. -Se propone una metodología para el aprendizaje de controladores utilizando programación dinámica aproximada a través de programación lineal. La metodología hace que ADP-LP funcione en aplicaciones prácticas de control con estados y acciones continuos. La metodología propuesta estima una cota inferior y superior de la función de valor óptima a través de aproximadores funcionales. Se establecen pautas para los datos y la regularización de regresores con el fin de obtener resultados satisfactorios evitando soluciones no acotadas o mal condicionadas. -Se plantea una metodología bajo el enfoque de programación lineal aplicada a programación dinámica aproximada para obtener una mejor aproximación de la función de valor óptima en una determinada región del espacio de estados. La metodología propone aprender gradualmente una política utilizando datos disponibles sólo en la región de exploración. La exploración incrementa progresivamente la región de aprendizaje hasta obtener una política convergida.[CA] La present Tesi empra tècniques de programació dinàmica i aprenentatge per reforç per al control de sistemes no lineals en espais discrets i continus. Inicialment es realitza una revisió dels conceptes bàsics de programació dinàmica i aprenentatge per reforç per a sistemes amb un nombre finit d'estats. S'analitza l'extensió d'aquestes tècniques mitjançant l'ús de funcions d'aproximació que permeten ampliar la seua aplicabilitat a sistemes amb un gran nombre d'estats o sistemes continus. Les contribucions de la Tesi són: -Es presenta una metodologia que combina identificació i ajust de la funció Q, que inclou la identificació d'un model Takagi-Sugeno, el càlcul de controladors subòptims a partir de desigualtats matricials lineals i el consegüent ajust basat en dades de la funció Q a través d'una optimització monotónica. -Es proposa una metodologia per a l'aprenentatge de controladors utilitzant programació dinàmica aproximada a través de programació lineal. La metodologia fa que ADP-LP funcione en aplicacions pràctiques de control amb estats i accions continus. La metodologia proposada estima una cota inferior i superior de la funció de valor òptima a través de aproximadores funcionals. S'estableixen pautes per a les dades i la regularització de regresores amb la finalitat d'obtenir resultats satisfactoris evitant solucions no fitades o mal condicionades. -Es planteja una metodologia sota l'enfocament de programació lineal aplicada a programació dinàmica aproximada per a obtenir una millor aproximació de la funció de valor òptima en una determinada regió de l'espai d'estats. La metodologia proposa aprendre gradualment una política utilitzant dades disponibles només a la regió d'exploració. L'exploració incrementa progressivament la regió d'aprenentatge fins a obtenir una política convergida.[EN] The present Thesis employs dynamic programming and reinforcement learning techniques in order to obtain optimal policies for controlling nonlinear systems with discrete and continuous states and actions. Initially, a review of the basic concepts of dynamic programming and reinforcement learning is carried out for systems with a finite number of states. After that, the extension of these techniques to systems with a large number of states or continuous state systems is analysed using approximation functions. The contributions of the Thesis are: -A combined identification/Q-function fitting methodology, which involves identification of a Takagi-Sugeno model, computation of (sub)optimal controllers from Linear Matrix Inequalities, and the subsequent data-based fitting of Q-function via monotonic optimisation. -A methodology for learning controllers using approximate dynamic programming via linear programming is presented. The methodology makes that ADP-LP approach can work in practical control applications with continuous state and input spaces. The proposed methodology estimates a lower bound and upper bound of the optimal value function through functional approximators. Guidelines are provided for data and regressor regularisation in order to obtain satisfactory results avoiding unbounded or ill-conditioned solutions. -A methodology of approximate dynamic programming via linear programming in order to obtain a better approximation of the optimal value function in a specific region of state space. The methodology proposes to gradually learn a policy using data available only in the exploration region. The exploration progressively increases the learning region until a converged policy is obtained.This work was supported by the National Department of Higher Education, Science, Technology and Innovation of Ecuador (SENESCYT), and the Spanish ministry of Economy and European Union, grant DPI2016-81002-R (AEI/FEDER,UE). The author also received the grant for a predoctoral stay, Programa de Becas Iberoamérica- Santander Investigación 2018, of the Santander Bank.Díaz Iza, HP. (2020). Value Function Estimation in Optimal Control via Takagi-Sugeno Models and Linear Programming [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/139135TESI

Analysis and Design of Hybrid Control Systems

Different aspects of hybrid control systems are treated: analysis, simulation, design and implementation. A systematic methodology using extended Lyapunov theory for design of hybrid systems is developed. The methodology is based on conventional control designs in separate regions together with a switching strategy. Dynamics are not well defined if the control design methods lead to fast mode switching. The dynamics depend on the salient features of the implementation of the mode switches. A theorem for the stability of second order switching together with the resulting dynamics is derived. The dynamics on an intersection of two sliding sets are defined for two relays working on different time scales. The current simulation packages have problems modeling and simulating hybrid systems. It is shown how fast mode switches can be found before or during simulation. The necessary analysis work is a very small overhead for a modern simulation tool. To get some experience from practical problems with hybrid control the switching strategy is implemented in two different software environments. In one of them a time-optimal controller is added to an existing PID controller on a commercial control system. Successful experiments with this hybrid controller shows the practical use of the method

The 1993 NASA-ODU American Society for Engineering Education (ASEE) Summer Faculty Fellowship Program

Since 1964, the National Aeronautics and Space Administration has supported a program of summer faculty fellowships for engineering and science educators. In a series of collaborations between NASA research and development centers and nearby universities, engineering faculty members spend 10 weeks working with professional peers on research. The Summer Faculty Program Committee of the American Society for Engineering Education supervises the programs. Objectives are: to further the professional knowledge of qualified engineering and science faculty members; to stimulate and exchange ideas between participants and NASA; to enrich and refresh the research and teaching activities of participants' institutions; and to contribute to the research objectives of the NASA center

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

Water Resources Management and Modeling

Hydrology is the science that deals with the processes governing the depletion and replenishment of water resources of the earth's land areas. The purpose of this book is to put together recent developments on hydrology and water resources engineering. First section covers surface water modeling and second section deals with groundwater modeling. The aim of this book is to focus attention on the management of surface water and groundwater resources. Meeting the challenges and the impact of climate change on water resources is also discussed in the book. Most chapters give insights into the interpretation of field information, development of models, the use of computational models based on analytical and numerical techniques, assessment of model performance and the use of these models for predictive purposes. It is written for the practicing professionals and students, mathematical modelers, hydrogeologists and water resources specialists

Numerical studies of reactive polymer flows in porous materials

This dissertation presents a new numerical model of reactive polymer flow in heterogeneous porous media. A moment representation of the log-normal polymer molecular weight distribution is used to model polymer as a multi-component species. Three leading moments are used to simulate the polymer transport and reaction processes in a two-dimensional porous medium. A new operator splitting technique that allows the moment equations for polymerization to be incorporated into a finite-difference transport model is developed. The novelty of this approach is the use of two different dependent variables (for the transport versus reaction parts of the problem). It is significant from a physical standpoint because previous techniques did not allow us to observe the full evolution of polymer molecular weights in space and time. In this dissertation, two types of flows are examined. The first is the injection of a polymerizing fluid into a heterogeneous material containing low viscosity fluid (e.g., air or water). Simulations show that, depending on the Damkohler number, preferential loss of material permeability can occur in either low or high permeability regions. Because this effect dictates subsequent flow patterns, this result suggests that front stability can be controlled through proper design of the flow dynamics versus reaction dynamics. The formation of steady viscous fingers was observed, which is a fundamentally different phenomenon than previously observed transient viscous fingering formed during displacements. It is affected by the competition between reaction and convection, which allows the behavior to be correlated with the Damkohler number. A critical Damkohler number exists, above which steady-state conditions are not observed. The critical Damkohler number is affected by the Peclet number and permeability field. The second type of simulation is the injection of polymerizing fluids under conditions that lead to viscous instabilities. Results show the Damkohler number again to be a critical parameter. In both cases, the scale and structure of the material heterogeneities have a significant effect on the resulting flow. These research results provide important information for various polymer processing applications