Value Function Estimation in Optimal Control via Takagi-Sugeno Models and Linear Programming

[ES] La presente Tesis emplea técnicas de programación dinámica y aprendizaje por refuerzo para el control de sistemas no lineales en espacios discretos y continuos. Inicialmente se realiza una revisión de los conceptos básicos de programación dinámica y aprendizaje por refuerzo para sistemas con un número finito de estados. Se analiza la extensión de estas técnicas mediante el uso de funciones de aproximación que permiten ampliar su aplicabilidad a sistemas con un gran número de estados o sistemas continuos. Las contribuciones de la Tesis son: -Se presenta una metodología que combina identificación y ajuste de la función Q, que incluye la identificación de un modelo Takagi-Sugeno, el cálculo de controladores subóptimos a partir de desigualdades matriciales lineales y el consiguiente ajuste basado en datos de la función Q a través de una optimización monotónica. -Se propone una metodología para el aprendizaje de controladores utilizando programación dinámica aproximada a través de programación lineal. La metodología hace que ADP-LP funcione en aplicaciones prácticas de control con estados y acciones continuos. La metodología propuesta estima una cota inferior y superior de la función de valor óptima a través de aproximadores funcionales. Se establecen pautas para los datos y la regularización de regresores con el fin de obtener resultados satisfactorios evitando soluciones no acotadas o mal condicionadas. -Se plantea una metodología bajo el enfoque de programación lineal aplicada a programación dinámica aproximada para obtener una mejor aproximación de la función de valor óptima en una determinada región del espacio de estados. La metodología propone aprender gradualmente una política utilizando datos disponibles sólo en la región de exploración. La exploración incrementa progresivamente la región de aprendizaje hasta obtener una política convergida.[CA] La present Tesi empra tècniques de programació dinàmica i aprenentatge per reforç per al control de sistemes no lineals en espais discrets i continus. Inicialment es realitza una revisió dels conceptes bàsics de programació dinàmica i aprenentatge per reforç per a sistemes amb un nombre finit d'estats. S'analitza l'extensió d'aquestes tècniques mitjançant l'ús de funcions d'aproximació que permeten ampliar la seua aplicabilitat a sistemes amb un gran nombre d'estats o sistemes continus. Les contribucions de la Tesi són: -Es presenta una metodologia que combina identificació i ajust de la funció Q, que inclou la identificació d'un model Takagi-Sugeno, el càlcul de controladors subòptims a partir de desigualtats matricials lineals i el consegüent ajust basat en dades de la funció Q a través d'una optimització monotónica. -Es proposa una metodologia per a l'aprenentatge de controladors utilitzant programació dinàmica aproximada a través de programació lineal. La metodologia fa que ADP-LP funcione en aplicacions pràctiques de control amb estats i accions continus. La metodologia proposada estima una cota inferior i superior de la funció de valor òptima a través de aproximadores funcionals. S'estableixen pautes per a les dades i la regularització de regresores amb la finalitat d'obtenir resultats satisfactoris evitant solucions no fitades o mal condicionades. -Es planteja una metodologia sota l'enfocament de programació lineal aplicada a programació dinàmica aproximada per a obtenir una millor aproximació de la funció de valor òptima en una determinada regió de l'espai d'estats. La metodologia proposa aprendre gradualment una política utilitzant dades disponibles només a la regió d'exploració. L'exploració incrementa progressivament la regió d'aprenentatge fins a obtenir una política convergida.[EN] The present Thesis employs dynamic programming and reinforcement learning techniques in order to obtain optimal policies for controlling nonlinear systems with discrete and continuous states and actions. Initially, a review of the basic concepts of dynamic programming and reinforcement learning is carried out for systems with a finite number of states. After that, the extension of these techniques to systems with a large number of states or continuous state systems is analysed using approximation functions. The contributions of the Thesis are: -A combined identification/Q-function fitting methodology, which involves identification of a Takagi-Sugeno model, computation of (sub)optimal controllers from Linear Matrix Inequalities, and the subsequent data-based fitting of Q-function via monotonic optimisation. -A methodology for learning controllers using approximate dynamic programming via linear programming is presented. The methodology makes that ADP-LP approach can work in practical control applications with continuous state and input spaces. The proposed methodology estimates a lower bound and upper bound of the optimal value function through functional approximators. Guidelines are provided for data and regressor regularisation in order to obtain satisfactory results avoiding unbounded or ill-conditioned solutions. -A methodology of approximate dynamic programming via linear programming in order to obtain a better approximation of the optimal value function in a specific region of state space. The methodology proposes to gradually learn a policy using data available only in the exploration region. The exploration progressively increases the learning region until a converged policy is obtained.This work was supported by the National Department of Higher Education, Science, Technology and Innovation of Ecuador (SENESCYT), and the Spanish ministry of Economy and European Union, grant DPI2016-81002-R (AEI/FEDER,UE). The author also received the grant for a predoctoral stay, Programa de Becas Iberoamérica- Santander Investigación 2018, of the Santander Bank.Díaz Iza, HP. (2020). Value Function Estimation in Optimal Control via Takagi-Sugeno Models and Linear Programming [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/139135TESI

CONTROL PREDICTIVO BASADO EN ESCENARIOS PARA SISTEMAS LINEALES CON SALTOS MARKOVIANOS

[EN] In this thesis, invariant-set theory is used to study the stability and feasibility of constrained scenario-based predictive controllers for Markov-jump linear systems. In the underlying optimisation problem of the predictive controllers technique, considers all possible future realisations of certain variables (uncertainty, disturbances, operating mode) or just a subset of those. Two different scenarios denoted as not risky and risky are studied. In the former, the trajectories of the system with initial states belonging to certain invariant sets, converge (in mean square sense) to the origin or an invariant neighbourhood of it with 100% probability. In such cases, the conditions that scenario trees must meet in order to guarantee stability and feasibility of the optimisation problem are analysed. Afterwards, the scenario-based predictive controller for Markov-jump linear systems under hard constraints and no disturbances is formulated. A study is presented for risky scenarios to determine sequence-dependent controllable sets, for which there exists a control law such that the system can be driven to the origin only for a particular realisation of uncertainty, disturbances, etc. A control law (optimal for disturbances-free systems and suboptimal for disturbed systems) able to steer the system to the origin with a probability less than 100% (denoted as reliability bound), is proposed for states belonging to those regions. Note that closed-loop unstable systems have zero reliability bound. Hence, an algorithm to determine the mean-time to failure is developed. In this context, failure means a violation in the constraints of the process' states and/or inputs in a future time.[ES] La presente tesis emplea la teoría de conjuntos invariantes para el estudio de estabilidad y factibilidad de controladores predictivos basados en escenarios para sistemas lineales con saltos markovianos sujetos a restricciones. En el problema de optimización subyacente a la técnica de controladores predictivos, se consideran bien sea todas las posibles realizaciones futuras de una variable (incertidumbres, perturbaciones, modo de funcionamiento) o solo un subconjunto de estas. Se estudian dos escenarios diferentes, denotados como: a) escenarios no arriesgados y b) escenarios arriesgados, entendiéndose como no arriesgados, aquellos en donde las trayectorias del sistema con estados iniciales pertenecientes a ciertos conjuntos invariantes, convergen --en media-- al origen o a una vecindad invariante de este con un 100% de probabilidad. Para estos casos, se presenta un análisis de las condiciones que deben cumplir los árboles de escenarios para garantizar estabilidad --en media-- y factibilidad del problema de optimización. Luego se formula el control predictivo basado en escenarios para sistema lineales con saltos markovianos sujeto a restricciones y en ausencia de perturbaciones. En presencia de escenarios arriesgados, se propone el cálculo de conjuntos controlables dependientes de secuencias para los cuales existen una ley de control tal que el sistema puede ser conducido al origen, solo para una realización en particular de la incertidumbre, perturbaciones, etc. Para estados pertenecientes a estos conjuntos, se propone una ley de control (óptima para el caso de sistemas libres de perturbaciones y, subóptima para sistemas perturbados) capaz de dirigir el sistema al origen con una probabilidad menor al 100%, dicha probabilidad es denotada como cota de confiabilidad. Sistemas inestables en lazo cerrado tienen cota de confiabilidad igual a cero, por consiguiente se diseña un algoritmo que determina el tiempo medio para fallar. En este contexto, un fallo se entiende como la violación de las restricciones en los estados y/o entradas del proceso en algún instante de tiempo futuro.[CA] La present tesi empra la teoria de conjunts invariants per a l'estudi d'estabili-tat i factibilidad de controladors predictius basats en escenaris per a sistemes lineals amb salts markovians subjectes a restriccions. En el problema d'optimit-zació subjacent a la tècnica de controladors predictius, es consideren bé siga totes les possibles realitzacions futures d'una variable (incerteses, pertorbacions, modes de funcionament) o només un subconjunt d'aquestes. S'estudien dos escenaris diferents, denotats com a escenaris no arriscats i arriscats, entenent-se com no arriscats, aquells on les trajectòries del sistema amb estats inicials pertanyents a certs conjunts invariants, convergeixen --en mitjana-- a l'origen o a un veïnatge invariant d'est amb un 100% de probabilitat. Per a aquests casos, es presenta una anàlisi de les condicions que han de complir els arbres d'escenaris per a garantir estabilitat --en mitjana-- i factibilidad del problema d'optimització. Després es formula el control predictiu basat en escenaris per a sistema lineals amb salts markovians subjecte a restriccions i en absència de pertorbacions. En presència d'escenaris arriscats, es proposa el càlcul de conjunts controlables dependents de seqüències per als quals existeix una llei de control tal que el sistema pot ser conduït a l'origen, solament per a una realització en particular de l'incertesa, pertorbacions, etc. Per a estats pertanyents a aquests conjunts, es proposa una llei de control (òptima per al cas de sistemes lliures de pertorbacions i, subóptima per a sistemes pertorbats) capaç de dirigir el sistema cap a l'origen amb una probabilitat menor del 100%, aquesta probabilitat és denotada com a cota de confiabilitat. Sistemes inestables en llaç tancat tenen cota de confiabilitat igual a zero, per tant es dissenya un algoritme que determina el temps mitjà per a fallar. En aquest context, una fallada s'entén com la violació de les restriccions en els estats i/o entrades del procés en algun instant de temps futur.Hernández Mejías, MA. (2016). CONTROL PREDICTIVO BASADO EN ESCENARIOS PARA SISTEMAS LINEALES CON SALTOS MARKOVIANOS [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/68512TESI

Discrete-time Contraction Analysis and Controller Design for Nonlinear Processes

Shifting away from the traditional mass production approach, the process industry is moving towards more agile, cost-effective and dynamic process operation (next-generation smart plants). This warrants the development of control systems for nonlinear chemical processes to be capable of tracking time-varying setpoints to produce products with different specifications as per market demand and deal with variations in the raw materials and utility (e.g., energy). This thesis aims to develop controllers to achieve time-varying setpoints tracking using contraction theory. Through the differential dynamic system framework, the contraction conditions for discrete-time systems, which ensure the exponential convergence between system responses and feasible time-varying references, are derived. The discrete-time differential dissipativity condition is further developed, which can be used for disturbance rejection control designs. Computationally tractable equivalent conditions are then derived and additionally transformed into an Sum of Squares programming problem, such that a discrete-time control contraction metric and stabilising feedback controller can be jointly obtained. Synthesis and implementation details of the resulting contraction-based controller are provided, which can achieve offset-free tracking of feasible time-varying references. To do contraction analysis and control design for systems with uncertainties, which are often complex and difficult, neural networks are used. It involves training and constructing a neural network embedded contraction-based controller. Learning algorithms of uncertain system model parameters are developed. The resulting control scheme is capable of achieving efficient offset-free tracking of time-varying references, with a full range of model uncertainties, without the need for controller structure redesign as the reference or uncertain parameter changes. This neural network based approach also ensures process stability during online simultaneous control and learning of uncertain parameters. To further improve the economics of contraction-based controller, a nonlinear model predictive control approach is developed. Contraction condition is imposed as a constraint on the optimisation problem for model predictive control with an economic cost function, utilising Riemannian weighted graphs and shortest path techniques. The result is a reference flexible and fast optimal controller that can trade off between the rate of target trajectory convergence and economic benefit (away from the desired process objective)

Distributed Model Predictive Control for Reconfigurable Large-Scale Systems

Large-scale Systems are gaining more importance in the modern world requiring flexible techniques capable of handling interactions. This thesis is concerned with the development of suitable algorithms based on Model Predictive Control (MPC) that guarantee stability, recursive feasibility and constraint satisfaction. In the first part of this thesis, the main properties and control challenges for controlling an Large-Scale System are brought together, and the main distributed approaches for solving these problems are surveyed. Also, two novel Distributed MPC algorithms are presented. A non-centralised approach to the output-feedback variant of tube-based model predictive control of dynamically coupled linear time-invariant systems with shared constraints. A tube-based algorithm capable of handling the interactions–not rejecting them– that replaces the conventional linear disturbance rejection controller with a second MPC controller, as is done in tube-based nonlinear MPC. Following this, a smart-grids application of the developed algorithm is presented to solve the load frequency control for a power network. The approach achieves guaranteed constraint satisfaction, the recursive feasibility of the MPC problems and stability while maintaining on-line complexity similar to conventional MPC. The second part of the thesis covers reconfigurable distributed MPC. Two novel approaches are considered: a nominal MPC methodology that incorporates information of external disturbances, and a coalitional approach for robust distributed MPC. The first approach uses available disturbance predictions within a nominal model predictive control formulation is studied. The main challenge that arises is the loss of recursive feasibility and stability guarantees when a disturbance, which may change from time step to time step, is resent in the model and on the system. We show how standard stabilising terminal conditions may be modified to account for the use of disturbances in the prediction model. Robust stability and feasibility are established under the assumption that the disturbance change across sampling instances is limited. The proposed coalitional approach to robust Distributed MPC aims to tackle the existing trade-off between communication and performance in Large-Scale System by exploiting the different network topologies of system dynamics. The algorithm employs a method to switch between topologies using a multi-rate control approach. The optimal topology selection problem is solved using a consensus approach appropriately constrained to reduce the effects of any combinatorial explosion. The robust control algorithm is capable of recomputing the necessary parameters online to readjust to new partitions. Robust constraint satisfaction, recursive and stability are guaranteed by the proposed algorithm