Contributions to distributed MPC: coalitional and learning approaches

A growing number of works and applications are consolidating the research area of distributed control with partial and varying communication topologies. In this context, many of the works included in this thesis focus on the so-called coalitional MPC. This approach is characterized by the dynamic formation of groups of cooperative MPC agents (referred to as coalitions) and seeks to provide a performance close to the centralized one with lighter computations and communication demands. The thesis includes a literature review of existing distributed control methods that boost scalability and flexibility by exploiting the degree of interaction between local controllers. Likewise, we present a hierarchical coalitional MPC for traffic freeways and new methods to address the agents' clustering problem, which, given its combinatoria! nature, becomes a key issue for the real-time implementation of this type of controller. Additionally, new theoretical results to provide this clustering strategy with robust and stability guarantees to track changing targets are included. Further works of this thesis focus on the application of learning techniques in distributed and decentralized MPC schemes, thus paving the way for a future extension to the coalitional framework. In this regard, we have focused on the use of neural networks to aid distributed negotiations, and on the development of a multi­ agent learning MPC based on a collaborative data collection

Multi-Agent Reinforcement Learning for Connected and Automated Vehicles Control: Recent Advancements and Future Prospects

Connected and automated vehicles (CAVs) have emerged as a potential solution to the future challenges of developing safe, efficient, and eco-friendly transportation systems. However, CAV control presents significant challenges, given the complexity of interconnectivity and coordination required among the vehicles. To address this, multi-agent reinforcement learning (MARL), with its notable advancements in addressing complex problems in autonomous driving, robotics, and human-vehicle interaction, has emerged as a promising tool for enhancing the capabilities of CAVs. However, there is a notable absence of current reviews on the state-of-the-art MARL algorithms in the context of CAVs. Therefore, this paper delivers a comprehensive review of the application of MARL techniques within the field of CAV control. The paper begins by introducing MARL, followed by a detailed explanation of its unique advantages in addressing complex mobility and traffic scenarios that involve multiple agents. It then presents a comprehensive survey of MARL applications on the extent of control dimensions for CAVs, covering critical and typical scenarios such as platooning control, lane-changing, and unsignalized intersections. In addition, the paper provides a comprehensive review of the prominent simulation platforms used to create reliable environments for training in MARL. Lastly, the paper examines the current challenges associated with deploying MARL within CAV control and outlines potential solutions that can effectively overcome these issues. Through this review, the study highlights the tremendous potential of MARL to enhance the performance and collaboration of CAV control in terms of safety, travel efficiency, and economy

TOWARDS OPTIMAL OPERATION AND CONTROL OF EMERGING ELECTRIC DISTRIBUTION NETWORKS

The growing integration of power-electronics converters enabled components causes low inertia in the evolving electric distribution networks, which also suffer from uncertainties due to renewable energy sources, electric demands, and anomalies caused by physical or cyber attacks, etc. These issues are addressed in this dissertation. First, a virtual synchronous generator (VSG) solution is provided for solar photovoltaics (PVs) to address the issues of low inertia and system uncertainties. Furthermore, for a campus AC microgrid, coordinated control of the PV-VSG and a combined heat and power (CHP) unit is proposed and validated. Second, for islanded AC microgrids composed of SGs and PVs, an improved three-layer predictive hierarchical power management framework is presented to provide economic operation and cyber-physical security while reducing uncertainties. This scheme providessuperior frequency regulation capability and maintains low system operating costs. Third, a decentralized strategy for coordinating adaptive controls of PVs and battery energy storage systems (BESSs) in islanded DC nanogrids is presented. Finally, for transient stability evaluation (TSE) of emerging electric distribution networks dominated by EV supercharging stations, a data-driven region of attraction (ROA) estimation approach is presented. The proposed data-driven method is more computationally efficient than traditional model-based methods, and it also allows for real-time ROA estimation for emerging electric distribution networks with complex dynamics

Coordinated Smart Home Thermal and Energy Management System Using a Co-simulation Framework

The increasing demand for electricity especially during the peak hours threaten the grid reliability. Demand response (DR), changing the load pattern of the consumer in response to system conditions, can decrease energy consumption during periods of high wholesale market price and also maintain system reliability. Residential homes consume 38% of the total electric energy in the U.S., making them promising for DR participation. Consumers can be motivated to participate in DR programs by providing incentives (incentive-based DR), or by introducing a time-varying tariff for electricity consumption (price-based DR). A home energy management system (HEMS), an automated system which can alter the residential consumer’s energy consumption pattern based on the price of electricity or financial incentives, enables the consumers to participate in such DR programs. HEMS also should consider consumer comfort during the scheduling of the heating, ventilation, and air conditioning (HVAC) and other appliances. As internal heat gain of appliances and people have a significant effect in the HVAC energy consumption, an integrated HVAC and appliance scheduling are necessary to properly evaluate potential benefits of HEMS. This work presents the formulation of HEMS considering combined scheduling of HVAC and appliances in time-varying tariff. The HEMS also considers the consumer comfort for the HVAC and appliances while minimizing the total electricity cost. Similarly, the HEMS also considers the detailed building model in EnergyPlus, a building energy analysis tool, to evaluate the effectiveness of the HEMS. HEMS+, a communication interface to EnergyPlus, is designed to couple HEMS and EnergyPlus in this work. Furthermore, a co-simulation framework coupling EnergyPlus and GridLAB-D, a distribution system simulation tool, is developed. This framework enables incorporation of the controllers such as HEMS and aggregator, allowing controllers to be tested in detail in both building and power system domains. Lack of coordination among a large number of HEMS responding to same price signal results in peak more severe than the normal operating condition. This work presents an incentive-based hierarchical control framework for coordinating and controlling a large number of residential consumers’ thermostatically controlled loads (TCLs) such as HVAC and electric water heater (EWH). The potential market-level economic benefits of the residential demand reduction are also quantified

Coalitional model predictive control for systems of systems

An aspect so far rarely contemplated in distributed control problems is the explicit consideration of individual (local) interests of the components of a complex system. Indeed, the focus of the majority of the literature about distributed control has been the overall system performance. While on one hand this permitted to address fundamental properties of centralized control, such as system-wide optimality and stability, one the other hand it implied assuming unrestricted cooperation across local controllers. However, when dealing with multi-agent systems with a strong heterogeneous character, cooperation between the agents cannot be taken for granted (due to, for example, logistics, market competition), and selfish interests may not be neglected. Another critical point that must be kept into consideration is the diversity characterizing systems of systems (SoS), yielding very complex interactions between the agents involved (one example of such system is the smart grid). In order to tackle such inherent aspects of SoS, the research presented in this thesis has been concerned with the development of a novel framework, the coalitional control, that extends the scope of advanced control methods (in particular MPC) by drawing concepts from cooperative game theory that are suited for the inherent heterogeneity of SoS, providing as well an economical interpretation useful to explicitly take into account local selfish interests. Thus, coalitional control aims at governing the association/dissociation dynamics of the agents controlling the system, according to the expected benefits of their possible cooperation. From a control theoretical perspective, this framework is founded on the theory of switched systems and variable structure/topology networked systems, topics that are recently experiencing a renewed interest within the community. The main concepts and challenges in coalitional control, and the links with cooperative network game theory are presented in this document, tracing a path from model partitioning to the control schemes whose principles delineate the idea of coalitional control. This thesis focuses on two basic architectures: (i) a hierarchically supervised evolution of the coalitional structure, and (ii) a protocol for autonomous negotiation between the agents, with specific mechanisms for benefit redistribution, leading to the emergence of cooperating clusters.Premio Extraordinario de Doctorado U

Distributed Planning for Self-Organizing Production Systems

Für automatisierte Produktionsanlagen gibt es einen fundamentalen Tradeoff zwischen Effizienz und Flexibilität. In den meisten Fällen sind die Abläufe nicht nur durch den physischen Aufbau der Produktionsanlage, sondern auch durch die spezielle zugeschnittene Programmierung der Anlagensteuerung fest vorgegeben. Änderungen müssen aufwändig in einer Vielzahl von Systemen nachgezogen werden. Das macht die Herstellung kleiner Stückzahlen unrentabel. In dieser Dissertation wird ein Ansatz entwickelt, um eine automatische Anpassung des Verhaltens von Produktionsanlagen an wechselnde Aufträge und Rahmenbedingungen zu erreichen. Dabei kommt das Prinzip der Selbstorganisation durch verteilte Planung zum Einsatz. Die aufeinander aufbauenden Ergebnisse der Dissertation sind wie folgt: 1. Es wird ein Modell von Produktionsanlagen entwickelt, dass nahtlos von der detaillierten Betrachtung physikalischer Produktionsprozesse bis hin zu Lieferbeziehungen zwischen Unternehmen skaliert. Im Vergleich zu existierenden Modellen von Produktionsanlagen werden weniger limitierende Annahmen gestellt. In diesem Sinne ist der Modellierungsansatz ein Kandidat für eine häufig geforderte "Theorie der Produktion". 2. Für die so modellierten Szenarien wird ein Algorithmus zur Optimierung der nebenläufigen Abläufe entwickelt. Der Algorithmus verbindet Techniken für die kombinatorische und die kontinuierliche Optimierung: Je nach Detailgrad und Ausgestaltung des modellierten Szenarios kann der identische Algorithmus kombinatorische Fertigungsfeinplanung (Scheduling) vornehmen, weltweite Lieferbeziehungen unter Einbezug von Unsicherheiten und Risiko optimieren und physikalische Prozesse prädiktiv regeln. Dafür werden Techniken der Monte-Carlo Baumsuche (die auch bei Deepminds Alpha Go zum Einsatz kommen) weiterentwickelt. Durch Ausnutzung zusätzlicher Struktur in den Modellen skaliert der Ansatz auch auf große Szenarien. 3. Der Planungsalgorithmus wird auf die verteilte Optimierung durch unabhängige Agenten übertragen. Dafür wird die sogenannte "Nutzen-Propagation" als Koordinations-Mechanismus entwickelt. Diese ist von der Belief-Propagation zur Inferenz in Probabilistischen Graphischen Modellen inspiriert. Jeder teilnehmende Agent hat einen lokalen Handlungsraum, in dem er den Systemzustand beobachten und handelnd eingreifen kann. Die Agenten sind an der Maximierung der Gesamtwohlfahrt über alle Agenten hinweg interessiert. Die dafür notwendige Kooperation entsteht über den Austausch von Nachrichten zwischen benachbarten Agenten. Die Nachrichten beschreiben den erwarteten Nutzen für ein angenommenes Verhalten im Handlungsraum beider Agenten. 4. Es wird eine Beschreibung der wiederverwendbaren Fähigkeiten von Maschinen und Anlagen auf Basis formaler Beschreibungslogiken entwickelt. Ausgehend von den beschriebenen Fähigkeiten, sowie der vorliegenden Aufträge mit ihren notwendigen Produktionsschritten, werden ausführbare Aktionen abgeleitet. Die ausführbaren Aktionen, mit wohldefinierten Vorbedingungen und Effekten, kapseln benötigte Parametrierungen, programmierte Abläufe und die Synchronisation von Maschinen zur Laufzeit. Die Ergebnisse zusammenfassend werden Grundlagen für flexible automatisierte Produktionssysteme geschaffen -- in einer Werkshalle, aber auch über Standorte und Organisationen verteilt -- welche die ihnen innewohnenden Freiheitsgrade durch Planung zur Laufzeit und agentenbasierte Koordination gezielt einsetzen können. Der Bezug zur Praxis wird durch Anwendungsbeispiele hergestellt. Die Machbarkeit des Ansatzes wurde mit realen Maschinen im Rahmen des EU-Projekts SkillPro und in einer Simulationsumgebung mit weiteren Szenarien demonstriert