Distributed optimization for control and learning

Large scale multi-agent networked systems are becoming increasingly popular in industry and academia as they can be applied to represent systems in diverse application areas, such as intelligent surveillance and reconnaissance, mobile robotics, transportation networks and complex buildings. In such systems, issues related to control and learning have been significant technical challenges to affect system performance and overall cost. While centralized optimization approaches have been widely used by the engineering and computer science community, advanced and effective distributed optimization techniques have not been explored sufficiently and thoroughly in this regard. This study explores various categories of centralized and distributed optimization methods that have been applied or may be applicable for diverse engineering and science problems. The performance of centralized or distributed optimization schemes significantly depends on various factors including the types of objective functions, constraints, step sizes, and communication networks, etc. In this context, the focus of this dissertation is towards developing novel distributed optimization algorithms in order to solve challenging control and learning problems in various domains such as large-scale building energy systems and robotic networks. Specifically, we develop a generalized gossip-based subgradient method for solving distributed optimization problems in large-scale networked systems, e.g., larger-scale commercial building energy systems. Different from previous work, a user-defined control parameter is introduced to control a spectrum from globally optimal solution to suboptimal solutions and the trade-off between the solution accuracy and temporal convergence. We test and validate our proposed algorithm on a real testbed involving multiple zones incorporating a distributed control and sensing platform. In addition, we extend the distributed optimization to the deep learning area for solving an emerging topic, i.e., distributed deep learning, in fixed topology networks. While some previous work exists on this topic, the data parallelism and distributed computation are still not sufficiently explored. Therefore, we propose a class of distributed deep learning methods to tackle such issues by combining the consensus protocol and stochastic gradient descent approach. Moreover, to address the consensus-optimality trade-offs in distributed convex and nonconvex optimization, especially in deep learning when the training datasets for agents are non-balanced (non-iid), we propose and develop new approaches in this research, namely, incremental consensus-based distributed stochastic gradient descent and generalized consensus-based distributed (stochastic) gradient descent approach

Simulation-Based Approximate Policy Gradient and Its Building Control Application

The goal of this paper is to study the potential applicability of a stochastic approximationbased policy gradient method for optimal office building HVAC (Heating, Ventilation, and Air Conditioning) control systems. A real-world building thermal dynamics with occupant interactions is the main focus of this paper. It is a complex stochastic system in the sense that its statistical properties depend on its state variables. In this case, existing approaches, for instance, stochastic model predictive control methods, cannot be applied to optimal control designs. As a remedy, we approximate the gradient of the cost function using simulations and use a gradient descent type algorithm to design a suboptimal control policy. We assess its performance through a simulation study of building HVAC systems

Real-Time Data-Driven Optimization Algorithms for Modern Power Systems

Power systems have experienced significant transformations in recent years driven by the integration of new technologies, such as renewable generation and distributed energy resources (DERs). However, the variability of renewable generation and uncontrollable loads, along with the increasing grid-to-user interactions, pose severe challenges to the grid operation. Therefore, ensuring a reliable, safe, and optimal real-time operation of modern power systems requires new scalable optimization approaches that can cope with uncertainty and learn models in real-time. Towards this end, this dissertation presents contributions to the field of time-varying optimization, focusing on applications where renewable generation and controllable DERs interact with the grid. The dissertation is divided into two parts. Part I (Chapter 2 and 3) investigates time-varying optimization problems associated with a physical system and human-in-the-loop interactions. In this part, we study two settings. First, we consider a cost comprising a partially known time-varying function capturing performance metrics of the system, as well as unknown functions associated with users interacting with the physical system. Second, we feature an optimization problem where the objective is to minimize cost functions related to the individuals&rsquo; preferences, subject to timevarying constraints that capture the physical or operational limits of the network. Based on these time-varying optimization problems, we develop learning-based distributed online optimization algorithms. Our focus lies in the synthesis of first-order online methods, where feedback from the user is leveraged to learn the unknown functions concurrently with the execution of the online algorithm, and measurements of the output of the system are used to estimate the gradient and evaluate the partially known engineering functions. Part II (Chapter 4 - 7) focuses on optimization and control techniques for different applications where power systems interact with controllable DERs. First, we present an application of the theoretical framework of Part I to the real-time management of DERs. Second, we present a demand response application on commercial buildings. In this case, we introduce a predictive controller for a grid-interactive multi-zone building where the temperature dynamics are learned via a supervised learning technique. This controller uses the learned dynamics to solve a multi-objective problem to guarantee occupants&rsquo; comfort and energy efficiency during normal conditions and demand response events. Third, we study the estimation of sensitivity matrices in power grids with applications in transmission and distribution systems. By leveraging a low-rank approximation of certain classes of sensitivity matrices, we propose an online proximal-gradient algorithm based on a robust nuclear norm minimization problem to estimate linear sensitivities from measurements. Finally, we address a problem related to the charging schedule of electric vehicles (EVs). We consider a scenario where a ride-service provider utilizes a 100%-EV fleet to serve customers, while a power utility company aims to maximize the utilization of renewable generation at specific charging stations. To achieve this, we propose a novel mechanism that encourages EV charging during hours of high renewable generation while minimizing the impact on the quality of service for the ride-service provider.</p

Toward optimal operation of multienergy home-microgrids for power balancing in distribution networks: a model predictive control approach

The energy policy objectives of the German government regarding renewable energy sources and energy efficiency will lead to a significantly increase in the share of photovoltaics, storage systems, CHP plants, and heat pumps, especially at the distribution grid level. In the future, inside a household, such systems must be coordinated in such a way that they can respond to variable network conditions as a single flexible unit. This dissertation defines home-microgrid as a residential building with integrated distributed energy resources, and follows a bottom-up approach, based on the cellular approach, which aims at improving local balancing in low-voltage grids by using the flexibilities of home-microgrids. For this purpose, the dissertation develops optimization-based strategies for the coordination of multienergy home-microgrids, focusing on the use of model predictive control. The main core of the work is the formulation of the underlying optimization problems and the investigation of coordination strategies for interconnected home-microgrids. In this context, the work presents the use of the dual decomposition and the alternating direction method of multipliers for hierarchical-distributed coordination strategies. Finally, this dissertation introduces a framework for the co-simulation of electrical networks with penetration of multienergy home-microgrids

Open-Source, Open-Architecture SoftwarePlatform for Plug-InElectric Vehicle SmartCharging in California

This interdisciplinary eXtensible Building Operating System–Vehicles project focuses on controlling plug-in electric vehicle charging at residential and small commercial settings using a novel and flexible open-source, open-architecture charge communication and control platform. The platform provides smart charging functionalities and benefits to the utility, homes, and businesses.This project investigates four important areas of vehicle-grid integration research, integrating technical as well as social and behavioral dimensions: smart charging user needs assessment, advanced load control platform development and testing, smart charging impacts, benefits to the power grid, and smart charging ratepayer benefits

Co-design for Multi-subsystem and Vehicle Routing-and-Control Problems

Co-design refers to the process of integrating optimization of the physical design with a controller for a system. The challenge in co-design is that the optimization is simultaneously applied to both static/time-invariant (e.g., physical plant design) variables and dynamic/time-variant (e.g., state and control) variables, which can be coupled with each other. The objective of this dissertation is to explore new formulations and approaches in co-design for multi-subsystem and vehicle routing-and-control problems. Specifically, four research questions are considered and resolved. In Research Question 1 (RQ1), the critical issue is how to formulate a class of multi-subsystem co-design problems with convex physical design subproblems and linear quadratic regulator control subproblems, and construct a decentralized solution approach for such problems. In Research Question 2 (RQ2), solution methods for a broader class of multi-subsystem co-design problems than those considered in RQ1 are investigated. In Research Question 3 (RQ3), the question is whether, in the context of co-design, the combined routing and control costs of a fleet of vehicles can be improved if optimal control is introduced into the routing. Finally, an extension of RQ3 is considered in Research Question 4 (RQ4), where the possibility of constructing an integrated vehicle routing-and-control problem with load-dependent dynamics is investigated. Beyond the articles published by the author of this dissertation, the proposed research questions, models and methods presented have not been considered elsewhere in the literature

Provision of energy and regulation reserve services by buildings

Power consumption and generation in the electrical grid must be balanced at all times. This balance is maintained by the grid operator through the procurement of energy and regulation reserve services in the wholesale electricity market. Traditionally, these services could only be procured from generation resources. However, helped by the advances in the computational and communication infrastructure, the demand resources are increasingly being leveraged in this regard. In particular, the Heating, Ventilation and Air-Conditioning (HVAC) systems of buildings are gaining traction due to the consumption flexibility provided by their energy-shifting and fast-response capabilities. The provision of energy and regulation reserve services in the wholesale market, from the perspective of a typical building’s HVAC system, can be construed in terms of two synergistic problems: an hourly deterministic optimization problem, referred to as Scheduling Problem, and a real-time (seconds timescale) stochastic control problem, referred to as Deployment Problem. So far, the existing literature has synthesized the solutions of these two problems in a simplistic, sequential/iterative manner without employing an integrated approach that captures explicitly their cost and constraint interactions. Moreover, the deployment problem has only been solved with classical controllers which are not optimal, whereas the non-convexities in the scheduling problem have been addressed with methods that are sensitive to initialization. The current approaches therefore do not fully optimize the decisions of the two problems either individually or collectively, and hence do not fully exploit the HVAC resource. The goal of the proposed research is to have optimal decision-making across both the scheduling and deployment problems. Our approach proposes deriving an optimal control policy for the deployment problem, and expressing the corresponding expected sum of deployment costs over the hour (called ‘expected intra-hour costs’) as a closed-form analytic function of the scheduling decisions. The inclusion of these expected intra-hour costs into the scheduling problem allows the optimization of the hourly scheduling decisions, pursuant to the real-time use of the optimal deployment control policy. Thus, our approach captures the interaction of the two problems and optimizes decisions across timescales yielding a truly integrated solution. We investigate the estimation of the expected intra-hour costs (based on a myopic policy optimizing instantaneous tracking error and utility loss), and solve the integrated problem with tight relaxations, demonstrating the value and applicability of the approach. Further, we investigate the derivation of the optimal control policy for the deployment problem, formulating and solving it as discounted-cost infinite horizon Dynamic Program (DP) and Reinforcement Learning (RL) problems. We also characterize the optimal policy as a closed-form analytic mapping from state-space to action-space, and obtain the corresponding expected cost-to-go (a.k.a. expected intra-hour costs) as a closed-form analytic function of the scheduling decisions. We further illustrate that the optimal policy better captures the deployment costs compared to existing approaches. As such, our work represents a structured, interpretable and automated way for the end-to-end consideration of energy and regulation reserve market participation, and can be extended to other demand side resources

Leveraging deep reinforcement learning in the smart grid environment

L’apprentissage statistique moderne démontre des résultats impressionnants, où les or- dinateurs viennent à atteindre ou même à excéder les standards humains dans certaines applications telles que la vision par ordinateur ou les jeux de stratégie. Pourtant, malgré ces avancées, force est de constater que les applications fiables en déploiement en sont encore à leur état embryonnaire en comparaison aux opportunités qu’elles pourraient apporter. C’est dans cette perspective, avec une emphase mise sur la théorie de décision séquentielle et sur les recherches récentes en apprentissage automatique, que nous démontrons l’applica- tion efficace de ces méthodes sur des cas liés au réseau électrique et à l’optimisation de ses acteurs. Nous considérons ainsi des instances impliquant des unités d’emmagasinement éner- gétique ou des voitures électriques, jusqu’aux contrôles thermiques des bâtiments intelligents. Nous concluons finalement en introduisant une nouvelle approche hybride qui combine les performances modernes de l’apprentissage profond et de l’apprentissage par renforcement au cadre d’application éprouvé de la recherche opérationnelle classique, dans le but de faciliter l’intégration de nouvelles méthodes d’apprentissage statistique sur différentes applications concrètes.While modern statistical learning is achieving impressive results, as computers start exceeding human baselines in some applications like computer vision, or even beating pro- fessional human players at strategy games without any prior knowledge, reliable deployed applications are still in their infancy compared to what these new opportunities could fathom. In this perspective, with a keen focus on sequential decision theory and recent statistical learning research, we demonstrate efficient application of such methods on instances involving the energy grid and the optimization of its actors, from energy storage and electric cars to smart buildings and thermal controls. We conclude by introducing a new hybrid approach combining the modern performance of deep learning and reinforcement learning with the proven application framework of operations research, in the objective of facilitating seamlessly the integration of new statistical learning-oriented methodologies in concrete applications