Parallel discrete differential dynamic programming for multireservoir

Author name used in this publication: Chau, Kwok-Wing.2014-2015 > Academic research: refereed > Publication in refereed journalAccepted ManuscriptPublishe

Improving the multi-objective evolutionary optimization algorithm for hydropower reservoir operations in the California Oroville-Thermalito complex

This study demonstrates the application of an improved Evolutionary optimization Algorithm (EA), titled Multi-Objective Complex Evolution Global Optimization Method with Principal Component Analysis and Crowding Distance Operator (MOSPD), for the hydropower reservoir operation of the Oroville-Thermalito Complex (OTC) - a crucial head-water resource for the California State Water Project (SWP). In the OTC's water-hydropower joint management study, the nonlinearity of hydropower generation and the reservoir's water elevation-storage relationship are explicitly formulated by polynomial function in order to closely match realistic situations and reduce linearization approximation errors. Comparison among different curve-fitting methods is conducted to understand the impact of the simplification of reservoir topography. In the optimization algorithm development, techniques of crowding distance and principal component analysis are implemented to improve the diversity and convergence of the optimal solutions towards and along the Pareto optimal set in the objective space. A comparative evaluation among the new algorithm MOSPD, the original Multi-Objective Complex Evolution Global Optimization Method (MOCOM), the Multi-Objective Differential Evolution method (MODE), the Multi-Objective Genetic Algorithm (MOGA), the Multi-Objective Simulated Annealing approach (MOSA), and the Multi-Objective Particle Swarm Optimization scheme (MOPSO) is conducted using the benchmark functions. The results show that best the MOSPD algorithm demonstrated the best and most consistent performance when compared with other algorithms on the test problems. The newly developed algorithm (MOSPD) is further applied to the OTC reservoir releasing problem during the snow melting season in 1998 (wet year), 2000 (normal year) and 2001 (dry year), in which the more spreading and converged non-dominated solutions of MOSPD provide decision makers with better operational alternatives for effectively and efficiently managing the OTC reservoirs in response to the different climates, especially drought, which has become more and more severe and frequent in California

Design of optimal reservoir operating rules in large water resources systems combining stochastic programming, fuzzy logic and expert criteria

Given the high degree of development of hydraulic infrastructure in the developed countries, and with the increasing opposition to constructing new facilities in developing countries, the focus of water resource system analysis has turned into defining adequate operation strategies. Better management is necessary to cope with the challenge of supplying increasing demands and conflicts on water allocation while facing climate change impacts. To do so, a large set of mathematical simulation and optimization tools have been developed. However, the real application of these techniques is still limited. One of the main lines of research to fix this issue regards to the involvement of experts' knowledge in the definition of mathematical algorithms. To define operating rules in a way in which system operators could rely, their expert knowledge should be fully accounted and merged with the results from mathematical algorithms. This thesis develops a methodological framework and the required tools to improve the operation of large-scale water resource systems. In such systems, decision-making processes are complex and supported, at least partially, by the expert knowledge of decision-makers. This importance of expert judgment in the operation strategies requires mathematical tools able to embed and combine it with optimization algorithms. The methods and tools developed in this thesis rely on stochastic programming, fuzzy logic and the involvement of system operators during the whole rule-defining process. An extended stochastic programming algorithm, able to be used in large-scale water resource systems including stream-aquifer interactions, has been developed (the CSG-SDDP). The methodological framework proposed uses fuzzy logic to capture the expert knowledge in the definition of optimal operating rules. Once the current decision-making process is fairly reproduced using fuzzy logic and expert knowledge, stochastic programming results are introduced and thus the performance of the rules is improved. The framework proposed in this thesis has been applied to the Jucar river system (Eastern Spain), in which scarce resources are allocated following complex decision-making processes. We present two applications. In the first one, the CSG-SDDP algorithm has been used to define economically-optimal conjunctive use strategies for a joint operation of reservoirs andaquifers. In the second one, we implement a collaborative framework to couple historical records with expert knowledge and criteria to define a decision support system (DSS) for the seasonal operation of the reservoirs of the Jucar River system. The co-developed DSS tool explicitly reproduces the decision-making processes and criteria considered by the system operators. Two fuzzy logic systems have been developed and linked with this purpose, as well as with fuzzy regressions to preview future inflows. The DSS developed was validated against historical records. The developed framework offers managers a simple way to define a priori suitable decisions, as well as to explore the consequences of any of them. The resulting representation has been then combined with the CSG-SDDP algorithm in order to improve the rules following the current decision-making process. Results show that reducing pumping from the Mancha Oriental aquifer would lead to higher systemwide benefits due to increased flows by stream-aquifer interaction. The operating rules developed successfully combined fuzzy logic, expert judgment and stochastic programming, increasing water allocations to the demands by changing the way in which Alarcon, Contreras and Tous are balanced. These rules follow the same decision-making processes currently done in the system, so system operators would feel familiar with them. In addition, they can be contrasted with the current operating rules to determine what operation options can be coherent with the current management and, at the same time, achieve an optimal operationDado el alto número de infraestructuras construidas en los países desarrollados, y con una oposición creciente a la construcción de nuevas infraestructuras en los países en vías de desarrollo, la atención del análisis de sistemas de recursos hídricos ha pasado a la definición de reglas de operación adecuadas. Una gestión más eficiente del recurso hídrico es necesaria para poder afrontar los impactos del cambio climático y de la creciente demanda de agua. Para lograrlo, un amplio abanico de herramientas y modelos matemáticos de optimización se han desarrollado. Sin embargo, su aplicación práctica en la gestión hídrica sigue siendo limitada. Una de las más importantes líneas de investigación para solucionarlo busca la involucración de los expertos en la definición de dichos modelos matemáticos. Para definir reglas de operación en las cuales los gestores confíen, es necesario tener en cuenta su criterio experto y combinarlo con algoritmos de optimización. La presente tesis desarrolla una metodología, y las herramientas necesarias para aplicarla, con el fin de mejorar la operación de sistemas complejos de recursos hídricos. En éstos, los procesos de toma de decisiones son complicados y se sustentan, al menos en parte, en el juicio experto de los gestores. Esta importancia del criterio de experto en las reglas de operación requiere herramientas matemáticas capaces de incorporarlo en su estructura y de unirlo con algoritmos de optimización. Las herramientas y métodos desarrollados se basan en la optimización estocástica, en la lógica difusa y en la involucración de los expertos durante todo el proceso. Un algoritmo estocástico extendido, capaz de ser usado en sistemas complejos con interacciones río-acuífero se ha desarrollado (el CSG-SDDP). La metodología definida usa lógica difusa para capturar el criterio de experto en la definición de reglas óptimas. En primer lugar se reproducen los procesos de toma de decisiones actuales y, tras ello, el algoritmo de optimización estocástica se emplea para mejorar las reglas previamente obtenidas. La metodología propuesta en esta tesis se ha aplicado al sistema Júcar (Este de España), en el que los recursos hídricos son gestionados de acuerdo a complejos procesos de toma de decisiones. La aplicación se ha realizado de dos formas. En la primera, el algoritmo CSG-SDDP se ha utilizado para definir una estrategia óptima para el uso conjunto de embalses y acuíferos. En la segunda, la metodología se ha usado para reproducir las reglas de operación actuales en base a criterio de expertos. La herramienta desarrollada reproduce de forma explícita los procesos de toma de decisiones seguidos por los operadores del sistema. Dos sistemas lógicos difusos se han empleado e interconectado con este fin, así como regresiones difusas para predecir aportaciones. El Sistema de Ayuda a la Decisión (SAD) creado se ha validado comparándolo con los datos históricos. La metodología desarrollada ofrece a los gestores una forma sencilla de definir decisiones a priori adecuadas, así como explorar las consecuencias de una decisión concreta. La representación matemática resultante se ha combinado entonces con el CSG-SDDP para definir reglas óptimas que respetan los procesos actuales. Los resultados obtenidos indican que reducir el bombeo del acuífero de la Mancha Oriental conlleva una mejora en los beneficios del sistema debido al incremento de caudal por relación río-acuífero. Las reglas de operación han sido adecuadamente desarrolladas combinando lógica difusa, juicio experto y optimización estocástica, aumentando los suministros a las demandas mediante modificaciones el balance de Alarcón, Contreras y Tous. Estas reglas siguen los procesos de toma de decisiones actuales en el Júcar, por lo que pueden resultar familiares a los gestores. Además, pueden compararse con las reglas de operación actuales para establecer qué decisiones entreDonat l'alt nombre d'infraestructures construïdes en els països desenrotllats, i amb una oposició creixent a la construcció de noves infraestructures en els països en vies de desenrotllament, l'atenció de l'anàlisi de sistemes de recursos hídrics ha passat a la definició de regles d'operació adequades. Una gestió més eficient del recurs hídric és necessària per a poder afrontar els impactes del canvi climàtic i de la creixent demanda d'aigua. Per a aconseguir-ho, una amplia selecció de ferramentes i models matemàtics d'optimització s'han desenrotllat. No obstant això, la seua aplicació pràctica en la gestió hídrica continua sent limitada. Una de les més importants línies d'investigació per a solucionar-ho busca la col·laboració activa dels experts en la definició dels models matemàtics. Per a definir regles d'operació en les quals els gestors confien, és necessari tindre en compte el seu criteri expert i combinar-ho amb algoritmes d'optimització. La present tesi desenrotlla una metodologia, i les ferramentes necessàries per a aplicar-la, amb la finalitat de millorar l'operació de sistemes complexos de recursos hídrics. En estos, els processos de presa de decisions són complicats i se sustenten, almenys en part, en el juí expert dels gestors. Esta importància del criteri d'expert en les regles d'operació requereix ferramentes matemàtiques capaces d'incorporar-lo en la seua estructura i d'unir-lo amb algoritmes d'optimització. Les ferramentes i mètodes desenrotllats es basen en l'optimització estocàstica, en la lògica difusa i en la col·laboració activa dels experts durant tot el procés. Un algoritme estocàstic avançat, capaç de ser usat en sistemes complexos amb interaccions riu-aqüífer, s'ha desenrotllat (el CSG-SDDP) . La metodologia definida utilitza lògica difusa per a capturar el criteri d'expert en la definició de regles òptimes. En primer lloc es reprodueixen els processos de presa de decisions actuals i, després d'això, l'algoritme d'optimització estocàstica s'empra per a millorar les regles prèviament obtingudes. La metodologia proposada en esta tesi s'ha aplicat al sistema Xúquer (Est d'Espanya), en el que els recursos hídrics són gestionats d'acord amb complexos processos de presa de decisions. L'aplicació s'ha realitzat de dos formes. En la primera, l'algoritme CSG-SDDP s'ha utilitzat per a definir una estratègia òptima per a l'ús conjunt d'embassaments i aqüífers. En la segona, la metodologia s'ha usat per a reproduir les regles d'operació actuals basant-se en criteri d'experts. La ferramenta desenvolupada reprodueix de forma explícita els processos de presa de decisions seguits pels operadors del sistema. Dos sistemes lògics difusos s'han empleat i interconnectat amb este fi, al igual què regressions difuses per preveure cabdals. El Sistema d'Ajuda a la Decisió (SAD) creat s'ha validat comparant-lo amb les dades històriques. La metodologia desenvolupada ofereix als gestors una manera senzilla de definir decisions a priori adequades, així com per explorar les conseqüències d'una decisió concreta. La representació matemàtica resultant s'ha combinat amb el CSG-SDDP per a definir regles òptimes que respecten els processos actuals. Els resultats obtinguts indiquen que reduir el bombament de l'aqüífer de la Mancha Oriental comporta una millora en els beneficis del sistema a causa de l'increment de l'aigua per relació riu-aqüífer. Les regles d'operació han sigut adequadament desenrotllades combinant lògica difusa, juí expert i optimització estocàstica, augmentant els subministres a les demandes per mitjà de modificacions del balanç d'Alarcón, Contreras i Tous. Estes regles segueixen els processos de presa de decisions actuals en el Xúquer, per la qual cosa poden resultar familiars als gestors. A més, poden comparar-se amb les regles d'operació actuals per a establir quines decisions entre les possibles serien coherentsMacián Sorribes, H. (2017). Design of optimal reservoir operating rules in large water resources systems combining stochastic programming, fuzzy logic and expert criteria [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/82554TESI

Stochastic Dynamic Programming Applied to Hydrothermal Power Systems Operation Planning Based on the Convex Hull Algorithm

This paper presents a new approach for the expected cost-to-go functions modeling used in the stochastic dynamic programming (SDP) algorithm. The SDP technique is applied to the long-term operation planning of electrical power systems. Using state space discretization, the Convex Hull algorithm is used for constructing a series of hyperplanes that composes a convex set. These planes represent a piecewise linear approximation for the expected cost-to-go functions. The mean operational costs for using the proposed methodology were compared with those from the deterministic dual dynamic problem in a case study, considering a single inflow scenario. This sensitivity analysis shows the convergence of both methods and is used to determine the minimum discretization level. Additionally, the applicability of the proposed methodology for two hydroplants in a cascade is demonstrated. With proper adaptations, this work can be extended to a complete hydrothermal system

Short term management of hydro-power system using reinforcement learning

The fundamental objective in operation of reservoir complex is to specify an optimal decision policy so that it can maximize the expected value of reward function over the planning horizon. This control problem becomes more challenging as a result of existing different sources of uncertainties that reservoir planner needs to deal with. Usually, a trade-off exists between a value of water in storage and the electricity production. The function on the side of the value of water is uncertain and nonlinear in the reservoir management problem and it heavily depends on storage of reservoir and storage of other reservoirs as well. The challenging task is then how to solve this large-scale multireservoir problem under the presence of several uncertainties. In this thesis, the integration of a novel approach known as Reinforcement Learning (RL) is presented in order to provide an efficient optimization of a large-scale hydroelectric power system. RL is a branch of artificial intelligence method that presents several key benefits in treating problems that are too large to be handled by traditional dynamic programming techniques. In this approach, an agent tries to learn the optimal decision continuously so as to maximize the reward function based on interacting with the environment. This study presents the major concepts and computational aspects of using RL for the short-term planning problem of multireservoir system. The developed reinforcement learning based optimization model was successfully implemented on the Hydro-Quebec multireservoir complex located at the Rivière Romaine, north of the municipality of Havre-Saint-Pierre on the north shore of the St. Lawrence. This model was subsequently used to obtain optimal water release policies for the previously-mentioned reservoir complex. The output of the designed model was compared to the conventional optimization methods known as deterministic dynamic programming. The results show that the RL model is much more efficient and reliable in solving large-scale reservoir operations problems and can give a very good approximate solution to this complex problem

Optimal weekly releases from a multireservoir hydropower system

The operation of a multi-unit electric energy generating system is studied under certain and uncertain future inflow conditions. The generating units include thermoplants, hydroplants with regulating reservoir and run-of-river hydroplants. The objective is to minimize the expected cost of the operation of the system while meeting a previously defined energy demand. A case study is formulated based on the electric energy generating system of the South of Brazil. The system is composed of 6 hydroplants with regulating reservoirs, 2 run-of-river hydroplants, and 8 thermoplants. In order to obtain a better insight into the nature and peculiarities of the system's operation it is initially studied considering the future to be deterministic. An aggregation-optimization-disaggregation procedure is proposed to identify a near optimal solution while reducing substantially the computational effort. This consists of the development of an aggregated representation of the system composed of a hypothetical and unique reservoir with overall energy inflows and releases. Optimal operation of the aggregated system is determined by a new and efficient optimization technique specifically developed for this problem. A disaggregation procedure defines the operation of each system's unit given the operation of the aggregated system. The procedure is based on a heuristic approach that has as a main objective to minimize water spills. An aggregated representation of the system is again adopted for the definition of optimal strategy of operation when the future inflows are uncertain. The characteristics of operation of each reservoir are introduced into the aggregated formulation utilizing the peculiarities of the optimal deterministic operation. A modification of Massé's Chain of Marginal Expectations is used in the computations. The resultant strategy of operation can be presented as a function of aggregated values of energy storage and inflow. The strategy explicitly considers the autocorrelation of aggregated energy inflows. The strategy also implicitly accounts for the cross-correlations among the energy storages and inflows to each reservoir. Finally, a substantial part of the autocorrelation of the energy inflows and storages in each reservoir is indirectly considered in the strategy. Theoretical significance of the strategy is obtained without burdensome computational effort

Application of dynamic programming for the analysis of complex water resources systems : a case study on the Mahaweli River basin development in Sri Lanka

The technique of Stochastic Dynamic Programming (SDP) is ideally suited for operation policy analyses of water resources systems. However SDP has a major drawback which is appropriately termed as its "curse of dimensionality".Aggregation/Disaggregation techniques based on SDP and simulation are presented to analyze a complex water resources system. The system under consideration serves two major purposes: hydropower generation and irrigation. The identification of subsystems by their functional and physical characteristics was an important first step in the analysis. Subsequently each subsystem is represented by a hypothetical composite reservoir to arrive at an operation policy for the interface point of the subsystems. A more detailed analysis which considers the real configurations of the subsystems is performed by following this operation policy of the interface point. Two approaches: sequential optimization and iterative optimization are presented. In these approaches, each subsystem is individually analyzed using two-reservoir SDP models.The applicability of an Implicit Stochastic Approach in which the operation of the system is optimized for a number of deterministic hydrologic data series is also investigated. To complement the aggregation technique of the Composite Reservoir, subsequent disaggregation techniques are proposed. Three different techniques: (1) A statistical disaggregation, (2) An optimization/simulation-based technique, and (3) The disaggregation of the composite policy in the actual operation by incorporating a single-time-step optimization are tested.The accuracy of the sequential and iterative optimization approaches are evaluated by applying them to a subsystem of three reservoirs in a cascade for which the deterministic optimum pattern is also determined by an Incremental Dynamic Programming (IDP) model. In the case of the Implicit Stochastic Approach, the results are compared with the results of the explicit SDP approach and the deterministic optimum operation pattern, in addition to the historical operation pattern of the system. The results of the Composite Policy Disaggregation techniques are compared to the results obtained by real multireservoir optimizations carried out by the use of explicit SDP models

The value of hydrological information in multireservoir systems operation

La gestion optimale d’un système hydroélectrique composé de plusieurs réservoirs est un problème multi-étapes complexe de prise de décision impliquant, entre autres, (i) un compromis entre les conséquences immédiates et futures d’une décision, (ii) des risques et des incertitudes importantes, et (iii) de multiple objectifs et contraintes opérationnelles. Elle est souvent formulée comme un problème d’optimisation, mais il n’existe pas, à ce jour, de technique de référence même si la programmation dynamique (DP) a été souvent utilisée. La formulation stochastique de DP (SDP) permet la prise en compte explicite de l’incertitude entourant les apports hydrologiques futurs. Différentes approches ont été développées pour incorporer des informations hydrologiques et climatiques autres que les apports. Ces études ont révélé un potentiel d’amélioration des politiques de gestion proposées par les formulations SDP. Cependant, ces formulations sont applicables aux systèmes de petites tailles en raison de la célèbre « malédiction de la dimensionnalité ». La programmation dynamique stochastique duale (SDDP) est une extension de SDP développée dans les années 90. Elle est l’une des rares solutions algorithmiques utilisées pour déterminer les politiques de gestion des systèmes hydroélectriques de grande taille. Dans SDDP, l’incertitude hydrologique est capturée à l’aide d’un modèle autorégressif avec corrélation spatiale des résidus. Ce modèle analytique permet d’obtenir certains des paramètres nécessaires à l’implémentation de la technique d’optimisation. En pratique, les apports hydrologiques peuvent être influencés par d’autres variables observables, telles que l’équivalent de neige en eau et / ou la température de la surface des océans. La prise en compte de ces variables, appelées variables exogènes, permet de mieux décrire les processus hydrologiques et donc d’améliorer les politiques de gestion des réservoirs. L’objectif principal de ce doctorat est d’évaluer la valeur économique des politiques de gestion proposées par SDDP et ce pour diverses informations hydro-climatiques. En partant d’un modèle SDDP dans lequel la modélisation hydrologique est limitée aux processus Makoviens, la première activité de recherche a consisté à augmenter l’ordre du modèle autorégressif et à adapter la formulation SDDP. La seconde activité fut dédiée à l’incorporation de différentes variables hydrologiques exogènes dans l’algorithme SDDP. Le système hydroélectrique de Rio Tinto (RT) situé dans le bassin du fleuve Saguenay-Lac-Saint-Jean fut utilisé comme cas d’étude. Étant donné que ce système n’est pas capable de produire la totalité de l’énergie demandée par les fonderies pour assurer pleinement la production d’aluminium, le modèle SDDP a été modifié de manière à considérer les décisions de gestion des contrats avec Hydro Québec. Le résultat final est un système d’aide à la décision pour la gestion d’un large portefeuille d’actifs physiques et financiers en utilisant diverses informations hydro-climatiques. Les résultats globaux révèlent les gains de production d’énergie auxquels les opérateurs peuvent s’attendre lorsque d’autres variables hydrologiques sont incluses dans le vecteur des variables d’état de SDDP.The optimal operation of a multireservoir hydroelectric system is a complex, multistage, stochastic decision-making problem involving, among others, (i) a trade-off between immediate and future consequences of a decision, (ii) considerable risks and uncertainties, and (iii) multiple objectives and operational constraints. The reservoir operation problem is often formulated as an optimization problem but not a single optimization approach/algorithm exists. Dynamic programming (DP) has been the most popular optimization technique applied to solve the optimization problem. The stochastic formulation of DP (SDP) can be performed by explicitly considering streamflow uncertainty in the DP recursive equation. Different approaches to incorporate more hydrologic and climatic information have been developed and have revealed the potential to enhance SDP- derived policies. However, all these techniques are limited to small-scale systems due to the so-called curse of dimensionality. Stochastic Dual Dynamic Programming (SDDP), an extension of the traditional SDP developed in the 90ies, is one of the few algorithmic solutions used to determine the operating policies of large-scale hydropower systems. In SDDP the hydrologic uncertainty is captured through a multi-site periodic autoregressive model. This analytical linear model is required to derive some of the parameters needed to implement the optimization technique. In practice, reservoir inflows can be affected by other observable variables, such snow water equivalent and/or sea surface temperature. These variables, called exogenous variables, can better describe the hydrologic processes, and therefore enhance reservoir operating policies. The main objective of this PhD is to assess the economic value of SDDP-derived operating policies in large-scale water systems using various hydro-climatic information. The first task focuses on the incorporation of the multi-lag autocorrelation of the hydrologic variables in the SDDP algorithm. Afterwards, the second task is devoted to the incorporation of different exogenous hydrologic variables. The hydroelectric system of Rio Tinto (RT) located in the Saguenay-Lac-Saint-Jean River Basin is used as case study. Since, RT’s hydropower system is not able to produce the entire amount of energy demanded at the smelters to fully assure the aluminum production, a portfolio of energy contacts with Hydro-Québec is available. Eventually, we end up with a decision support system for the management of a large portfolio of physical and financial assets using various hydro-climatic information. The overall results reveal the extent of the gains in energy production that the operators can expect as more hydrologic variables are included in the state-space vector