Reinforcement Learning Policy Gradient Methods for Reservoir Operation Management and Control

Changes in demand, various hydrological inputs, and environmental stressors are among issues that water managers and policymakers face on a regular basis. These concerns have sparked interest in applying different techniques to determine reservoir operation policy and improve reservoir release decisions. As the resolution of the analysis rises, it becomes more difficult to effectively represent a real-world system using traditional approaches for determining the best reservoir operation policy. One of the challenges is the “curse of dimensionality,” which occurs when the discretization of the state and action spaces becomes finer or when more state or action variables are taken into account. Because of the dimensionality curse, the number of state-action variables is limited, rendering Dynamic Programming (DP) and Stochastic Dynamic Programming (SDP) ineffective in handling complex reservoir optimization issues. Deep Reinforcement Learning (DRL) is an intelligent approach to overcome the aforementioned curses of stochastic optimization of reservoir system planning. This study examined various novel DRL continuous-action policy gradient methods (PGMs), including Deep Deterministic Policy Gradients (DDPG), Twin Delayed DDPG (TD3), and two different versions of Soft Actor-Critic (SAC18 and SAC19) to identify optimal reservoir operation policy for the Folsom Reservoir located in California, US. The Folsom Reservoir supplies agricultural and municipal water, hydropower, environmental flows, and flood protection to the City of Sacramento. We concluded DRL methods release decisions with respect to these demands as well as by comparing the results to standard operating policy (SOP) and base conditions using different performance criteria and sustainability indices. TD3 and SAC methods have shown promising performance in providing optimal operation policy. Experiments on continuous-action spaces of reservoir operation policy decisions demonstrated that the DRL techniques could efficiently learn strategic policies in space with the curse of dimensionality and modeling

Natural and Technological Hazards in Urban Areas

Natural hazard events and technological accidents are separate causes of environmental impacts. Natural hazards are physical phenomena active in geological times, whereas technological hazards result from actions or facilities created by humans. In our time, combined natural and man-made hazards have been induced. Overpopulation and urban development in areas prone to natural hazards increase the impact of natural disasters worldwide. Additionally, urban areas are frequently characterized by intense industrial activity and rapid, poorly planned growth that threatens the environment and degrades the quality of life. Therefore, proper urban planning is crucial to minimize fatalities and reduce the environmental and economic impacts that accompany both natural and technological hazardous events

How does regulation affect innovation and technology change in the water sector in England and Wales?

This thesis examines the role of regulation in technological change in the water sector in England and Wales. Based on a combination of Social-Ecological Systems (SES) theory and the Multi-Level Perspective on technological transitions a Comparative Information-Graded Approach (CIGA) is developed in Part 1. As part of the CIGA, a series of tools is used for characterizing and evaluating the relationship between regulation and technology. In Part 2, the CIGA is applied to characterize the relationship between regulation and water innovation in England and Wales based on official publications, Environment Agency data, and interviews. In particular, 7 mechanisms are identified by which regulation affects innovation and 5 issues of trust negatively interact with innovation. As trust is established through these mechanisms, opportunities for innovation are at times sacrificed. Part 3 develops and analyses a set of models based on findings in Part 2. Dynamical systems and fictitious play analysis of a trustee game model of regulation exhibits cyclicality providing an explanation for observed cycles which create an inconsistent drive for innovation. Trustee and coordination models are evaluated in Chapter 7 highlighting how most tools struggle with the issue of technological lock-in. Chapter 8 develops a model of two innovators and a public good water technology over time, showing the role foresight plays in this context as well as the disincentive to develop it. Taken together, the CIGA characterization and modelling work provide a series of recommendations and insights into how the system of regulation affects technology change.Open Acces

A Data-driven Approach to Revenue Management Problem with Behavioral Considerations

RÉSUMÉ: Cette thèse porte sur l’étude de diverses méthodes avancées et orientées données pour résoudre la problématique de gestion de revenu plus souvent désigné par “Revenue Management” (RM). Nous nous intéressons à deux sous-problèmes du RM que sont la prévision de demande et le contrôle d’inventaire, poursuivons avec une revue de la littérature suivi d’une synthèse des contributions de la thèse. Nous commençons par une introduction générale sur la métodologie usuelle pour traiter la prédiction de la demande et les politiques optimales de contrôle d’inventaire. Nous présentons dans les trois chapitres suivants, nos travaux sur ces problématiques, chacun correspondant à un article soumis dans une revue internationale. Finalement nous concluons par des remarques sur le travail actuel et une discussion sur les possibles futurs travaux. Nous présentons maintenant brièvement les trois articles. Dans le premier article, nous nous intéressons à la prévision de la demande pour une importante compagnie ferroviaire. Pour cela, nous explorons diverses approches de prétraitement, apprentissage machine et sélection de caractéristiques des données. Comme cette prévision est utilisée pour différents objectifs, nous travaillons sur deux niveaux d’agrégation différents. La solution devant être industrialisable, nous mettons l’emphase sur la rapidité, la simplicité et la robustesse. Nous combinons alors des méthodes de l’état de l’art avec des techniques innovantes de construction des caractéristiques des données pour arriver à des résultats prometteurs. Bien que nous traitons la prévision de demande pour le domaine ferroviaire, nos résultats s’appliquent également aux autres domaines de transport et à l’hôtellerie. Dans le second article, nous considérons le problème du contrôle d’inventaire du RM sous comportement d’achat pour le domaine aérien avec une méthode d’apprentissage par renforcement du type “Deep Q-Network” (DQN). Par rapport aux approches traditionnelles en RM, DQN ne dépend pas d’une prévision de la demande pour retourner de bonnes décisions de contrôle. Il fonctionne en utilisant des données historiques et/ou une interaction directe avec les clients. Nous nous concentrons essentiellement sur l’aspect comportemental de notre modèle. Nous entraînons et évaluons notre solution avec des données synthétiques puis la comparons avec des méthodes tradionelles de RM sur des instances aériennes fournies par la littérature. Dans le troisième article, nous abordons des instances de taille plus importante pour des problèmes de RM que l’on retrouve en pratique. Nous proposons un algorithme “Action Generation” (AGen) à intégrer au DQN pour étendre son utilisation à des problèmes de plus grande taille de RM sans trop augmenter le coût de calcul. La motivation derrière cette approche vient d’une analyse des offres optimales à travers l’horizon de réservation qui montre qu’elles sont souvent les mêmes, nous les appelons alors “offres efficaces”. À partir de cette information nous pouvons considérablement réduire le temps de calcul dans les cas pratiques. AGen est un algorithme heuristique de type glouton qui mimique la génération de colonnes dans le but de générer ces “offres efficaces”. La combinaison de DQN et AGen donne des résultats prometteurs sur les problèmes de plus grandes tailles.---------ABSTRATC: This dissertation presents a systematic study of various data-driven advanced methodologies employed to solve a Revenue Management (RM) problem. We address two main modules within an RM system; namely, demand forecasting and inventory control. We start with a general introduction into the thesis and then proceed to overall methodology used to both predict customer demand and analyze the capacity control policies. The methodologies are explained in detail in the three following chapters each of which corresponds to an article already submitted to an international journal. Finally, we conclude with final remarks and discussions of implications for further work. Following is a brief explanation of each article. In the first article, we study a demand forecasting problem to be addressed for a major railway company. To do so, we explore various preprocessing, machine learning and feature engineering techniques. Moreover, the demand is estimated in two different aggregation levels of data in order to serve different purposes. To comply with the industry-specific requirements, the emphasis of our solution method is on speed, simplicity, and robustness. In this study, the use of state-of-the-art machine learning methods along with innovative feature construction techniques led to high quality results. Although railway industry is the representative of our problem, the studied demand forecasting approaches can easily be extended to other transportation industries or hospitality businesses. In the second article, we address a choice-based seat inventory control problem in airline industry using a deep reinforcement learning method named Deep Q-Network (DQN). In contrast to traditional RM techniques, DQN does not rely on predicted demand to make informed capacity control decisions. It operates using historical data and/or real-time interaction with customers. In this study, we mainly focus on the choice-based characteristic of our model. We train and evaluate our solution method with synthetic data and compare the final performance to those of well-known RM methods using common flight examples provided in the literature. In the third article, we tackle large-scale practical RM problems. We propose an “Action Generation” (AGen) algorithm to be integrated into DQN and extend its application to larger RM problems without incurring enormous computational costs. The analysis of the optimal offersets offered to customers throughout the booking horizon shows that only particular offersets (i.e., actions), which we call them “effective sets”, are repeatedly used. Thus, if we manage to develop a method to generate such actions, we will be able to substantially reduce the processing time in practical cases. AGen is a greedy heuristic algorithm that mimics the column generation algorithm [1] with the aim of generating “effective sets”. The AGen embedded DQN yields promising results in large-size network problems

Climate Change and Environmental Sustainability-Volume 2

Our world is facing many challenges, such as poverty, hunger, resource shortage, environmental degradation, climate change, and increased inequalities and conflicts. To address such challenges, the United Nations proposed the Sustainable Development Goals (SDG), consisting of 17 interlinked global goals, as the strategic blueprint of world sustainable development. Nevertheless, the implementation of the SDG framework has been very challenging and the COVID-19 pandemic has further impeded the SDG implementation progress. Accelerated efforts are needed to enable all stakeholders, ranging from national and local governments, civil society, private sector, academia and youth, to contribute to addressing this dilemma. This volume of the Climate Change and Environmental Sustainability book series aims to offer inspiration and creativity on approaches to sustainable development. Among other things, it covers topics of COVID-19 and sustainability, environmental pollution, food production, clean energy, low-carbon transport promotion, and strategic governance for sustainable initiatives. This book can reveal facts about the challenges we are facing on the one hand and provide a better understanding of drivers, barriers, and motivations to achieve a better and more sustainable future for all on the other. Research presented in this volume can provide different stakeholders, including planners and policy makers, with better solutions for the implementation of SDGs. Prof. Bao-Jie He acknowledges the Project NO. 2021CDJQY-004 supported by the Fundamental Research Funds for the Central Universities. We appreciate the assistance from Mr. Lifeng Xiong, Mr. Wei Wang, Ms. Xueke Chen and Ms. Anxian Chen at School of Architecture and Urban Planning, Chongqing University, China

Technology and Management for Sustainable Buildings and Infrastructures

A total of 30 articles have been published in this special issue, and it consists of 27 research papers, 2 technical notes, and 1 review paper. A total of 104 authors from 9 countries including Korea, Spain, Taiwan, USA, Finland, China, Slovenia, the Netherlands, and Germany participated in writing and submitting very excellent papers that were finally published after the review process had been conducted according to very strict standards. Among the published papers, 13 papers directly addressed words such as sustainable, life cycle assessment (LCA) and CO2, and 17 papers indirectly dealt with energy and CO2 reduction effects. Among the published papers, there are 6 papers dealing with construction technology, but a majority, 24 papers deal with management techniques. The authors of the published papers used various analysis techniques to obtain the suggested solutions for each topic. Listed by key techniques, various techniques such as Analytic Hierarchy Process (AHP), the Taguchi method, machine learning including Artificial Neural Networks (ANNs), Life Cycle Assessment (LCA), regression analysis, Strength–Weakness–Opportunity–Threat (SWOT), system dynamics, simulation and modeling, Building Information Model (BIM) with schedule, and graph and data analysis after experiments and observations are identified

International Conference on Civil Infrastructure and Construction (CIC 2020)

This is the proceedings of the CIC 2020 Conference, which was held under the patronage of His Excellency Sheikh Khalid bin Khalifa bin Abdulaziz Al Thani in Doha, Qatar from 2 to 5 February 2020. The goal of the conference was to provide a platform to discuss next-generation infrastructure and its construction among key players such as researchers, industry professionals and leaders, local government agencies, clients, construction contractors and policymakers. The conference gathered industry and academia to disseminate their research and field experiences in multiple areas of civil engineering. It was also a unique opportunity for companies and organizations to show the most recent advances in the field of civil infrastructure and construction. The conference covered a wide range of timely topics that address the needs of the construction industry all over the world and particularly in Qatar. All papers were peer reviewed by experts in their field and edited for publication. The conference accepted a total number of 127 papers submitted by authors from five different continents under the following four themes: Theme 1: Construction Management and Process Theme 2: Materials and Transportation Engineering Theme 3: Geotechnical, Environmental, and Geo-environmental Engineering Theme 4: Sustainability, Renovation, and Monitoring of Civil InfrastructureThe list of the Sponsors are listed at page 1