Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning

Wastewater treatment plants are designed to eliminate pollutants and alleviate environmental pollution. However, the construction and operation of WWTPs consume resources, emit greenhouse gases (GHGs) and produce residual sludge, thus require further optimization. WWTPs are complex to control and optimize because of high nonlinearity and variation. This study used a novel technique, multi-agent deep reinforcement learning, to simultaneously optimize dissolved oxygen and chemical dosage in a WWTP. The reward function was specially designed from life cycle perspective to achieve sustainable optimization. Five scenarios were considered: baseline, three different effluent quality and cost-oriented scenarios. The result shows that optimization based on LCA has lower environmental impacts compared to baseline scenario, as cost, energy consumption and greenhouse gas emissions reduce to 0.890 CNY/m3-ww, 0.530 kWh/m3-ww, 2.491 kg CO2-eq/m3-ww respectively. The cost-oriented control strategy exhibits comparable overall performance to the LCA driven strategy since it sacrifices environmental bene ts but has lower cost as 0.873 CNY/m3-ww. It is worth mentioning that the retrofitting of WWTPs based on resources should be implemented with the consideration of impact transfer. Specifically, LCA SW scenario decreases 10 kg PO4-eq in eutrophication potential compared to the baseline within 10 days, while significantly increases other indicators. The major contributors of each indicator are identified for future study and improvement. Last, the author discussed that novel dynamic control strategies required advanced sensors or a large amount of data, so the selection of control strategies should also consider economic and ecological conditions

Management of Uncertainty by Statistical Process Control and a Genetic Tuned Fuzzy System

In food industry, bioprocesses like fermentation often are a crucial part of the manufacturing process and decisive for the final product quality. In general, they are characterized by highly nonlinear dynamics and uncertainties that make it difficult to control these processes by the use of traditional control techniques. In this context, fuzzy logic controllers offer quite a straightforward way to control processes that are affected by nonlinear behavior and uncertain process knowledge. However, in order to maintain process safety and product quality it is necessary to specify the controller performance and to tune the controller parameters. In this work, an approach is presented to establish an intelligent control system for oxidoreductive yeast propagation as a representative process biased by the aforementioned uncertainties. The presented approach is based on statistical process control and fuzzy logic feedback control. As the cognitive uncertainty among different experts about the limits that define the control performance as still acceptable may differ a lot, a data-driven design method is performed. Based upon a historic data pool statistical process corridors are derived for the controller inputs control error and change in control error. This approach follows the hypothesis that if the control performance criteria stay within predefined statistical boundaries, the final process state meets the required quality definition. In order to keep the process on its optimal growth trajectory (model based reference trajectory) a fuzzy logic controller is used that alternates the process temperature. Additionally, in order to stay within the process corridors, a genetic algorithm was applied to tune the input and output fuzzy sets of a preliminarily parameterized fuzzy controller. The presented experimental results show that the genetic tuned fuzzy controller is able to keep the process within its allowed limits. The average absolute error to the reference growth trajectory is 5.2 × 106 cells/mL. The controller proves its robustness to keep the process on the desired growth profile

Real-Time Substrate Feed Optimization of Anaerobic Co-Digestion Plants

In anaerobic co-digestion plants a mix of organic materials is converted to biogas using the anaerobic digestion process. These organic materials, called substrates, can be crops, sludge, manure, organic wastes and many more. They are fed on a daily basis and significantly affect the biogas production process. In this thesis dynamic real-time optimization of the substrate feed for anaerobic co-digestion plants is developed. In dynamic real-time optimization a dynamic simulation model is used to predict the future performance of the controlled plant. Therefore, a complex simulation model for biogas plants is developed, which uses the famous Anaerobic Digestion Model No. 1 (ADM1). With this model the future economics as well as stability can be calculated resulting in a multi-objective performance criterion. Using multi-objective nonlinear model predictive control (NMPC) the model predictions are used to find the optimal substrate feed for the biogas plant. Therefore, NMPC solves an optimization problem over a moving horizon and applies the optimal substrate feed to the plant for a short while before recalculating the new optimal solution. The multi-objective optimization problem is solved using state-of-the-art methods such as SMS-EMOA and SMS-EGO. The performance of the proposed approach is validated in a detailed simulation studyAlgorithms and the Foundations of Software technolog

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Dynamical Analysis and Robust Control Synthesis for Water Treatment Processes

Nowadays, water demand and water scarcity are very urgent issues due to population growth, drought and poor water quality all over the world. Therefore, water treatment plants are playing a vital role for good living condition of human. Water area needs more concentration study to increase water productivity and decrease water cost. This dissertation presents the analysis and control of water treatment plants using robust control techniques. The applied control algorithms include H∞, gain scheduled and observer-based loop-shaping control technique. They are modern control algorithms and very powerful in robust controlling of systems with uncertainties and disturbances. The water treatment plants include a desalination system and a wastewater process. Since fresh water scarcity is getting more serious, the desalination plants are to produce drinking water and wastewater treatment plants give the reusable water. The desalination system is a RO one used to produce drinking water from seawater and brackish water. The nonlinear behaviors of this system is carefully analyzed before the linearization. Due to the uncertainty caused by concentration polarization, the system is linearized using linear state-space parametric uncertainty framework. The system also suffer from many disturbances which water hammer is one of the most influential one. The mixed robust H∞ and μ-synthesis control algorithm is applied to control the RO system coping with large uncertainties, disturbances and noises. The wastewater treatment process is an activated sludge process. This biological process use microorganisms to convert organic and certain inorganic matter from wastewater into cell mass. The process is very complex with many coupled biological and chemical reactions. Due to the large variation in the influent flow, the system is modelized and linearized as a linear parametric varying system using affine parameter-dependent representation. Since the influent flow is quickly variable and easily to be measured, the robust gain scheduled robust controller is applied to deal with the large uncertainty caused by the scheduled parameter. This control algorithm often gives better performances than those of general robust H∞ one. In the wastewater treatment plant, there exist an anaerobic digestion, which is controlled by the observer-based loop-shaping algorithm. The simulations show that all the controllers can effectively deal with large uncertainties, disturbances and noises in water treatment plants. They help improve the system performances and safeties, save energy and reduce product water costs. The studies contribute some potential control approaches for water treatment plants, which is currently a very active research area in the world.Contents ······················································································· iv List of Tables ··············································································· viii List of Figures ··············································································· ix Chapter 1. Introduction ···································································· 1 1.1 Reverse osmosis process ···································································· 2 1.2 Activated sludge process ···································································· 6 1.3 Robust H∞ and gain scheduling control ··················································· 10 Chapter 2. Robust H∞ controller ······················································· 13 2.1 Introduction ·················································································· 13 2.2 Uncertainty modelling ······································································ 13 2.2.1 Unstructured uncertainties ···························································· 14 2.2.2 Parametric uncertainties ······························································· 15 2.2.3 Structured uncertainties ································································ 16 2.2.4 Linear fractional transformation ······················································ 16 2.3 Stability criterion ············································································ 17 2.3.1 Small gain theorem ····································································· 17 2.3.2 Structured singular value (muy) synthesis brief definition ·························· 19 2.4 Robustness analysis and controller design ··············································· 20 2.4.1 Forming generalised plant and N-delta structure ····································· 20 2.4.2 Robustness analysis ···································································· 24 2.5 Reduced controller ·········································································· 26 2.5.1 Truncation ··············································································· 27 2.5.2 Residualization ········································································· 29 2.5.3 Balanced realization···································································· 29 2.5.4 Optimal Hankel norm approximation ················································ 31 Chapter 3. Robust gain scheduling controller ······································· 37 3.1 Introduction ·················································································· 37 3.2 Linear parameter varying (LPV) system ·················································· 39 3.3 Matrix Polytope ·············································································· 40 3.4 Polytope and affine parameter-dependent representation ······························· 41 3.4.1 Polytope representation ································································ 41 3.4.2 Affine parameter-dependent representation ········································· 42 3.5 Quadratic stability of LPV systems and quadratic (robust) H∞ performance ········· 43 3.6 Robust gain scheduling ····································································· 44 3.6.1 LPV system linearization ······························································ 44 3.6.2 Polytope-based gain scheduling ······················································ 45 3.6.3 LFT-based gain scheduling ··························································· 48 Chapter 4. Mixed robust H∞ and μ-synthesis controller applied for a reverse osmosis desalination system ····························································· 52 4.1 RO principles ················································································ 52 4.1.1 Osmosis and reverse osmosis ························································· 52 4.1.2 Dead-end filtration and cross-flow filtration ········································ 53 4.2 Membranes ··················································································· 54 4.2.1 Structure and material ································································· 54 4.2.2 Hollow fine fiber membrane module ················································ 55 4.2.3 Spiral wound membrane module ····················································· 57 4.3 Nonlinear RO modelling and analysis ···················································· 58 4.3.1 RO system introduction ······························································· 58 4.3.2 Modelling ··············································································· 60 4.3.3 Nonlinear analysis ······································································ 62 4.3.4 Concentration polarization ···························································· 64 4.4 Water hammer phenomenon ······························································· 66 4.4.1 Water hammer, column separation and vaporous cavitation ······················ 66 4.4.2 Water hammer analysis and simulation ·············································· 69 4.4.3 Prevention of water hammer effect··················································· 78 4.5 RO linearization ············································································· 79 4.5.1 Nominal linearization ·································································· 79 4.5.2 Uncertainty modeling ·································································· 81 4.5.3 Parametric uncertainty linearization ················································· 83 4.6 Robust H∞ controller design for RO system ·············································· 85 4.6.1 Control of uncertain RO system ······················································ 85 4.6.2 Robustness analysis and H∞ controller design ······································ 86 4.7 Simulation result and discussion··························································· 90 4.8 Conclusion ··················································································· 95 Chapter 5. Robust gain scheduling control of activated sludge process ······· 96 5.1 Introduction about activated sludge process ············································· 96 5.1.1 State variables ·········································································· 98 5.1.2 ASM1 processes ······································································ 100 5.1.3 The control problem of activated sludge process ································· 102 5.2 System modelling ········································································· 104 5.3 Model linearization ········································································ 107 5.4 Robust gain-schedule controller design for activated sludge process ··············· 109 5.5 Simulation result and discussion························································· 115 5.6 Conclusion ················································································· 120 Chapter 6. Observer-based loop-shaping control of anaerobic digestion ···· 121 6.1 Introduction ················································································ 121 6.1.1 Control problem in anaerobic digestion ··········································· 122 6.2 System modelling ········································································· 123 6.3 Controller design ·········································································· 124 6.3.1 H∞ loop-shaping controller ························································· 125 6.3.2 Coprime factor uncertainty ·························································· 126 6.3.3 Control synthesis ····································································· 127 6.4 Simulation result ··········································································· 131 6.5 Conclusion ················································································· 133 Chapter 7. Conclusion ··································································· 134 References ·················································································· 136 Appendices ················································································· 144Docto

Wastewater Treatment and Reuse Technologies

This edited volume is a collection of 12 publications from esteemed research groups around the globe. The articles belong to the following broad categories: biological treatment process parameters, sludge management and disinfection, removal of trace organic contaminants, removal of heavy metals, and synthesis and fouling control of membranes for wastewater treatment