Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Optimal control towards sustainable wastewater treatment plants based on multi-agent reinforcement learning

Wastewater treatment plants are designed to eliminate pollutants and alleviate environmental pollution. However, the construction and operation of WWTPs consume resources, emit greenhouse gases (GHGs) and produce residual sludge, thus require further optimization. WWTPs are complex to control and optimize because of high nonlinearity and variation. This study used a novel technique, multi-agent deep reinforcement learning, to simultaneously optimize dissolved oxygen and chemical dosage in a WWTP. The reward function was specially designed from life cycle perspective to achieve sustainable optimization. Five scenarios were considered: baseline, three different effluent quality and cost-oriented scenarios. The result shows that optimization based on LCA has lower environmental impacts compared to baseline scenario, as cost, energy consumption and greenhouse gas emissions reduce to 0.890 CNY/m3-ww, 0.530 kWh/m3-ww, 2.491 kg CO2-eq/m3-ww respectively. The cost-oriented control strategy exhibits comparable overall performance to the LCA driven strategy since it sacrifices environmental bene ts but has lower cost as 0.873 CNY/m3-ww. It is worth mentioning that the retrofitting of WWTPs based on resources should be implemented with the consideration of impact transfer. Specifically, LCA SW scenario decreases 10 kg PO4-eq in eutrophication potential compared to the baseline within 10 days, while significantly increases other indicators. The major contributors of each indicator are identified for future study and improvement. Last, the author discussed that novel dynamic control strategies required advanced sensors or a large amount of data, so the selection of control strategies should also consider economic and ecological conditions

Management of Uncertainty by Statistical Process Control and a Genetic Tuned Fuzzy System

In food industry, bioprocesses like fermentation often are a crucial part of the manufacturing process and decisive for the final product quality. In general, they are characterized by highly nonlinear dynamics and uncertainties that make it difficult to control these processes by the use of traditional control techniques. In this context, fuzzy logic controllers offer quite a straightforward way to control processes that are affected by nonlinear behavior and uncertain process knowledge. However, in order to maintain process safety and product quality it is necessary to specify the controller performance and to tune the controller parameters. In this work, an approach is presented to establish an intelligent control system for oxidoreductive yeast propagation as a representative process biased by the aforementioned uncertainties. The presented approach is based on statistical process control and fuzzy logic feedback control. As the cognitive uncertainty among different experts about the limits that define the control performance as still acceptable may differ a lot, a data-driven design method is performed. Based upon a historic data pool statistical process corridors are derived for the controller inputs control error and change in control error. This approach follows the hypothesis that if the control performance criteria stay within predefined statistical boundaries, the final process state meets the required quality definition. In order to keep the process on its optimal growth trajectory (model based reference trajectory) a fuzzy logic controller is used that alternates the process temperature. Additionally, in order to stay within the process corridors, a genetic algorithm was applied to tune the input and output fuzzy sets of a preliminarily parameterized fuzzy controller. The presented experimental results show that the genetic tuned fuzzy controller is able to keep the process within its allowed limits. The average absolute error to the reference growth trajectory is 5.2 × 106 cells/mL. The controller proves its robustness to keep the process on the desired growth profile

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Pathways to Water Sector Decarbonization, Carbon Capture and Utilization

The water sector is in the middle of a paradigm shift from focusing on treatment and meeting discharge permit limits to integrated operation that also enables a circular water economy via water reuse, resource recovery, and system level planning and operation. While the sector has gone through different stages of such revolution, from improving energy efficiency to recovering renewable energy and resources, when it comes to the next step of achieving carbon neutrality or negative emission, it falls behind other infrastructure sectors such as energy and transportation. The water sector carries tremendous potential to decarbonize, from technological advancements, to operational optimization, to policy and behavioural changes. This book aims to fill an important gap for different stakeholders to gain knowledge and skills in this area and equip the water community to further decarbonize the industry and build a carbon-free society and economy. The book goes beyond technology overviews, rather it aims to provide a system level blueprint for decarbonization. It can be a reference book and textbook for graduate students, researchers, practitioners, consultants and policy makers, and it will provide practical guidance for stakeholders to analyse and implement decarbonization measures in their professions

Real-Time Substrate Feed Optimization of Anaerobic Co-Digestion Plants

In anaerobic co-digestion plants a mix of organic materials is converted to biogas using the anaerobic digestion process. These organic materials, called substrates, can be crops, sludge, manure, organic wastes and many more. They are fed on a daily basis and significantly affect the biogas production process. In this thesis dynamic real-time optimization of the substrate feed for anaerobic co-digestion plants is developed. In dynamic real-time optimization a dynamic simulation model is used to predict the future performance of the controlled plant. Therefore, a complex simulation model for biogas plants is developed, which uses the famous Anaerobic Digestion Model No. 1 (ADM1). With this model the future economics as well as stability can be calculated resulting in a multi-objective performance criterion. Using multi-objective nonlinear model predictive control (NMPC) the model predictions are used to find the optimal substrate feed for the biogas plant. Therefore, NMPC solves an optimization problem over a moving horizon and applies the optimal substrate feed to the plant for a short while before recalculating the new optimal solution. The multi-objective optimization problem is solved using state-of-the-art methods such as SMS-EMOA and SMS-EGO. The performance of the proposed approach is validated in a detailed simulation studyAlgorithms and the Foundations of Software technolog