Constrained Model-Free Reinforcement Learning for Process Optimization

Reinforcement learning (RL) is a control approach that can handle nonlinear stochastic optimal control problems. However, despite the promise exhibited, RL has yet to see marked translation to industrial practice primarily due to its inability to satisfy state constraints. In this work we aim to address this challenge. We propose an 'oracle'-assisted constrained Q-learning algorithm that guarantees the satisfaction of joint chance constraints with a high probability, which is crucial for safety critical tasks. To achieve this, constraint tightening (backoffs) are introduced and adjusted using Broyden's method, hence making them self-tuned. This results in a general methodology that can be imbued into approximate dynamic programming-based algorithms to ensure constraint satisfaction with high probability. Finally, we present case studies that analyze the performance of the proposed approach and compare this algorithm with model predictive control (MPC). The favorable performance of this algorithm signifies a step toward the incorporation of RL into real world optimization and control of engineering systems, where constraints are essential in ensuring safety

Two-phase flow modeling of solid dissolution in liquid for nutrient mixing improvement in algal raceway ponds

Achieving optimal nutrient concentrations is essential to increasing the biomass productivity of algal raceway ponds. Nutrient mixing or distribution in raceway ponds is significantly affected by hydrodynamic and geometric properties. The nutrient mixing in algal raceway ponds under the influence of hydrodynamic and geometric properties of ponds is yet to be explored. Such a study is required to ensure optimal nutrient concentrations in algal raceway ponds. A novel computational fluid dynamics (CFD) model based on the Euler–Euler numerical scheme was developed to investigate nutrient mixing in raceway ponds under the effects of hydrodynamic and geometric properties. Nutrient mixing was investigated by estimating the dissolution of nutrients in raceway pond water. Experimental and CFD results were compared and verified using solid–liquid mass transfer coefficient and nutrient concentrations. Solid–liquid mass transfer coefficient, solid holdup, and nutrient concentrations in algal pond were estimated with the effects of pond aspect ratios, water depths, paddle wheel speeds, and particle sizes of nutrients. From the results, it was found that the proposed CFD model effectively simulated nutrient mixing in raceway ponds. Nutrient mixing increased in narrow and shallow raceway ponds due to effective solid–liquid mass transfer. High paddle wheel speeds increased the dissolution rate of nutrients in raceway ponds

Using process data to generate an optimal control policy via apprenticeship and reinforcement learning

From Wiley via Jisc Publications RouterHistory: received 2020-10-04, rev-recd 2021-04-23, accepted 2021-05-03, pub-electronic 2021-05-15Article version: VoRPublication status: PublishedAbstract: Reinforcement learning (RL) is a data‐driven approach to synthesizing an optimal control policy. A barrier to wide implementation of RL‐based controllers is its data‐hungry nature during online training and its inability to extract useful information from human operator and historical process operation data. Here, we present a two‐step framework to resolve this challenge. First, we employ apprenticeship learning via inverse RL to analyze historical process data for synchronous identification of a reward function and parameterization of the control policy. This is conducted offline. Second, the parameterization is improved online efficiently under the ongoing process via RL within only a few iterations. Significant advantages of this framework include to allow for the hot‐start of RL algorithms for process optimal control, and robust abstraction of existing controllers and control knowledge from data. The framework is demonstrated on three case studies, showing its potential for chemical process control

Photocatalytic production of bisabolene from green microalgae mutant: process analysis and kinetic modeling

Currently, algal fuel research has commenced to shift toward genetically engineered mutants able to express and excrete desired products directly into the culture. In this study, a mutant strain of Chlamydomonas reinhardtii, engineered for bisabolene (alternative biodiesel) excretion, was cultivated at different illumination and temperatures to investigate their effects on cell growth and bisabolene production. Moreover, a kinetic model was constructed to identify the desirable conditions for biofuel synthesis. Three original contributions were concluded. First, this work confirmed that bisabolene was partially synthesized independently of biomass growth, indicating its feasibility for continuous production. Second, it was found that while bisabolene synthesis was independent of light intensity, it was strongly affected by temperature, resulting in conflicting desirable conditions for cell growth and product synthesis. Finally, through model prediction, optimal operating conditions were identified for mutant growth and bisabolene synthesis. This study therefore paves the way toward chemostat production and process scale-up

Deep learning based surrogate modeling and optimization for Microalgal biofuel production and photobioreactor design

Identifying optimal photobioreactor configurations and process operating conditions is critical to industrialize microalgae-derived biorenewables. Traditionally, this was addressed by testing numerous design scenarios from integrated physical models coupling computational fluid dynamics and kinetic modelling. However, this approach presents computational intractability and numerical instabilities when simulating large-scale systems, causing time-intensive computing efforts and infeasibility in mathematical optimization. Therefore, we propose an innovative data-driven surrogate modelling framework which considerably reduces computing time from months to days by exploiting state-of-the-art deep learning technology. The framework built upon a few simulated results from the physical model to learn the sophisticated hydrodynamic and biochemical kinetic mechanisms; then adopts a hybrid stochastic optimization algorithm to explore untested processes and find optimal solutions. Through verification, this framework was demonstrated to have comparable accuracy to the physical model. Moreover, multi-objective optimization was incorporated to generate a Pareto-frontier for decision-making, advancing its applications in complex biosystems modelling and optimization

Kinetic and hybrid modeling for yeast astaxanthin production under uncertainty

From Wiley via Jisc Publications RouterHistory: received 2021-03-22, rev-recd 2021-09-29, accepted 2021-09-30, pub-electronic 2021-10-12Article version: VoRPublication status: PublishedAbstract: Astaxanthin is a high‐value compound commercially synthesized through Xanthophyllomyces dendrorhous fermentation. Using mixed sugars decomposed from biowastes for yeast fermentation provides a promising option to improve process sustainability. However, little effort has been made to investigate the effects of multiple sugars on X. dendrorhous biomass growth and astaxanthin production. Furthermore, the construction of a high‐fidelity model is challenging due to the system's variability, also known as batch‐to‐batch variation. Two innovations are proposed in this study to address these challenges. First, a kinetic model was developed to compare process kinetics between the single sugar (glucose) based and the mixed sugar (glucose and sucrose) based fermentation methods. Then, the kinetic model parameters were modeled themselves as Gaussian processes, a probabilistic machine learning technique, to improve the accuracy and robustness of model predictions. We conclude that although the presence of sucrose does not affect the biomass growth kinetics, it introduces a competitive inhibitory mechanism that enhances astaxanthin accumulation by inducing adverse environmental conditions such as osmotic gradients. Moreover, the hybrid model was able to greatly reduce model simulation error and was particularly robust to uncertainty propagation. This study suggests the advantage of mixed sugar‐based fermentation and provides a novel approach for bioprocess dynamic modeling