Advancing Carbon Sequestration through Smart Proxy Modeling: Leveraging Domain Expertise and Machine Learning for Efficient Reservoir Simulation

Geological carbon sequestration (GCS) offers a promising solution to effectively manage extra carbon, mitigating the impact of climate change. This doctoral research introduces a cutting-edge Smart Proxy Modeling-based framework, integrating artificial neural networks (ANNs) and domain expertise, to re-engineer and empower numerical reservoir simulation for efficient modeling of CO2 sequestration and demonstrate predictive conformance and replicative capabilities of smart proxy modeling. Creating well-performing proxy models requires extensive human intervention and trial-and-error processes. Additionally, a large training database is essential to ANN model for complex tasks such as deep saline aquifer CO2 sequestration since it is used as the neural network\u27s input and output data. One major limitation in CCS programs is the lack of real field data due to a lack of field applications and issues with confidentiality. Considering these drawbacks, and due to high-dimensional nonlinearity, heterogeneity, and coupling of multiple physical processes associated with numerical reservoir simulation, novel research to handle these complexities as it allows for the creation of possible CO2 sequestration scenarios that may be used as a training set. This study addresses several types of static and dynamic realistic and practical field-base data augmentation techniques ranging from spatial complexity, spatio-temporal complexity, and heterogeneity of reservoir characteristics. By incorporating domain-expertise-based feature generation, this framework honors precise representation of reservoir overcoming computational challenges associated with numerical reservoir tools. The developed ANN accurately replicated fluid flow behavior, resulting in significant computational savings compared to traditional numerical simulation models. The results showed that all the ML models achieved very good accuracies and high efficiency. The findings revealed that the quality of the path between the focal cell and injection wells emerged as the most crucial factor in both CO2 saturation and pressure estimation models. These insights significantly contribute to our understanding of CO2 plume monitoring, paving the way for breakthroughs in investigating reservoir behavior at a minimal computational cost. The study\u27s commitment to replicating numerical reservoir simulation results underscores the model\u27s potential to contribute valuable insights into the behavior and performance of CO2 sequestration systems, as a complimentary tool to numerical reservoir simulation when there is no measured data available from the field. The transformative nature of this research has vast implications for advancing carbon storage modeling technologies. By addressing the computational limitations of traditional numerical reservoir models and harnessing the synergy between machine learning and domain expertise, this work provides a practical workflow for efficient decision-making in sequestration projects

Auto-weighted Bayesian Physics-Informed Neural Networks and robust estimations for multitask inverse problems in pore-scale imaging of dissolution

In this article, we present a novel data assimilation strategy in pore-scale imaging and demonstrate that this makes it possible to robustly address reactive inverse problems incorporating Uncertainty Quantification (UQ). Pore-scale modeling of reactive flow offers a valuable opportunity to investigate the evolution of macro-scale properties subject to dynamic processes. Yet, they suffer from imaging limitations arising from the associated X-ray microtomography (X-ray microCT) process, which induces discrepancies in the properties estimates. Assessment of the kinetic parameters also raises challenges, as reactive coefficients are critical parameters that can cover a wide range of values. We account for these two issues and ensure reliable calibration of pore-scale modeling, based on dynamical microCT images, by integrating uncertainty quantification in the workflow. The present method is based on a multitasking formulation of reactive inverse problems combining data-driven and physics-informed techniques in calcite dissolution. This allows quantifying morphological uncertainties on the porosity field and estimating reactive parameter ranges through prescribed PDE models with a latent concentration field and dynamical microCT. The data assimilation strategy relies on sequential reinforcement incorporating successively additional PDE constraints. We guarantee robust and unbiased uncertainty quantification by straightforward adaptive weighting of Bayesian Physics-Informed Neural Networks (BPINNs), ensuring reliable micro-porosity changes during geochemical transformations. We demonstrate successful Bayesian Inference in 1D+Time and 2D+Time calcite dissolution based on synthetic microCT images with meaningful posterior distribution on the reactive parameters and dimensionless numbers

Development and evaluation of models for assessing geochemical pollution sources with multiple reactive chemical species for sustainable use of aquifer systems: source characterization and monitoring network design

Michael designed a groundwater flow and reactive transport optimization model. He applied this model to characterize contaminant sources in Australia's first large scale uranium mine site in the Northern Territory. He identified the contamination sources to the groundwater system in the area. His findings will assist planning actions and steps needed to implement the mitigation strategy of this contaminated aquifer

Coupled simulation-optimization model for coastal aquifer management using genetic programming-based ensemble surrogate models and multiple-realization optimization

Approximation surrogates are used to substitute the numerical simulation model within optimization algorithms in order to reduce the computational burden on the coupled simulation-optimization methodology. Practical utility of the surrogate-based simulation-optimization have been limited mainly due to the uncertainty in surrogate model simulations. We develop a surrogate-based coupled simulation-optimization methodology for deriving optimal extraction strategies for coastal aquifer management considering the predictive uncertainty of the surrogate model. Optimization models considering two conflicting objectives are solved using a multiobjective genetic algorithm. Objectives of maximizing the pumping from production wells and minimizing the barrier well pumping for hydraulic control of saltwater intrusion are considered. Density-dependent flow and transport simulation model FEMWATER is used to generate input-output patterns of groundwater extraction rates and resulting salinity levels. The nonparametric bootstrap method is used to generate different realizations of this data set. These realizations are used to train different surrogate models using genetic programming for predicting the salinity intrusion in coastal aquifers. The predictive uncertainty of these surrogate models is quantified and ensemble of surrogate models is used in the multiple-realization optimization model to derive the optimal extraction strategies. The multiple realizations refer to the salinity predictions using different surrogate models in the ensemble. Optimal solutions are obtained for different reliability levels of the surrogate models. The solutions are compared against the solutions obtained using a chance-constrained optimization formulation and single-surrogate-based model. The ensemble-based approach is found to provide reliable solutions for coastal aquifer management while retaining the advantage of surrogate models in reducing computational burden

Development of a GPGPU accelerated tool to simulate advection-reaction-diffusion phenomena in 2D

Computational models are powerful tools to the study of environmental systems, playing a fundamental role in several fields of research (hydrological sciences, biomathematics, atmospheric sciences, geosciences, among others). Most of these models require high computational capacity, especially when one considers high spatial resolution and the application to large areas. In this context, the exponential increase in computational power brought by General Purpose Graphics Processing Units (GPGPU) has drawn the attention of scientists and engineers to the development of low cost and high performance parallel implementations of environmental models. In this research, we apply GPGPU computing for the development of a model that describes the physical processes of advection, reaction and diffusion. This presentation is held in the form of three self-contained articles. In the first one, we present a GPGPU implementation for the solution of the 2D groundwater flow equation in unconfined aquifers for heterogenous and anisotropic media. We implement a finite difference solution scheme based on the Crank- Nicolson method and show that the GPGPU accelerated solution implemented using CUDA C/C++ (Compute Unified Device Architecture) greatly outperforms the corresponding serial solution implemented in C/C++. The results show that accelerated GPGPU implementation is capable of delivering up to 56 times acceleration in the solution process using an ordinary office computer. In the second article, we study the application of a diffusive-logistic growth (DLG) model to the problem of forest growth and regeneration. The study focuses on vegetation belonging to preservation areas, such as riparian buffer zones. The study was developed in two stages: (i) a methodology based on Artificial Neural Network Ensembles (ANNE) was applied to evaluate the width of riparian buffer required to filter 90% of the residual nitrogen; (ii) the DLG model was calibrated and validated to generate a prognostic of forest regeneration in riparian protection bands considering the minimum widths indicated by the ANNE. The solution was implemented in GPGPU and it was applied to simulate the forest regeneration process for forty years on the riparian protection bands along the Ligeiro river, in Brazil. The results from calibration and validation showed that the DLG model provides fairly accurate results for the modelling of forest regeneration. In the third manuscript, we present a GPGPU implementation of the solution of the advection-reaction-diffusion equation in 2D. The implementation is designed to be general and flexible to allow the modeling of a wide range of processes, including those with heterogeneity and anisotropy. We show that simulations performed in GPGPU allow the use of mesh grids containing more than 20 million points, corresponding to an area of 18,000 km? in a standard Landsat image resolution.Os modelos computacionais s?o ferramentas poderosas para o estudo de sistemas ambientais, desempenhando um papel fundamental em v?rios campos de pesquisa (ci?ncias hidrol?gicas, biomatem?tica, ci?ncias atmosf?ricas, geoci?ncias, entre outros). A maioria desses modelos requer alta capacidade computacional, especialmente quando se considera uma alta resolu??o espacial e a aplica??o em grandes ?reas. Neste contexto, o aumento exponencial do poder computacional trazido pelas Unidades de Processamento de Gr?ficos de Prop?sito Geral (GPGPU) chamou a aten??o de cientistas e engenheiros para o desenvolvimento de implementa??es paralelas de baixo custo e alto desempenho para modelos ambientais. Neste trabalho, aplicamos computa??o em GPGPU para o desenvolvimento de um modelo que descreve os processos f?sicos de advec??o, rea??o e difus?o. Esta disserta??o ? apresentada sob a forma de tr?s artigos. No primeiro, apresentamos uma implementa??o em GPGPU para a solu??o da equa??o de fluxo de ?guas subterr?neas 2D em aqu?feros n?o confinados para meios heterog?neos e anisotr?picos. Foi implementado um esquema de solu??o de diferen?as finitas com base no m?todo Crank- Nicolson e mostramos que a solu??o acelerada GPGPU implementada usando CUDA C / C ++ supera a solu??o serial correspondente implementada em C / C ++. Os resultados mostram que a implementa??o acelerada por GPGPU ? capaz de fornecer acelera??o de at? 56 vezes no processo da solu??o usando um computador de escrit?rio comum. No segundo artigo estudamos a aplica??o de um modelo de crescimento log?stico difusivo (DLG) ao problema de crescimento e regenera??o florestal. O estudo foi desenvolvido em duas etapas: (i) Aplicou-se uma metodologia baseada em Comites de Rede Neural Artificial (ANNE) para avaliar a largura da faixa de prote??o rip?ria necess?ria para filtrar 90% do nitrog?nio residual; (ii) O modelo DLG foi calibrado e validado para gerar um progn?stico de regenera??o florestal em faixas de prote??o rip?rias considerando as larguras m?nimas indicadas pela ANNE. A solu??o foi implementada em GPGPU e aplicada para simular o processo de regenera??o florestal para um per?odo de quarenta anos na faixa de prote??o rip?ria ao longo do rio Ligeiro, no Brasil. Os resultados da calibra??o e valida??o mostraram que o modelo DLG fornece resultados bastante precisos para a modelagem de regenera??o florestal. No terceiro artigo, apresenta-se uma implementa??o em GPGPU para solu??o da equa??o advec??o-rea??o-difus?o em 2D. A implementa??o ? projetada para ser geral e flex?vel para permitir a modelagem de uma ampla gama de processos, incluindo caracter?sticas como heterogeneidade e anisotropia do meio. Neste trabalho mostra-se que as simula??es realizadas em GPGPU permitem o uso de malhas contendo mais de 20 milh?es de pontos (vari?veis), correspondendo a uma ?rea de 18.000 km? em resolu??o de 30m padr?o das imagens Landsat

Bio-inspired optimization in integrated river basin management

Water resources worldwide are facing severe challenges in terms of quality and quantity. It is essential to conserve, manage, and optimize water resources and their quality through integrated water resources management (IWRM). IWRM is an interdisciplinary field that works on multiple levels to maximize the socio-economic and ecological benefits of water resources. Since this is directly influenced by the river’s ecological health, the point of interest should start at the basin-level. The main objective of this study is to evaluate the application of bio-inspired optimization techniques in integrated river basin management (IRBM). This study demonstrates the application of versatile, flexible and yet simple metaheuristic bio-inspired algorithms in IRBM. In a novel approach, bio-inspired optimization algorithms Ant Colony Optimization (ACO) and Particle Swarm Optimization (PSO) are used to spatially distribute mitigation measures within a basin to reduce long-term annual mean total nitrogen (TN) concentration at the outlet of the basin. The Upper Fuhse river basin developed in the hydrological model, Hydrological Predictions for the Environment (HYPE), is used as a case study. ACO and PSO are coupled with the HYPE model to distribute a set of measures and compute the resulting TN reduction. The algorithms spatially distribute nine crop and subbasin-level mitigation measures under four categories. Both algorithms can successfully yield a discrete combination of measures to reduce long-term annual mean TN concentration. They achieved an 18.65% reduction, and their performance was on par with each other. This study has established the applicability of these bio-inspired optimization algorithms in successfully distributing the TN mitigation measures within the river basin. Stakeholder involvement is a crucial aspect of IRBM. It ensures that researchers and policymakers are aware of the ground reality through large amounts of information collected from the stakeholder. Including stakeholders in policy planning and decision-making legitimizes the decisions and eases their implementation. Therefore, a socio-hydrological framework is developed and tested in the Larqui river basin, Chile, based on a field survey to explore the conditions under which the farmers would implement or extend the width of vegetative filter strips (VFS) to prevent soil erosion. The framework consists of a behavioral, social model (extended Theory of Planned Behavior, TPB) and an agent-based model (developed in NetLogo) coupled with the results from the vegetative filter model (Vegetative Filter Strip Modeling System, VFSMOD-W). The results showed that the ABM corroborates with the survey results and the farmers are willing to extend the width of VFS as long as their utility stays positive. This framework can be used to develop tailor-made policies for river basins based on the conditions of the river basins and the stakeholders' requirements to motivate them to adopt sustainable practices. It is vital to assess whether the proposed management plans achieve the expected results for the river basin and if the stakeholders will accept and implement them. The assessment via simulation tools ensures effective implementation and realization of the target stipulated by the decision-makers. In this regard, this dissertation introduces the application of bio-inspired optimization techniques in the field of IRBM. The successful discrete combinatorial optimization in terms of the spatial distribution of mitigation measures by ACO and PSO and the novel socio-hydrological framework using ABM prove the forte and diverse applicability of bio-inspired optimization algorithms

Numerical Simulation and Effective Management of Saltwater Intrusion in Coastal Aquifers

Seawater intrusion (SWI) is a widespread environmental problem, particularly in arid and semi-arid coastal areas. Unplanned prolonged over-pumping of groundwater is the most important factor in SWI that could result in severe deterioration of groundwater quality. Therefore, appropriate management strategies should be implemented in coastal aquifers to control SWI with acceptable limits of economic and environmental costs. This PhD project presents the development and application of a simulation-optimization (S/O) model to assess different management methods of controlling saltwater intrusion while satisfying water demands, and with acceptable limits of economic and environmental costs, in confined and unconfined coastal aquifers. The first S/O model (FE-GA) is developed by direct linking of an FE simulation model with a multi-objective Genetic Algorithm (GA) to optimize the efficiency of a wide range of SWI management scenarios. However, in this S/O framework, several multiple calls of the simulation model by the population-based optimization model, evaluating best individual candidate solutions resulted in a considerable computational burden. To solve this problem the numerical simulation model is replaced by an Evolutionary Polynomial Regression (EPR)-based surrogate model in the next S/O model (EPR-GA). Through these S/O approaches (FE-GA and EPR-GA) the optimal coordinates and rates of the both abstraction and recharge barriers are determined in the studied management scenarios. As a result, a new combined methodology, so far called ADRTWW, is proposed to control SWI. The ADRTWW model consists of deep Abstraction of saline water near the coast followed by Desalination of the abstracted water to a potable level for public uses and simultaneously Recharging the aquifer using a more economic source of water such as treated wastewater (TWW). In accordance to the available recharge options (injection through well or infiltration from surface pond), the general performance of ADRTWW is evaluated in different hydro-geological settings of the aquifers indicating that it offers the least cost and least salinity in comparison with other scenarios. The great capabilities of both developed S/O models in identification of the best management solutions and the optimal coordinates and rates of the abstraction well and recharge well/pond are discussed. Both FE-GA and EPR-GA can be successfully employed by a robust decision support system. In the next phase of the study, the general impacts of sea level rise (SLR), associated with its transgression nature along the coastline surface on the saltwater intrusion mechanism are investigated in different hypothetical and real case studies of coastal aquifer systems. The results show that the rate and the amount of SWI are considerably greater in aquifers with flat shoreline slopes compared with those with steep slopes. The SWI process is followed by a significant depletion in quantity of freshwater resources at the end of the century. The situation is exacerbated with combined action of SLR and groundwater withdrawals. This finding is also confirmed by 3D simulation of SWI in a regional coastal aquifer (Wadi Ham aquifer) in the UAE subjected to the coupled actions of SLR and pumping.Ministry of Higher Education and Scientific Research in Kurdistan Regional Government of Iraq (KRG-HCDP Scholarship program)British Council under Water Security scheme (Project Code: SH- 04509

하천 오염물질 혼합 해석을 위한 저장대 모형의 매개변수 산정법 및 경험식 개발

학위논문(석사)--서울대학교 대학원 :공과대학 건설환경공학부,2019. 8. 서일원.Analyses of solute transport and retention mechanism are essential to manage water quality and river ecosystem. As reported by tracer injection studies that have been conducted to identify solute transport mechanism, concentration curves measured in natural stream have steep rising and long tail parts. This phenomenon is due to solute exchange process between transient storage zones and the main river stream. The transient storage model (TSM) is one of the most widely used models for describing solute transport in natural stream, taking transient storage exchange process into consideration. In order to use this model, calibration of four TSM parameters is necessary. Inverse modelling using measured breakthrough curves (BTCs) from tracer injection test is general method for TSM parameter calibration. However, it is not feasible to carry out performing tracer injection tests, for every parameter calibration. For that reasons, empirical formulae with hydraulic data, which is comparatively easier to obtain, have been proposed for the purpose of parameter estimation. This study presents two methods for TSM parameter estimation. At first, inverse modelling method employing global optimization framework Shuffled Complex-Self Adaptive Hybrid EvoLution (SC-SAHEL), that incorporating famous evolutionary algorithms in water resource management field, was suggested. Second, TSM parameter empirical equations were derived adopting Multigene Genetic Programming (MGGP) based symbolic regression library GPTIPS and using Principal Components Regression (PCR). In terms of general performance, equations of this study were superior to published empirical equations.하천의 수질을 관리하기 위해서는 자연하천에서 유입된 물질이 이송되고 지체되는 메카니즘을 규명하고 이해하는 것이 필요하다. 하천에서의 물질 혼합을 이해하기 위해 수행된 추적자 실험 연구들에 따르면 자연하천에서 계측되는 농도곡선에서는 가파른 상승부와 긴 꼬리기 관측되는 것으로 알려졌다. 이러한 현상은 주로 물질이 흐르는 본류대와 잠시 물질이 포획되었다가 재방출되는 본류대와 저장대 간의 물질교환 효과 때문에 일어난다고 알려져 있다. 이러한 저장대 물질교환 효과를 모사하는 저장대모형 중 Transient Storage zone Model (TSM)은 가장 광범위하게 이용되는 모형으로, 이를 이용하기 위해선 네 가지의 저장대 매개변수를 보정하여야 한다. 네 가지 저장대 매개변수를 결정하는 방법으로는 일반적으로 현장실험에서 측정된 농도곡선을 이용한 역산모형이 이용된다. 그러나 매개변수가 필요할 때마다 추적자실험을 수행하여 역산모형을 이용하는 것은 현실적으로 불가능한 경우가 있어 이러한 경우에는 비교적 취득하기 쉬운 수리지형학적 인자들을 이용해 매개변수를 산정하는 방법이 이용될 수 있다. 따라서 본 연구에서는 TSM 매개변수를 결정하기 위해 두 가지 방법을 제시하였다. 첫 번째로, 전역 최적화 프레임워크인 Shuffled Complex-Self Adaptive Hybrid EvoLution (SC-SAHEL)을 이용한 역산모형 기반 TSM 매개변수 산정 프레임워크를 제시하였다. 둘째로는 기호회귀법 라이브러리인 GPTIPS를 이용한 다중유전자 유전 프로그래밍(Multigene Genetic Programming, MGGP) 과 주성분회귀법(Principal Components Regression, PCR)을 통해 네 가지 매개변수 별로 각 두 개씩의 경험식이 개발되었다. 개발된 경험식들의 성능평가 결과, 선행 연구에서 제시된 저장대 매개변수 식에 비해 본 연구에서 제시된 방법이 대체적으로 우수한 것으로 나타났다. 결과적으로 본 연구에서는 분석을 통해 실무적으로 활용 가능한 TSM 매개변수 산정 프레임워크와 경험식들이 제시되었으며, 이 방법들은 추적자 실험 자료의 유무에 따라 TSM의 매개변수 결정에 유용하게 사용될 것으로 기대된다.Chapter 1. Introduction 1 1.1 Necessity and Background of Research 1 1.2 Objectives 12 Chapter 2. Theoretical Background 15 2.1 Transient Storage Model 15 2.1.1. Mechanisms of Transient Storage 15 2.1.2. Models Accounting for Transient Storage 21 2.1.2.1 The one Zone Transient Storage Model (1Z-TSM) 24 2.1.2.2 The two Zone Transient Storage Model (2Z-TSM) 25 2.1.2.3 The Continuous Time Random Walk Approach (CTRW) 26 2.1.2.4 The Modified Advection Dispersion Model (MADE) 27 2.1.2.5 The Fractional Advection Dispersion Equation Model (FADE) 28 2.1.2.6 The Multirate Mass Transfer Model (MRMT) 29 2.1.2.7 The Advective Storage Path Model (ASP) 30 2.1.2.8 The Solute Transport in Rivers Model (STIR) 31 2.1.2.9 The Aggregate Dead Zone Model (ADZ) 34 2.2 Empirical Equations for Predicting Transient Storage Model Parameters 39 2.3 Parameter Estimation 47 2.3.1. The SC-SAHEL Framework 50 2.3.1.1 Modified Competitive Complex Evolution (MCCE) 52 2.3.1.2 Modified Frog Leaping (MFL) 52 2.3.1.3 Modified Grey Wolf Optimizer (GWO) 53 2.3.1.4 Modified Differential Evolution (DE) 53 2.4 Regression Method 54 2.4.1. The Multi-Gene Genetic Programming (MGGP) 56 2.4.1.1 The Simple Genetic Programming 56 2.4.1.2 Scaled Symbolic Regression via Multi-Gene Genetic Programming 57 2.4.2. Evolutionary Polynomial Regression (EPR) 61 2.4.2.1 Main Flow of EPR Procedure 62 Chapter 3. Model Development 66 3.1 Numerical Model 66 3.1.1. Model Validation 69 3.2 Merger of TSM-SC-SAHEL 73 3.3 Further assessments for the parameter estimation framework 76 3.3.1. Tracer Test Description 76 3.3.2. Grid Independency of Estimation 81 3.3.3. Choice of Optimization Setting 85 Chapter 4. Development of Formulae for Predicting TSM Parameter 91 4.1 Dimensional Analysis 91 4.2 Data Collection via Meta Analysis 95 4.3 Formulae Development 106 Chapter 5. Result and Discussion 110 5.1 Model Performances 110 5.2 Sensitivity Analysis 118 5.3 In-stream Application of Empirical Equations 130 Chapter 6. Conclusion 140 References 144 Appendix. I. The mean, minimum, and maximum values of the model fitness value and number of evolution using the SC-SAHEL with single-EA and multi-EA 159 Appendix. II. Used dimensionless datasets for development of empirical equations 161 국문초록 165Maste

Uncertainty evaluation of reservoir simulation models using particle swarms and hierarchical clustering

History matching production data in finite difference reservoir simulation models has been and always will be a challenge for the industry. The principal hurdles that need to be overcome are finding a match in the first place and more importantly a set of matches that can capture the uncertainty range of the simulation model and to do this in as short a time as possible since the bottleneck in this process is the length of time taken to run the model. This study looks at the implementation of Particle Swarm Optimisation (PSO) in history matching finite difference simulation models. Particle Swarms are a class of evolutionary algorithms that have shown much promise over the last decade. This method draws parallels from the social interaction of swarms of bees, flocks of birds and shoals of fish. Essentially a swarm of agents are allowed to search the solution hyperspace keeping in memory each individual’s historical best position and iteratively improving the optimisation by the emergent interaction of the swarm. An intrinsic feature of PSO is its local search capability. A sequential niching variation of the PSO has been developed viz. Flexi-PSO that enhances the exploration and exploitation of the hyperspace and is capable of finding multiple minima. This new variation has been applied to history matching synthetic reservoir simulation models to find multiple distinct history 3 matches to try to capture the uncertainty range. Hierarchical clustering is then used to post-process the history match runs to reduce the size of the ensemble carried forward for prediction. The success of the uncertainty modelling exercise is then assessed by checking whether the production profile forecasts generated by the ensemble covers the truth case

Robust optimal control using computationally efficient deep reinforcement learning techniques

We investigate current challenges in the application of reinforcement learning (RL) to solve subsurface flow control problem which is the subject of intensive research in the field of reservoir management. In typical subsurface flow control problems, the system is partially observed because the data is often only available at well locations. Furthermore, the model parameters are highly uncertain as a result of the sparsity of available field data. As a result, we begin by presenting an RL framework to solve the stochastic optimal control for predefined model uncertainty and partially observable system. The numerical results are presented using two state-of-the-art model-free RL algorithms, proximal policy optimization (PPO) and advantage actor-critic (A2C), on two single-phase subsurface flow test cases representing two distinct flow scenarios. We identify that computational intractability is one of the major limitations for the proposed RL framework. This is because the model-free RL algorithms are by definition sample inefficient and require thousands if not millions of samples to learn optimal control policies. For subsurface control problems, this corresponds to performing a large number of simulations, which is computationally quite expensive. Our aim is to build a more generalized framework that can help alleviate this problem of computational complexity for the proposed RL framework. This is achieved by employing multiple levels of models. Here, the level refers to the accuracy or fidelity of the discretization of the domain grid of the underlying partial differential equations. We propose two distinct approaches that can be used in the most generalized manner. The first approach involves a more explicit modification of the proposed RL framework. In this approach, a multigrid framework is proposed that essentially takes advantage of the principles of sequential transfer learning. The second approach implicitly modifies the classical reinforcement learning framework itself to take advantage of information from lower-level models. This is achieved by modifying the classical framework of RL algorithms so that they use an approximate multilevel Monte Carlo estimates as opposed to Monte Carlo estimates of policy and/or value network objective functions.Engineering and Physical Sciences Research Council (EPSRC) Funding