An Evaluation and Comparison of GPU Hardware and Solver Libraries for Accelerating the OPM Flow Reservoir Simulator

Realistic reservoir simulation is known to be prohibitively expensive in terms of computation time when increasing the accuracy of the simulation or by enlarging the model grid size. One method to address this issue is to parallelize the computation by dividing the model in several partitions and using multiple CPUs to compute the result using techniques such as MPI and multi-threading. Alternatively, GPUs are also a good candidate to accelerate the computation due to their massively parallel architecture that allows many floating point operations per second to be performed. The numerical iterative solver takes thus the most computational time and is challenging to solve efficiently due to the dependencies that exist in the model between cells. In this work, we evaluate the OPM Flow simulator and compare several state-of-the-art GPU solver libraries as well as custom developed solutions for a BiCGStab solver using an ILU0 preconditioner and benchmark their performance against the default DUNE library implementation running on multiple CPU processors using MPI. The evaluated GPU software libraries include a manual linear solver in OpenCL and the integration of several third party sparse linear algebra libraries, such as cuSparse, rocSparse, and amgcl. To perform our bench-marking, we use small, medium, and large use cases, starting with the public test case NORNE that includes approximately 50k active cells and ending with a large model that includes approximately 1 million active cells. We find that a GPU can accelerate a single dual-threaded MPI process up to 5.6 times, and that it can compare with around 8 dual-threaded MPI processes

A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory's (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL's SummitDev and IBM's AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40\%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL's Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries

Large scale data analysis using MLlib

Recent advancements in the internet, social media, and internet of things (IoT) devices have significantly increased the amount of data generated in a variety of formats. The data must be converted into formats that is easily handled by the data analysis techniques. It is mathematically and physically expensive to apply machine learning algorithms to big and complicated data sets. It is a resource-intensive process that necessitates a huge amount of logical and physical resources. Machine learning is a sophisticated data analytics technology that has gained in importance as a result of the massive amount of data generated daily that needs to be examined. Apache Spark machine learning library (MLlib) is one of the big data analysis platforms that provides a variety of outstanding functions for various machine learning tasks, spanning from classification to regression and dimension reduction. From a computational standpoint, this research investigated Apache Spark MLlib 2.0 as an open source, autonomous, scalable, and distributed learning library. Several real-world machine learning experiments are carried out in order to evaluate the properties of the platform on a qualitative and quantitative level. Some of the fundamental concepts and approaches for developing a scalable data model in a distributed environment are also discussed

Finite element method for coupled thermo-hydro-mechanical processes in discretely fractured and non-fractured porous media

Numerical analysis of multi-field problems in porous and fractured media is an important subject for various geotechnical engineering tasks such as the management of geo-resources (e.g. engineering of geothermal, oil and gas reservoirs) as well as waste management. For practical usage, e.g. for geothermal, simulation tools are required which take into account both coupled thermo-hydro-mechanical (THM) processes and the uncertainty of geological data, i.e. the model parametrization. For modeling fractured rocks, equivalent porous medium or multiple continuum model approaches are often only the way currently due to difficulty to handle geomechanical discontinuities. However, they are not applicable for prediction of flow and transport in subsurface systems where a few fractures dominates the system behavior. Thus modeling coupled problems in discretely fractured porous media is desirable for more precise analysis. The subject of this work is developing a framework of the finite element method (FEM) for modeling coupled THM problems in discretely fractured and non-fractured porous media including thermal water flow, advective-diffusive heat transport, and thermoporoelasticity. Pre-existing fractures are considered. Systems of discretely fractured porous media can be considered as a problem of interacted multiple domains, i.e. porous medium domain and discrete fracture domain, for hydraulic and transport processes, and a discontinuous problem for mechanical processes. The FEM is required to take into account both kinds of the problems. In addition, this work includes developing a methodology for the data uncertainty using the FEM model and investigating the uncertainty impacts on evaluating coupled THM processes. All the necessary code developments in this work has been carried out with a scientific open source project OpenGeoSys (OGS). In this work, fluid flow and heat transport problems in interactive multiple domains are solved assuming continuity of filed variables (pressure and temperature) over the two domains. The assumption is reasonable if there are no infill materials in fractures. The method has been successfully applied for several numerical examples, e.g. modeling three-dimensional coupled flow and heat transport processes in discretely fractured porous media at the Gross Schoenebck geothermal site (Germany), and three-dimensional coupled THM processes in porous media at the Urach Spa geothermal site (Germany). To solve the mechanically discontinuous problems, lower-dimensional interface elements (LIEs) with local enrichments have been developed for coupled problems in a domain including pre-existing fractures. The method permits the possibility of using existing flow simulators and having an identical mesh for both processes. It enables us to formulate the coupled problems in monolithic scheme for robust computation. Moreover, it gives an advantage in practice that one can use existing standard FEM codes for groundwater flow and easily make a coupling computation between mechanical and hydraulic processes. Example of a 2D fluid injection problem into a single fracture demonstrated that the proposed method can produce results in strong agreement with semi-analytical solutions. An uncertainty analysis of THM coupled processes has been studied for a typical geothermal reservoir in crystalline rock based on the Monte-Carlo method. Fracture and matrix are treated conceptually as an equivalent porous medium, and the model is applied to available data from the Urach Spa and Falkenberg sites (Germany). Reservoir parameters are considered as spatially random variables and their realizations are generated using conditional Gaussian simulation. Two reservoir modes (undisturbed and stimulated) are considered to construct a stochastic model for permeability distribution. We found that the most significant factors in the analysis are permeability and heat capacity. The study demonstrates the importance of taking parameter uncertainties into account for geothermal reservoir evaluation in order to assess the viability of numerical modeling

Surrogate Reservoir Model for Generating Pressure, Saturation and Compositional Characteristics at the Grid-Block Level; SACROC field, Scurry County, Texas

Nowadays, due to advancements in data acquisition technologies in oil and gas industry more data are available for generating reservoir simulation models. This leads to high fidelity reservoir simulation models which are highly complex and computationally expensive. The conventional reservoir management studies require hundreds realizations of the simulation models. As S. Gencer (2007) described, the reservoir simulation trend is towards more: more users, more models, more cells, more wells, more cases, more data and more integration [1]. In order to enhance the reservoir model descriptions, more computational power would have to be designed and engineered to keep up with our modeling needs; hence, creating an unsustainable cyclical process. Therefore, even with the advancements in the computational powers, the industry cannot take advantage of the full potential of these full-field reservoir simulation models.;Many studies have tried to create alternative methods in order to replicate the performance of full-field reservoir simulation models and at the same time decrease the cost of operation. Traditional proxy models, such as statistical based approaches, are examples of these studies. The degree of success, particularly practical aspects, for these approaches remains to be argued.;As an alternative to traditional proxy modeling methods, the objective of this study is to investigate the feasibility of use of a fast intelligent approximation of the numerical simulation model. This replica will accurately reproduce dynamic reservoir properties of complex full-field numerical simulation models in matter of seconds. A Grid-based Surrogate Reservoir Model (GSRM) is developed based on data-driven and Artificial Intelligence techniques. This technology is able to learn from the provided examples of the reservoir simulation model. The robustness of this technology is validated by testing it on non-seen instances. Finally the trained and validated GSRM will produce the results of full-field simulation models accurately and in a very short time (seconds).;This concept will be proven by building a GSRM of CO2 injection--EOR numerical model of SACROC field, Scurry County, Texas. The SACROC model (CMG GEM) in use was previously generated and history matched by the Petroleum Engineering & Analytics Research Lab - PEARL - at West Virginia University, it is based on a comprehensive geological study that includes 3D seismic surveys and well logs; in order to generate the GSRM this model is to be ran using multiple injection scenarios that will create an appropriate solution space so we can comprehend and grasp its behavior using artificial intelligence

Integrating multiple clusters for compute-intensive applications

Multicluster grids provide one promising solution to satisfying the growing computational demands of compute-intensive applications. However, it is challenging to seamlessly integrate all participating clusters in different domains into a single virtual computational platform. In order to fully utilize the capabilities of multicluster grids, computer scientists need to deal with the issue of joining together participating autonomic systems practically and efficiently to execute grid-enabled applications. Driven by several compute-intensive applications, this theses develops a multicluster grid management toolkit called Pelecanus to bridge the gap between user\u27s needs and the system\u27s heterogeneity. Application scientists will be able to conduct very large-scale execution across multiclusters with transparent QoS assurance. A novel model called DA-TC (Dynamic Assignment with Task Containers) is developed and is integrated into Pelecanus. This model uses the concept of a task container that allows one to decouple resource allocation from resource binding. It employs static load balancing for task container distribution and dynamic load balancing for task assignment. The slowest resources become useful rather than be bottlenecks in this manner. A cluster abstraction is implemented, which not only provides various cluster information for the DA-TC execution model, but also can be used as a standalone toolkit to monitor and evaluate the clusters\u27 functionality and performance. The performance of the proposed DA-TC model is evaluated both theoretically and experimentally. Results demonstrate the importance of reducing queuing time in decreasing the total turnaround time for an application. Experiments were conducted to understand the performance of various aspects of the DA-TC model. Experiments showed that our model could significantly reduce turnaround time and increase resource utilization for our targeted application scenarios. Four applications are implemented as case studies to determine the applicability of the DA-TC model. In each case the turnaround time is greatly reduced, which demonstrates that the DA-TC model is efficient for assisting application scientists in conducting their research. In addition, virtual resources were integrated into the DA-TC model for application execution. Experiments show that the execution model proposed in this thesis can work seamlessly with multiple hybrid grid/cloud resources to achieve reduced turnaround time

Simulation Intelligence: Towards a New Generation of Scientific Methods

The original "Seven Motifs" set forth a roadmap of essential methods for the field of scientific computing, where a motif is an algorithmic method that captures a pattern of computation and data movement. We present the "Nine Motifs of Simulation Intelligence", a roadmap for the development and integration of the essential algorithms necessary for a merger of scientific computing, scientific simulation, and artificial intelligence. We call this merger simulation intelligence (SI), for short. We argue the motifs of simulation intelligence are interconnected and interdependent, much like the components within the layers of an operating system. Using this metaphor, we explore the nature of each layer of the simulation intelligence operating system stack (SI-stack) and the motifs therein: (1) Multi-physics and multi-scale modeling; (2) Surrogate modeling and emulation; (3) Simulation-based inference; (4) Causal modeling and inference; (5) Agent-based modeling; (6) Probabilistic programming; (7) Differentiable programming; (8) Open-ended optimization; (9) Machine programming. We believe coordinated efforts between motifs offers immense opportunity to accelerate scientific discovery, from solving inverse problems in synthetic biology and climate science, to directing nuclear energy experiments and predicting emergent behavior in socioeconomic settings. We elaborate on each layer of the SI-stack, detailing the state-of-art methods, presenting examples to highlight challenges and opportunities, and advocating for specific ways to advance the motifs and the synergies from their combinations. Advancing and integrating these technologies can enable a robust and efficient hypothesis-simulation-analysis type of scientific method, which we introduce with several use-cases for human-machine teaming and automated science

Reservoir Simulation of the Volve Oil field using AI-based Top-Down Modeling Approach

With the rise of high-performance computers, numerical reservoir simulators became popular among engineers to evaluate reservoirs and develop the fields. However, this technology is still unable to fully model the reservoirs with commingled production and highly complex geology, especially when it comes to uncertainty qualification and sensitivity analysis where hundreds of runs are required. This dissertation aims to provide a successful case study of the history matching of a complex reservoir in the North Sea (Volve field). The proposed model relies only on the measured field variables such as well, formation, completion characteristics, production rates, and operational conditions while it stays away from interpretation and assumption. More than eight years of data from the Volve field was used to generate a comprehensive dataset, and then key parameters were extracted using fuzzy pattern recognition. A system of fully coupled artificial neural networks (feed-forward and LSTM networks) was used to train, calibrate and validate the model. The Artificial neural network enables us to extract hidden patterns in the field by learning from historical data. The model successfully history-matched the well-head pressure, well-head temperature, and production rates of all the wells through a completely automated process. The forecasting capability of the model has been verified through blind validation in time and space using data that the model has not seen before. In contrast to the numerical simulator, which is only a reservoir model, this technology is a coupled reservoir and well-bore model which is able to learn the fluid motion behavior in a complex porous media with fewer resources and higher speed. The efficiency of this approach makes it a suitable tool for uncertainty quantification when a large number of runs is required. The combination of artificial intelligence and domain expertise makes this technology more reliable and closer to reality by staying loyal to field measurements