HPC-enabling technologies for high-fidelity combustion simulations

With the increase in computational power in the last decade and the forthcoming Exascale supercomputers, a new horizon in computational modelling and simulation is envisioned in combustion science. Considering the multiscale and multiphysics characteristics of turbulent reacting flows, combustion simulations are considered as one of the most computationally demanding applications running on cutting-edge supercomputers. Exascale computing opens new frontiers for the simulation of combustion systems as more realistic conditions can be achieved with high-fidelity methods. However, an efficient use of these computing architectures requires methodologies that can exploit all levels of parallelism. The efficient utilization of the next generation of supercomputers needs to be considered from a global perspective, that is, involving physical modelling and numerical methods with methodologies based on High-Performance Computing (HPC) and hardware architectures. This review introduces recent developments in numerical methods for large-eddy simulations (LES) and direct-numerical simulations (DNS) to simulate combustion systems, with focus on the computational performance and algorithmic capabilities. Due to the broad scope, a first section is devoted to describe the fundamentals of turbulent combustion, which is followed by a general description of state-of-the-art computational strategies for solving these problems. These applications require advanced HPC approaches to exploit modern supercomputers, which is addressed in the third section. The increasing complexity of new computing architectures, with tightly coupled CPUs and GPUs, as well as high levels of parallelism, requires new parallel models and algorithms exposing the required level of concurrency. Advances in terms of dynamic load balancing, vectorization, GPU acceleration and mesh adaptation have permitted to achieve highly-efficient combustion simulations with data-driven methods in HPC environments. Therefore, dedicated sections covering the use of high-order methods for reacting flows, integration of detailed chemistry and two-phase flows are addressed. Final remarks and directions of future work are given at the end. }The research leading to these results has received funding from the European Union’s Horizon 2020 Programme under the CoEC project, grant agreement No. 952181 and the CoE RAISE project grant agreement no. 951733.Peer ReviewedPostprint (published version

A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory's (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL's SummitDev and IBM's AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40\%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL's Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries

Resolving thermo-hydro-mechanical coupling: Spontaneous porous fluid and strain localisation

Localisation of deformation and flow is ubiquitously observed on Earth, spanning from sub-terraneous locations both in the deep interior and towards the shallow surface. Ductile strain localisation in tectonic processes or channelling and focusing of fluids in porous rocks are widely reported expressions of strain and flow localisation, governed by hydraulic, thermal and mechanical interactions. The intrinsic coupling of these different physical processes provides additional localisation mechanisms to well-established single-process physics. Models that address interactions between different physical processes must include non-linear feedbacks that may potentially trigger new and non-intuitive characteristic length and time scales. Accurately resolving this complex non-linear interplay resulting from coupled physics permits us to better understand the nature of multiphysics processes and to provide more accurate predictions on how, when and where to expect localisation. In many anthropogenic activities related to achieving a carbon-free energy transition, accurate predictions of mid-term to long-term behaviour for geosystems are vital. Engineered waste disposal solutions such as CO2 sequestration and nuclear waste deposits require coupled models in order to predict the complexities of the evolving system. However, there is a current lack in model capability to address the non-linear interactions resulting from multiphysics coupling. Available models often fail to reproduce major first-order field observations of localisation, mainly owing to poor coupling strategies and a lack of affordable resolution needed to resolve very local non-linear features, especially in three spatial dimensions. In this thesis, I address these issues using a supercomputing approach to resolve sufficiently high-resolution stain and flow localisation in non-linearly deforming porous media, relying on a thermodynamically consistent model formulation. The developed graphical processing unit-based parallel algorithms show close to linear weak scaling on the world’s third-largest supercomputer and are benchmarked against classical direct-iterative type solvers. The high-resolution computations are needed for the convergence of the calculations. The results confirm that a strong coupling between solid deformation, fluid flow and heat diffusion provides a viable mechanism for ‘chimney’ formation or strain localisation. Flow localisation in high-permeability chimneys provides efficient pathways for fast vertical fluid migration. By using model parameters relevant for sedimentary rocks, natural observations and their main characteristic features could be reproduced. In summary, this thesis provides an extensive study on hydro-mechanical interaction in fluid-saturated and non-linearly deforming porous rocks. Further, the predicted high-permeability pathways are vital to understand the formation of potential leakage pathways and are a prerequisite for reliable risk assessment in long-term waste storage. Finally, the developed solution strategy is successfully utilised to resolve strain localisation in thermo-mechanically coupled processes. -- La localisation de la déformation et des fluides est observée à l’échelle du Globe, allant des couches profondes jusqu’à la subsurface. Des phénomènes géologiques tels que la localisation de la déformation ductile ou la chenalisation des fluides dans les roches poreuses témoignent d’amplifications locales de la déformation et de la porosité et résultent d’interactions entre des processus hydrauliques, thermiques et mécaniques. Le couplage de ces divers processus physiques génère des rétroactions non-linéaires et aboutit à des nouvelles grandeurs caractéristiques non-triviales. Une résolution précise de ces interactions complexes permet de mieux comprendre la nature des processus multi-physiques et permet d’établir de meilleures prédictions quant à de possibles occurrences de localisation. Passablement d’activités anthropogéniques liées à la transition énergétique reposent sur des prédictions précises de l’évolution à long terme des géo-systèmes. La séquestration du CO2 ainsi que le stockage des déchets nucléaires requièrent l’utilisation de modèles couplés afin de prédire l’évolution des systèmes de confinement. Toutefois, les modèles actuels peinent à reproduire les observations de premier ordre, notamment les évidences de localisation des fluides et de la déformation. Les principales raisons sont le traitement des problèmes trop souvent effectué en deux dimensions, le manque de rigueur dans les stratégies de couplage entre les différents processus ainsi que l’utilisation de résolutions insuffisantes dans les modèles. Dans cette thèse, je propose une approche basée sur le calcul à haute performance permettant de résoudre avec des résolutions élevées les processus de localisation dans des milieux poreux déformables en utilisant des modèles thermodynamiquement consistants. Les algorithmes parallèles développés utilisent des processeurs graphiques disponibles entre autres sur le troisième plus performant superordinateur du monde et reportent un temps de calcul identique lorsque la taille du problème à résoudre grandi proportionnellement avec le nombre de ressources disponibles. Les résultats attestent de la convergence de la méthode et confirment le fait qu’un couplage important entre déformation, écoulement des fluides et diffusion de la chaleur permet la formation de chenaux à perméabilité élevée ainsi que la localisation de la déformation. Ces chenaux, ou drains, permettent l’écoulement focalisé ainsi qu’une migration verticale rapide des fluides. En prenant en compte les paramètres pétrophysiques caractéristiques des roches situées dans des bassins sédimentaires, ces écoulements préférentiels reproduisent les observations naturelles. La prédiction d’occurrence de chenaux à perméabilité élevée est vitale afin de mieux prévenir de potentiels risques de fuites et de fournir des solutions suˆres pour les générations futures en termes de stockage de déchets à risque. Pour conclure, cette thèse propose une étude extensive sur les interactions hydromécaniques dans des roches poreuses saturées avec des fluides. De manière analogue, la stratégie de solution développée a été appliquée pour étudier la localisation de la déformation ductile résultant d’un couplage thermomécanique

Microscale modeling of fluid flow in porous medium systems

Proper mathematical description of macroscopic porous medium flows is essential for the study of a wide range of subsurface contamination scenarios. Existing mathematical formulations, however, demonstrate inadequacies that preclude the accurate description of many systems. Multi-scale models developed using thermodynamically constrained averaging theory (TCAT) rigorously define macroscopic variables in terms of more well-understood microscopic counterparts, permitting detailed analysis of macroscopic model forms based on microscale simulation and experiment. Within this framework, the primary objectives of microscale modeling are to elucidate important physical mechanisms and to inform both the form of macroscale closure relations as well as associated parameter values. In order to meet these goals, numerical tools must include: (1) simulations that provide accurate microscopic solutions for physical phenomena in large, complex domains; (2) morphological analysis tools that can be used to upscale simulation results to larger scales as dictated by the associated theoretical framework. Development of a numerical toolbox for microscale porous medium studies is considered in line with these objectives, including both implementation and optimization strategies. High-performance implementations of the lattice Boltzmann method are developed to simulate one- and two-phase flows using several computing platforms. A modified marching cubes algorithm is developed to explicitly construct all entities in a two-phase system, including all interfaces between the fluid and solid phases in addition to the three phase contact curve. These entities serve as a numerical skeleton for upscaling multiphase porous medium simulation results to the macroscale. Based on these tools, development of macroscopic constitutive laws is illustrated for a special case of anisotropic flow in porous media. In this example, microscale simulation is used to demonstrate a limitation of existing macroscopic forms for cases in which the momentum resistance depends on the flow direction in addition to the orientation. A modified macroscopic form is proposed in order to properly account for this phenomenon

Classical and reactive molecular dynamics: Principles and applications in combustion and energy systems

Molecular dynamics (MD) has evolved into a ubiquitous, versatile and powerful computational method for fundamental research in science branches such as biology, chemistry, biomedicine and physics over the past 60 years. Powered by rapidly advanced supercomputing technologies in recent decades, MD has entered the engineering domain as a first-principle predictive method for material properties, physicochemical processes, and even as a design tool. Such developments have far-reaching consequences, and are covered for the first time in the present paper, with a focus on MD for combustion and energy systems encompassing topics like gas/liquid/solid fuel oxidation, pyrolysis, catalytic combustion, heterogeneous combustion, electrochemistry, nanoparticle synthesis, heat transfer, phase change, and fluid mechanics. First, the theoretical framework of the MD methodology is described systemically, covering both classical and reactive MD. The emphasis is on the development of the reactive force field (ReaxFF) MD, which enables chemical reactions to be simulated within the MD framework, utilizing quantum chemistry calculations and/or experimental data for the force field training. Second, details of the numerical methods, boundary conditions, post-processing and computational costs of MD simulations are provided. This is followed by a critical review of selected applications of classical and reactive MD methods in combustion and energy systems. It is demonstrated that the ReaxFF MD has been successfully deployed to gain fundamental insights into pyrolysis and/or oxidation of gas/liquid/solid fuels, revealing detailed energy changes and chemical pathways. Moreover, the complex physico-chemical dynamic processes in catalytic reactions, soot formation, and flame synthesis of nanoparticles are made plainly visible from an atomistic perspective. Flow, heat transfer and phase change phenomena are also scrutinized by MD simulations. Unprecedented details of nanoscale processes such as droplet collision, fuel droplet evaporation, and CO2 capture and storage under subcritical and supercritical conditions are examined at the atomic level. Finally, the outlook for atomistic simulations of combustion and energy systems is discussed in the context of emerging computing platforms, machine learning and multiscale modelling

Numerical investigations of heat and mass transport in fractured porous rock masses

Fluid flow processes in the subsurface are accompanied by heat and mass transport with several important feedbacks including reactive flow, and precipitation/dissolution processes. Heat and mass transport through fractured rock masses occurs in many natural systems such as the plumbing of volcanic systems, mesothermal ore deposits, and post-seismic fluid flow. Anthropogenically-driven systems, such as fluid-injection in Enhanced Geothermal Systems (EGS), and the injection of waste-water from hydrocarbon extraction also involve heat and mass transport through porous or fractured rocks. Understanding in detail how mass and heat transfer interact in natural or in industrial applications requires numerical models in combination with field and laboratory experiments to determine the dominating factors. This thesis examines the impact of heat and mass transport on high pressure fluid propagation in the subsurface, as well as different numerical approaches of transient heat flow in fractured porous media and the heat exchange between flowing fluid and host rock. Many fluid-triggered seismic events show a tendency for upward migration of the seismic cloud, generally assumed to reflect a fluid-pressure dependent permeability. In a numerical investigation that combines pressure-dependent permeability with thermal and salinity effects, it is found that over short timescales pressure-dependent permeability does indeed have the strongest influence on asymmetric diffusion. However, it is also demonstrated that over longer timescales, for example the lifetime of a geothermal reservoir, temperature and salinity effects play an increasingly important role. Assessing the thermal field of a geothermal resource or in a CO2 sequestration project is essential for proper design and management. Typically, numerical simulations assume that the fluid and solid phases are in thermal equilibrium, an assumption that has to date not been investigated in detail. This assumption is examined in this work by simulating fluid and heat flow in a simple geometry to analyse the influence of site specific parameters on the simulation result. It is shown that the equilibrium model is not sensitive to porosity contrasts, while the non-equilibrium model shows a sensitivity to porosity contrasts, with simulation results diverging more strongly in less permeable zones. In a simulation of a hypothetical geothermal system, the equilibrium model shows higher production temperatures with a divergence of up to 7% between the approaches, which could impact the economic feasibility of a project. Finally, a new approach is introduced to determine the heat transfer coefficient h between rock walls and flowing fluid using the non-equilibrium model. Based on a numerical experimental setup with simple geometry and steady state scenario, a dynamic heat transfer coefficient is derived that depends on fracture aperture and flow velocity. This model is based on well-defined physical parameters, it is adaptable to complex geometries, and intrinsically adjusts to spatial heterogeneities and temporal changes in flow and temperature field. A possible extension of this dynamic approach is demonstrated in numerical simulations the reservoir scale