A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory's (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL's SummitDev and IBM's AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40\%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL's Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries

Tetrahedral-Mesh Simulations of Shock-Turbulence Interaction

Despite decades of development of unstructured mesh methods, direct numerical simulations (DNS) of turbulent flows are still predominantly performed on structured or unstructured hexahedral meshes with high-order finite-difference methods, weighted essentially nonoscillatory (WENO) schemes, or hybrid schemes formed by their combinations. Tetrahedral meshes offer easy mesh generation and adaptation around complex geometries and the potential of an orientation-free grid that would benefit the isotropic nature of small-scale dissipation, as well as the solution accuracy of intermediate scales. To advance the state of the art of unstructured-mesh simulation capabilities for shock/turbulence interaction, DNS using pure tetrahedral meshes are carried out with the space-time conservation element, solution element (CESE) method in this research. By its design, the CESE method is constructed based on a non-dissipative scheme and is a genuinely multidimensional numerical framework that is free from the use of an approximate Riemann-solver. The numerical framework also provides the ability to add numerical dissipation (the nondissipative scheme acts as the reference state like that of the reversible state in thermodynamics) when needed (with justification from mathematics/physics). The above-mentioned features along with the CESE method's consistent shock-capturing approach and strong enforcement of flux conservation in spacetime offers a novel method to accurately simulate turbulent flows and their interaction with shocks using tetrahedral meshes. Two canonical problems, namely, isotropic turbulence interaction with a normal shock and a Mach 2.9 turbulent boundary layer flow over a 24deg compression corner are investigated in this study. Computational results show reasonably good agreement with experimental data and results from structured-mesh, high-order simulations available in the literature. Successful validation of these canonical problems demonstrated here paves the way for future high-fidelity supersonic flow simulations involving complex-geometries

Parallel Multiscale Contact Dynamics for Rigid Non-spherical Bodies

The simulation of large numbers of rigid bodies of non-analytical shapes or vastly varying sizes which collide with each other is computationally challenging. The fundamental problem is the identification of all contact points between all particles at every time step. In the Discrete Element Method (DEM), this is particularly difficult for particles of arbitrary geometry that exhibit sharp features (e.g. rock granulates). While most codes avoid non-spherical or non-analytical shapes due to the computational complexity, we introduce an iterative-based contact detection method for triangulated geometries. The new method is an improvement over a naive brute force approach which checks all possible geometric constellations of contact and thus exhibits a lot of execution branching. Our iterative approach has limited branching and high floating point operations per processed byte. It thus is suitable for modern Single Instruction Multiple Data (SIMD) CPU hardware. As only the naive brute force approach is robust and always yields a correct solution, we propose a hybrid solution that combines the best of the two worlds to produce fast and robust contacts. In terms of the DEM workflow, we furthermore propose a multilevel tree-based data structure strategy that holds all particles in the domain on multiple scales in grids. Grids reduce the total computational complexity of the simulation. The data structure is combined with the DEM phases to form a single touch tree-based traversal that identifies both contact points between particle pairs and introduces concurrency to the system during particle comparisons in one multiscale grid sweep. Finally, a reluctant adaptivity variant is introduced which enables us to realise an improved time stepping scheme with larger time steps than standard adaptivity while we still minimise the grid administration overhead. Four different parallelisation strategies that exploit multicore architectures are discussed for the triad of methodological ingredients. Each parallelisation scheme exhibits unique behaviour depending on the grid and particle geometry at hand. The fusion of them into a task-based parallelisation workflow yields promising speedups. Our work shows that new computer architecture can push the boundary of DEM computability but this is only possible if the right data structures and algorithms are chosen