A GPU-accelerated package for simulation of flow in nanoporous source rocks with many-body dissipative particle dynamics

Mesoscopic simulations of hydrocarbon flow in source shales are challenging, in part due to the heterogeneous shale pores with sizes ranging from a few nanometers to a few micrometers. Additionally, the sub-continuum fluid-fluid and fluid-solid interactions in nano- to micro-scale shale pores, which are physically and chemically sophisticated, must be captured. To address those challenges, we present a GPU-accelerated package for simulation of flow in nano- to micro-pore networks with a many-body dissipative particle dynamics (mDPD) mesoscale model. Based on a fully distributed parallel paradigm, the code offloads all intensive workloads on GPUs. Other advancements, such as smart particle packing and no-slip boundary condition in complex pore geometries, are also implemented for the construction and the simulation of the realistic shale pores from 3D nanometer-resolution stack images. Our code is validated for accuracy and compared against the CPU counterpart for speedup. In our benchmark tests, the code delivers nearly perfect strong scaling and weak scaling (with up to 512 million particles) on up to 512 K20X GPUs on Oak Ridge National Laboratory's (ORNL) Titan supercomputer. Moreover, a single-GPU benchmark on ORNL's SummitDev and IBM's AC922 suggests that the host-to-device NVLink can boost performance over PCIe by a remarkable 40\%. Lastly, we demonstrate, through a flow simulation in realistic shale pores, that the CPU counterpart requires 840 Power9 cores to rival the performance delivered by our package with four V100 GPUs on ORNL's Summit architecture. This simulation package enables quick-turnaround and high-throughput mesoscopic numerical simulations for investigating complex flow phenomena in nano- to micro-porous rocks with realistic pore geometries

Cell mechanics in flow: algorithms and applications

The computer simulations are pervasively used to improve the knowledge about biophysical phenomena and to quantify effects which are difficult to study experimentally. Generally, the numerical methods and models are desired to be as accurate as possible on the chosen length and time scales, but, at the same time, affordable in terms of computations. Until recently, the cell mechanics and blood flow phenomena on the sub-micron resolution could not be rigorously studied using computer simulations. However, within the last decade, advances in methods and hardware catalyzed the development of models for cells mechanics and blood flow modeling which, previously, were considered to be not feasible. In this context, a model should accurately describe a phenomenon, be computationally affordable, and be flexible to be applied to study different biophysical changes. This thesis focuses on the development of the new methods, models, and high-performance software implementation that expand the class of problems which can be studied numerically using particle-based methods. Microvascular networks have complex geometry, often without any symmetry, and to study them we need to tackle computational domains with several inlets and outlets. However, an absence of appropriate boundary conditions for particle- based methods hampers study of the blood flow in these domains. Another obstacle to model complex blood flow problems is the absence the highperformance software. This problem restricts the applicability of the of particlebased cell flow models to relatively small systems. Although there are several validated red blood cell models, to date, there are no models for suspended eukaryotic cells. The present thesis addresses these issues. We introduce new open boundary conditions for particle-based systems and apply them to study blood flow in a part of a microvascular network. We develop a software demonstrating outstanding performance on the largest supercomputers and used it to study blood flow in microfluidic devices. Finally, we present a new eukaryotic cell model which helps in quantifying the effect of sub-cellular components on the cell mechanics during deformations in microfluidic devices

Bridging the computational gap between mesoscopic and continuum modeling of red blood cells for fully resolved blood flow

We present a computational framework for the simulation of blood flow with fully resolved red blood cells (RBCs) using a modular approach that consists of a lattice Boltzmann solver for the blood plasma, a novel finite element based solver for the deformable bodies and an immersed boundary method for the fluid-solid interaction. For the RBCs, we propose a nodal projective FEM (npFEM) solver which has theoretical advantages over the more commonly used mass-spring systems (mesoscopic modeling), such as an unconditional stability, versatile material expressivity, and one set of parameters to fully describe the behavior of the body at any mesh resolution. At the same time, the method is substantially faster than other FEM solvers proposed in this field, and has an efficiency that is comparable to the one of mesoscopic models. At its core, the solver uses specially defined potential energies, and builds upon them a fast iterative procedure based on quasi-Newton techniques. For a known material, our solver has only one free parameter that demands tuning, related to the body viscoelasticity. In contrast, state-of-the-art solvers for deformable bodies have more free parameters, and the calibration of the models demands special assumptions regarding the mesh topology, which restrict their generality and mesh independence. We propose as well a modification to the potential energy proposed by Skalak et al. 1973 for the red blood cell membrane, which enhances the strain hardening behavior at higher deformations. Our viscoelastic model for the red blood cell, while simple enough and applicable to any kind of solver as a post-convergence step, can capture accurately the characteristic recovery time and tank-treading frequencies. The framework is validated using experimental data, and it proves to be scalable for multiple deformable bodies