Extending a serial 3D two-phase CFD code to parallel execution over MPI by using the PETSc library for domain decomposition

To leverage the last two decades' transition in High-Performance Computing (HPC) towards clusters of compute nodes bound together with fast interconnects, a modern scalable CFD code must be able to efficiently distribute work amongst several nodes using the Message Passing Interface (MPI). MPI can enable very large simulations running on very large clusters, but it is necessary that the bulk of the CFD code be written with MPI in mind, an obstacle to parallelizing an existing serial code. In this work we present the results of extending an existing two-phase 3D Navier-Stokes solver, which was completely serial, to a parallel execution model using MPI. The 3D Navier-Stokes equations for two immiscible incompressible fluids are solved by the continuum surface force method, while the location of the interface is determined by the level-set method. We employ the Portable Extensible Toolkit for Scientific Computing (PETSc) for domain decomposition (DD) in a framework where only a fraction of the code needs to be altered. We study the strong and weak scaling of the resulting code. Cases are studied that are relevant to the fundamental understanding of oil/water separation in electrocoalescers.Comment: 8 pages, 6 figures, final version for to the CFD 2014 conferenc

The prospect of using LES and DES in engineering design, and the research required to get there

In this paper we try to look into the future to divine how large eddy and detached eddy simulations (LES and DES, respectively) will be used in the engineering design process about 20-30 years from now. Some key challenges specific to the engineering design process are identified, and some of the critical outstanding problems and promising research directions are discussed.Comment: accepted for publication in the Royal Society Philosophical Transactions

Eulerian-Lagrangian method for simulation of cloud cavitation

We present a coupled Eulerian-Lagrangian method to simulate cloud cavitation in a compressible liquid. The method is designed to capture the strong, volumetric oscillations of each bubble and the bubble-scattered acoustics. The dynamics of the bubbly mixture is formulated using volume-averaged equations of motion. The continuous phase is discretized on an Eulerian grid and integrated using a high-order, finite-volume weighted essentially non-oscillatory (WENO) scheme, while the gas phase is modeled as spherical, Lagrangian point-bubbles at the sub-grid scale, each of whose radial evolution is tracked by solving the Keller-Miksis equation. The volume of bubbles is mapped onto the Eulerian grid as the void fraction by using a regularization (smearing) kernel. In the most general case, where the bubble distribution is arbitrary, three-dimensional Cartesian grids are used for spatial discretization. In order to reduce the computational cost for problems possessing translational or rotational homogeneities, we spatially average the governing equations along the direction of symmetry and discretize the continuous phase on two-dimensional or axi-symmetric grids, respectively. We specify a regularization kernel that maps the three-dimensional distribution of bubbles onto the field of an averaged two-dimensional or axi-symmetric void fraction. A closure is developed to model the pressure fluctuations at the sub-grid scale as synthetic noise. For the examples considered here, modeling the sub-grid pressure fluctuations as white noise agrees a priori with computed distributions from three-dimensional simulations, and suffices, a posteriori, to accurately reproduce the statistics of the bubble dynamics. The numerical method and its verification are described by considering test cases of the dynamics of a single bubble and cloud cavitaiton induced by ultrasound fields.Comment: 28 pages, 16 figure

Single-bubble dynamics in histotripsy and high-amplitude ultrasound: Modeling and validation

A variety of approaches have been used to model the dynamics of a single, isolated bubble nucleated by a microsecond length high-amplitude ultrasound pulse (e.g., a histotripsy pulse). Until recently, the lack of single--bubble experimental radius vs. time data for bubble dynamics under a well-characterized driving pressure has limited model validation efforts. This study uses radius vs. time measurements of single, spherical histotripsy-nucleated bubbles in water [Wilson et al., Phys. Rev. E, 2019, 99, 043103] to quantitatively compare and validate a variety of bubble dynamics modeling approaches, including compressible and incompressible models as well as different thermal models. A strategy for inferring an analytic representation of histotripsy waveforms directly from experimental radius vs. time and cavitation threshold data is presented. We compare distributions of a calculated validation metric obtained for each model applied to $88$ experimental data sets. There is minimal distinction ($< 1\%$) among the modeling approaches for compressibility and thermal effects considered in this study. These results suggest that our proposed strategy to infer the waveform, combined with simple models minimizing parametric uncertainty and computational resource demands accurately represent single-bubble dynamics in histotripsy, including at and near the maximum bubble radius. Remaining sources of parametric and model-based uncertainty are discussed

EPSILOD: efficient parallel skeleton for generic iterative stencil computations in distributed GPUs

Producción CientíficaIterative stencil computations are widely used in numerical simulations. They present a high degree of parallelism, high locality and mostly-coalesced memory access patterns. Therefore, GPUs are good candidates to speed up their computa- tion. However, the development of stencil programs that can work with huge grids in distributed systems with multiple GPUs is not straightforward, since it requires solv- ing problems related to the partition of the grid across nodes and devices, and the synchronization and data movement across remote GPUs. In this work, we present EPSILOD, a high-productivity parallel programming skeleton for iterative stencil computations on distributed multi-GPUs, of the same or different vendors that sup- ports any type of n-dimensional geometric stencils of any order. It uses an abstract specification of the stencil pattern (neighbors and weights) to internally derive the data partition, synchronizations and communications. Computation is split to better overlap with communications. This paper describes the underlying architecture of EPSILOD, its main components, and presents an experimental evaluation to show the benefits of our approach, including a comparison with another state-of-the-art solution. The experimental results show that EPSILOD is faster and shows good strong and weak scalability for platforms with both homogeneous and heterogene- ous types of GPUJunta de Castilla y León, Ministerio de Economía, Industria y Competitividad, y Fondo Europeo de Desarrollo Regional (FEDER): Proyecto PCAS (TIN2017-88614-R) y Proyecto PROPHET-2 (VA226P20).Ministerio de Ciencia e Innovación, Agencia Estatal de Investigación y “European Union NextGenerationEU/PRTR” : (MCIN/ AEI/10.13039/501100011033) - grant TED2021-130367B-I00CTE-POWER and Minotauro and the technical support provided by Barcelona Supercomputing Center (RES-IM-2021-2-0005, RES-IM-2021-3-0024, RES- IM-2022-1-0014).Publicación en abierto financiada por el Consorcio de Bibliotecas Universitarias de Castilla y León (BUCLE), con cargo al Programa Operativo 2014ES16RFOP009 FEDER 2014-2020 DE CASTILLA Y LEÓN, Actuación:20007-CL - Apoyo Consorcio BUCL

Solution of cavitating compressible flows using Discontinuous Galerkin discretisation

A methodology for modelling cavitating flows using a high-order Adaptive Mesh Refinement (AMR) approach based on the Discontinuous Galerkin method (DG) is presented. The AMR implementation used features on-the-fly adaptive mesh refinement for unstructured hybrid meshes. The specific implementation has been developed for the resolution of complex multi-scale phenomena where high accuracy p-adaptive discretisations are combined with an h-adaptive data structure. This approach accommodates the fine spatial resolution for the interface discontinuities and the shock waves observed in compressible cavitating flows. The Tait equation of state is used for the modelling of the liquid phase while an isentropic path is assumed for the liquid/vapour mixture. Second order spatial and a third order non-oscillatory temporal discretisation are used for the integration of the mass and momentum conservation equations, in order to resolve the flow structures responsible for the formation of cavitation bubbles and the resulting compression waves. Assessment of the developed methodology is performed for the one-dimensional advancement of a compressible liquid-vapour interface and the symmetric collapse of a spherical vapour bubble. Following, results obtained with the developed multi-scale modelling AMR approach has revealed a complex bubble collapse mechanism near a rigid wall, providing evidence of processes that have been unknown before due to reduced resolution and dissipative nature of past simulations. The impinging jet accompanying the collapse of a bubble near a wall, was found to induce vortical structures, which result to the formation of a secondary cavitation of a wall-attached bubble at the vicinity of the impingement jet shear layer. At the final stages of the initial bubble collapse, the impinging jet was found to penetrate the centre-line of the wall bubble inducing its partial collapse. This secondary collapse results to a rich spatial structure of shock waves, interacting with the secondary bubbles. Moreover, the calculated pressure level are found to be much higher than those reported from previous methodologies