Highly Scalable Asynchronous Computing Method for Partial Differential Equations: A Path Towards Exascale

Many natural and engineering systems are governed by nonlinear partial differential equations (PDEs) which result in a multiscale phenomena, e.g. turbulent flows. Numerical simulations of these problems are computationally very expensive and demand for extreme levels of parallelism. At realistic conditions, simulations are being carried out on massively parallel computers with hundreds of thousands of processing elements (PEs). It has been observed that communication between PEs as well as their synchronization at these extreme scales take up a significant portion of the total simulation time and result in poor scalability of codes. This issue is likely to pose a bottleneck in scalability of codes on future Exascale systems. In this work, we propose an asynchronous computing algorithm based on widely used finite difference methods to solve PDEs in which synchronization between PEs due to communication is relaxed at a mathematical level. We show that while stability is conserved when schemes are used asynchronously, accuracy is greatly degraded. Since message arrivals at PEs are random processes, so is the behavior of the error. We propose a new statistical framework in which we show that average errors drop always to first-order regardless of the original scheme. We propose new asynchrony-tolerant schemes that maintain accuracy when synchronization is relaxed. The quality of the solution is shown to depend, not only on the physical phenomena and numerical schemes, but also on the characteristics of the computing machine. A novel algorithm using remote memory access communications has been developed to demonstrate excellent scalability of the method for large-scale computing. Finally, we present a path to extend this method in solving complex multi-scale problems on Exascale machines

A co-kurtosis based dimensionality reduction method for combustion datasets

Principal Component Analysis (PCA) is a dimensionality reduction technique widely used to reduce the computational cost associated with numerical simulations of combustion phenomena. However, PCA, which transforms the thermo-chemical state space based on eigenvectors of co-variance of the data, could fail to capture information regarding important localized chemical dynamics, such as the formation of ignition kernels, appearing as outlier samples in a dataset. In this paper we propose an alternate dimensionality reduction procedure, co-kurtosis PCA (CoK-PCA), wherein the required principal vectors are computed from a high-order joint statistical moment, namely the co-kurtosis tensor, which may better identify directions in the state space that represent stiff dynamics. We first demonstrate the potential of the proposed CoK-PCA method using a synthetically generated dataset that is representative of typical combustion simulations. Thereafter, we characterize and contrast the accuracy of CoK-PCA against PCA for datasets representing spontaneous ignition of premixed ethylene in a simple homogeneous reactor and ethanol-fueled homogeneous charged compression ignition (HCCI) engine. Specifically, we compare the low-dimensional manifolds in terms of reconstruction errors of the original thermo-chemical state, and species production and heat release rates computed from the reconstructed state. We find that, even using a simplistic linear reconstruction, the co-kurtosis based reduced manifold represents the original thermo-chemical state more accurately than PCA, especially in the regions where chemical reactions are important. We observe that the accuracy of the CoK-PCA can be further improved by adopting many of the refinements (e.g., non-linear reconstruction, localized manifolds) already widely in use with PCA

Diva: A Declarative and Reactive Language for In-Situ Visualization

The use of adaptive workflow management for in situ visualization and analysis has been a growing trend in large-scale scientific simulations. However, coordinating adaptive workflows with traditional procedural programming languages can be difficult because system flow is determined by unpredictable scientific phenomena, which often appear in an unknown order and can evade event handling. This makes the implementation of adaptive workflows tedious and error-prone. Recently, reactive and declarative programming paradigms have been recognized as well-suited solutions to similar problems in other domains. However, there is a dearth of research on adapting these approaches to in situ visualization and analysis. With this paper, we present a language design and runtime system for developing adaptive systems through a declarative and reactive programming paradigm. We illustrate how an adaptive workflow programming system is implemented using our approach and demonstrate it with a use case from a combustion simulation.Comment: 11 pages, 5 figures, 6 listings, 1 table, to be published in LDAV 2020. The article has gone through 2 major revisions: Emphasized contributions, features and examples. Addressed connections between DIVA and FRP. In sec. 3, we fixed a design flaw and addressed it in sec. 3.3-3.4. Re-designed sec. 5 with a more concrete example and benchmark results. Simplified the syntax of DIV

Evaluation of finite difference based asynchronous partial differential equations solver for reacting flows

Next-generation exascale machines with extreme levels of parallelism will provide massive computing resources for large scale numerical simulations of complex physical systems at unprecedented parameter ranges. However, novel numerical methods, scalable algorithms and re-design of current state-of-the art numerical solvers are required for scaling to these machines with minimal overheads. One such approach for partial differential equations based solvers involves computation of spatial derivatives with possibly delayed or asynchronous data using high-order asynchrony-tolerant (AT) schemes to facilitate mitigation of communication and synchronization bottlenecks without affecting the numerical accuracy. In the present study, an effective methodology of implementing temporal discretization using a multi-stage Runge-Kutta method with AT schemes is presented. Together these schemes are used to perform asynchronous simulations of canonical reacting flow problems, demonstrated in one-dimension including auto-ignition of a premixture, premixed flame propagation and non-premixed autoignition. Simulation results show that the AT schemes incur very small numerical errors in all key quantities of interest including stiff intermediate species despite delayed data at processing element (PE) boundaries. For simulations of supersonic flows, the degraded numerical accuracy of well-known shock-resolving WENO (weighted essentially non-oscillatory) schemes when used with relaxed synchronization is also discussed. To overcome this loss of accuracy, high-order AT-WENO schemes are derived and tested on linear and non-linear equations. Finally the novel AT-WENO schemes are demonstrated in the propagation of a detonation wave with delays at PE boundaries

The Turbulent Schmidt Number

We analyze a large database generated from recent direct numerical simulations (DNS) of passive scalars sustained by a homogeneous mean gradient and mixed by homogeneous and isotropic turbulence on grid resolutions of up to 4096(3) and extract the turbulent Schmidt number over large parameter ranges: the Taylor microscale Reynolds number between 8 and 650 and the molecular Schmidt number between 1/2048 and 1024. While the turbulent Schmidt number shows considerable scatter with respect to the Reynolds and molecular Schmidt numbers separately, it exhibits a sensibly unique functional dependence with respect to the molecular Peclet number. The observed functional dependence is motivated by a scaling argument that is standard in the phenomenology of three-dimensional turbulence

Direct numerical simulations of premixed and stratified flame propagation in turbulent channel flow

Direct numerical simulations are performed to investigate the transient upstream flame propagation (flashback) through homogeneous and fuel-stratified hydrogen-air mixtures transported in fully-developed turbulent channel flows. Results indicate that, for both cases, the flame maintains steady propagation against the bulk flow direction and the global flame shape and the local flame characteristics are both affected by the occurrence of fuel stratification. Globally, the mean flame shape undergoes an abrupt change when the approaching reactants transition from an homogeneous to a stratified mixing configuration. A V-shaped flame surface, whose leading-edge is located in the near-wall region, characterizes the non-stratified, homogeneous mixture case while a U-shaped flame surface, whose leading-edge propagates upstream at the channel centreline, distinguishes the case with fuel stratification (fuel-lean in the near-wall region and fuel-rich away from the wall). The characteristic thickness, wrinkling and displacement speed of the turbulent flame brush are subject to considerable changes across the channel due to the dependence of the turbulence and mixture properties on the distance from the channel walls. More specifically, the flame transitions from a moderately wrinkled, thin-flamelet combustion regime in the homogeneous mixture case to a strongly wrinkled flame brush more representative of a thickened-flame combustion regime in the near-wall region of the fuel-stratified case. The combustion regime may be related to Karlovitz number and it is shown that a nominal channel-flow Karlovitz number, Kachin, based on the wallnormal variation of canonical turbulence (tη = (ν/ε) 1/2) and chemistry (tl = δl/Sl) time scales in fully-developed channel flow, compares well with an effective Karlovitz number, Kachfl, extracted from the present DNS datasets using conditionally sampled values of tη and tl in the immediate vicinity of the flame (0.1 < C < 0.3)

Direct numerical simulations of premixed and stratified flame propagation in turbulent channel flow

Direct numerical simulations are performed to investigate the transient upstream flame propagation (flashback) through homogeneous and fuel-stratified hydrogen-air mixtures transported in fully developed turbulent channel flows. Results indicate that, for both cases, the flame maintains steady propagation against the bulk flow direction, and the global flame shape and the local flame characteristics are both affected by the occurrence of fuel stratification. Globally, the mean flame shape undergoes an abrupt change when the approaching reactants transition from an homogeneous to a stratified mixing configuration. A V-shaped flame surface, whose leading-edge is located in the near-wall region, characterizes the nonstratified, homogeneous mixture case, while a U-shaped flame surface, whose leading edge propagates upstream at the channel centerline, distinguishes the case with fuel stratification (fuel-lean in the near-wall region and fuel-rich away from the wall). The characteristic thickness, wrinkling, and displacement speed of the turbulent flame brush are subject to considerable changes across the channel due to the dependence of the turbulence and mixture properties on the distance from the channel walls. More specifically, the flame transitions from a moderately wrinkled, thin-flamelet combustion regime in the homogeneous mixture case to a strongly wrinkled flame brush more representative of a thickened-flame combustion regime in the near-wall region of the fuel-stratified case. The combustion regime may be related to the Karlovitz number, and it is shown that a nominal channel-flow Karlovitz number, Kach in , based on the wall-normal variation of canonical turbulence (tη = (ν/ ) 1/2) and chemistry (tl = δl/Sl) timescales in fully developed channel flow, compares well with an effective Karlovitz number, Kach fl , extracted from the present DNS datasets using conditionally sampled values of tη and tl in the immediate vicinity of the flame (0.1publishedVersion© 2018 American Physical Societ