Barad, Michael F.

Kiris, Cetin C.

Kocheemoolayil, Joseph G.

Stich, Gerrit-Daniel

NASA Technical Reports Server

Towards NASA’s In House 
Lattice-Boltzmann Solver
Joseph G. Kocheemoolayil, Michael F. Barad, 
Gerrit-Daniel Stich, Cetin C. Kiris
Computational Aerosciences Branch
NASA Ames Research Center
High-Fidelity Industrial LES/DNS Symposium (HiFiLeD)
Nov 14-16, 2018, Brussels, Belgium
https://ntrs.nasa.gov/search.jsp?R=20190028243 2019-09-26T20:11:39+00:00Z
ü Computational Requirements
• Space-time resolution requirements for acoustics problems are demanding
• Resources used for Cartesian Navier-Stokes examples shown above:
• Launch Environment: ~200 million cells, ~7 days of wall time (1000 cores)
• Parachute: 200 million cells, 3 days of wall time (2000 cores)
• Contra-Rotating Open Rotor: 360 million cells, 14 days (1400 cores)
• Launch Abort System: 400 million cells, 28 days of wall time (2000 cores)
• Landing Gear: 298 million cells, 20 days of wall time (3000 cores)
• LAVA Cartesian infrastructure has been re-factored into Navier-Stokes (NS) and Lattice 
Boltzmann Method (LBM)
• 10-50 times speed-up can be achieved with LBM vs NS-WENO without any 
compromise in accuracy or robustness
Challenges in Computational Aero-Acoustics
2
• Physics:
• Governs space time evolution of Density Distribution Functions
• Equilibrium distribution functions are truncated Maxwell-Boltzmann distributions
• Relaxation time related to kinematic viscosity
• Pressure related to density through the isothermal ideal gas law
• Lattice Boltzmann Equations (LBE) recover the Navier-Stokes equations in the 
low Mach number limit
• Numerics:
• Extremely efficient ‘collide at nodes and stream along links’ discrete analog to the 
Boltzmann equation 
• Particles bound to a regularly spaced lattice collide at nodes relaxing towards the 
local equilibrium
• Post-collision distribution functions hop on to neighboring nodes along the lattice 
links – Exact, dissipation-free advection from simple ‘copy’ operation   
• Macroscopic quantities such as density and momentum are moments of the 
density distribution functions in the discrete velocity space 
LAVA LBM: Governing Equations
3
LAVA LBM: Progress
Implementation to Date:
• Lattices: including D2Q9, D3Q15, D3Q19, D3Q27, D3Q39 …
• Collision Models:  
• Bhatnagar-Gross-Krook (BGK) 
• Multi-Relaxation Time (MRT)
• Entropic and positivity preserving variants of BGK
• Entropic Multi-Relaxation Time (EMRT)
• Regularized BGK
• Turbulence Models: Smagorinsky, Vreman, Sigma and Spalart-Allmaras models 
• Wall Models: Tamm-Mott-Smith boundary condition, filter-based slip wall model, Wall 
functions based on log law and power law
• Parallelization:
• Structured adaptive mesh refinement (SAMR) based LBM requires parallel ghost cell 
exchanges: 
• Fine-fine for communication within levels
• Conservative Coarse-fine interface treatment
• Efficient parallel I/O
• Multi-Resolution with Recursive Sub-Cycling 
• Boundary Conditions:
• No-slip and slip bounce back walls
• Accurate and robust curved walls (stationary and moving)
• Inflow/outflow, and periodic
D3Q19D2Q9
D2Q9 = 2D w/ 9-velocities…
Level=1Level=0 Level=2
t=dt0
t=dt2
t=0
t=dt =2*dt21
(coarse) (medium) (fine)
T
I
M
E
RESOLUTION
LAVA LBM: Verification and Validation
Turbulent Taylor Green Vortex 
Breakdown Test Case:
• Motivation:
• Simple low speed workshop case for 
testing high-order solvers
• Illustrates ability of solver to simulate 
turbulent energy cascade
• Periodic boundary conditions
• Setup:
• Analytic initial condition
• Mach = 0.1
• Reynolds Number = 1600
• Triply periodic flow in a box
• Comparisons:
• LAVA’s Lattice Boltzmann (LB) solver 
captures the turbulent kinetic energy 
cascade from large scales to small 
scales extremely well.
• Performance compared to LAVA’s 
Cartesian grid Navier-Stokes WENO 
solver showed a factor of 50 speedup.
 0
 0.2
 0.4
 0.6
 0.8
 1
 0  5  10  15  20
E(
t)/
E(
0)
t
Lattice Boltzmann (1283)
Navier-Stokes (1283)
Reference (5123)
Taylor Green kinetic energy decay using LAVA Solvers
Taylor Green vorticity breakdown. Image credit: 3rd International 
Workshop on High-Order CFD Methods (Beck et al) 
CFD
LAVA LBM: Verification and Validation
6
-3
-2
-1
 0
 1
 2
 3
-1  0  1  2  3  4  5  6
y/D
U/U0 + x/D - 1.0
Circles - Simulations (Black - DNS at Re = 3300 (Wissink and Rodi),
 Red - LES (Kravchenko and Moin))
 Squares - PIV measurements (Parnaudeau et al.) 
 Blue lines - LAVA-Navier-Stokes (Solid - 8M cells, Dashed - 1M cells)
-3
-2
-1
 0
 1
 2
 3
-1  0  1  2  3  4  5  6
y/D
U/U0 + x/D - 1.0
Circles - Simulations (Black - DNS at Re = 3300 (Wissink and Rodi),
 Red - LES (Kravchenko and Moin))
 Squares - PIV measurements (Parnaudeau et al.) 
 Black lines - LAVA-Lattice-Boltzmann (Solid - 8M cells, Dashed - 1M cells)
LES of Flow Past a Cylinder
• Well documented prototypical turbulent separated 
flow
• Detailed comparisons made with measurements and 
benchmark simulations
• Setup: Reynolds number = 3900
• Comparisons:
• LBM at 1M and 8M compares well with DNS @ 
400M (M = million points)
• 20x speedup even with embedded geometry
• Good comparison with benchmark datasets (PIV, 
LES, DNS) even with just 8 lattice nodes across the 
cylinder
• More accurate than high-order upwind biased NS 
schemes for identical resolution
Navier-Stokes
Lattice BoltzmannLattice Boltzmann (passive particles for visualization)
Cavity-Closed Nose Landing Gear
7
Grid Topology and Computational Setup
Mach = 0.166
Re = 66423 (D=Dstrut)
Uref = 58.32 m/s
Tref = 307.05 K
Pref = 98605 Pa
No-slip BC 
on landing gear
Subson
ic inflow
Subson
ic outflo
wFar-field BC
Setup follows the partially-dressed, cavity-closed nose landing gear (PDCC-
NLG) noise problem from AIAA’s Benchmark problems for Airframe Noise 
Computations (BANC) series of workshops. (Problem 4. Nose landing gear)
https://info.aiaa.org/tac/ASG/FDTC/DG/BECAN_files_/BANCIII.htm
LAVA Cartesian options:
• LBM uses EMRT with 
D3Q27
• NS uses WENO5 or 
WENO6 (as noted)
Cartesian Grid Resolution
8
9 Levels (56M) 10 Levels (91 M)
11 Levels (260M) 12 Levels (1.6B)
Δx = 3.91e-3m Δx = 1.95e-3m 
Δx = 9.77e-4m Δx = 4.88e-4m 
Grid Sensitivity: Vorticity Colored by Mach
9
9 Levels (56M) 10 Levels (91 M)
11 Levels (260M) 12 Levels (1.6B)
Velocity Magnitude (Center-plane)
10
LBM @ 1.6 billion: expense = 7.9 normalized wall time units (relative to 260M calc)
Passive Particle Colored by Mach Number
11
LBM @ 1.6 billion
Grid Sensitivity - PSD
12
Channel 5: Upper Drag Link
102 103 104
Frequency (Hz)
10 14
10 13
10 12
10 11
10 10
10 9
10 8
10 7
10 6
10 5
10 4
P
S
D
(p
si
2 /
H
z)
Channel 5
LB: 90 Million
LB: 260 Million
LB: 1.6 Billion
EXP-UFAFF
Near Field PSD
Frequency (Hz)
102 103 104
10-4
10-7
10-10
10-13
3
4
8
5
7
13
15
1012
PS
D
(p
si
2 /H
z)
Surface Pressure Spectra at Sensor Locations
9
BANCIII
Submissions
LBM vs NS - PSD
13
Channel 5: Upper Drag Link
102 103 104
Frequency (Hz)
10 14
10 13
10 12
10 11
10 10
10 9
10 8
10 7
10 6
10 5
10 4
P
S
D
(p
si
2 /
H
z)
Channel 5
NS-WENO6: 298 Million
NS-WENO5: 298 Million
LB: 260 Million
EXP-UFAFF
Near Field PSD
Frequency (Hz)
102 103 104
10-4
10-7
10-10
10-13
3
4
8
5
7
13
15
1012
PS
D
(p
si
2 /H
z)
LBM vs NS - PSD
14
Channel 13: Outer Wheel
102 103 104
Frequency (Hz)
10 14
10 13
10 12
10 11
10 10
10 9
10 8
10 7
10 6
10 5
10 4
P
S
D
(p
si
2 /
H
z)
Channel 13
NS-WENO6: 298 Million
NS-WENO5: 298 Million
LB: 260 Million
EXP-UFAFF
Near Field PSD
Frequency (Hz)
102 103 104
10-4
10-7
10-10
10-13
3
4
8
5
7
13
15
1012
PS
D
(p
si
2 /H
z)
Grid and Performance Statistics
15
Method CPU Cores(type)
Cells 
(million)
Wall Days
to 0.19 sec
Core Days 
to 0.19 sec
Relative 
SBU 
Expense
NS-GCM 3000 (ivy) 298 20.5 61352 12.1
NS-IIM 9600 (has) 222 6.1 58490 15.3
LBM 1400 (bro) 260 2.25 3156 1
• For a comparable mesh size, LBM is 12-15 times faster computationally than Navier-Stokes and is 
equally accurate. “Apples-to-apples” comparison with the exact same mesh & CPU-type is ongoing. 
Note: LBM code is not yet optimized, and we output volume data every 50 steps!
• LBM at 1.6 billion cells is ~2 times faster than NS at 298 million. This is a key enabler for 
unprecedented high resolution simulations.
• Performance details:
• Both Cartesian Navier-Stokes and LBM are memory-bound (not compute-bound) algorithms, the 
latter much more so than the former. 
• Non-linear, LBM collision operation where all the work happens is entirely local!! Data locality is 
critical to the computational efficiency of LBM relative to high-order Cartesian NS codes.
Velocity Magnitude (Center-plane)
16
NS-GCM @ 298 million: expense = 12.1NS-IIM @ 222 million: : expense = 15.3 
LBM @ 260 million: expense = 1.0 LBM @ 90 million: expense = 0.182
• RANS unreliable beyond 14°
• Higher fidelity approaches with fast 
turnaround times necessary
17
RANS CFD
Experimental 
Oilflow
AIAA High Lift Prediction Workshop 3 
⍺=18.58°
NASA 2-D Hump – Experimental Setup
2@D$NASA$Hump$
Greenblal$et$al$
•  RaVonale$for:$excellent$high@quality$reference$experimental$
data$set;$good$2@D$characterisVcs;$includes$both$baseline$and$
ﬂow$control;$RANS$known$to$do$poorly;$eddy@resolving$
methods$have$been$shown$to$do$well;$well@veled$in$previous$
workshop$
•  RaVonale$against:$endplates$introduced$some$blockage$
5$
M=0.1 
Rec=0.936 million 
- Greenblatt, D., Paschal, K. B., Yao, C.-S., Harris, J., Schaeffler, N. W., Washburn, A. E., 
“Experimental Investigation of Separation Control Part 1: Baseline and Steady Suction,” AIAA 
Journal, vol. 44, no. 12, pp. 2820-2830, 2006. 
- Rumsey, C. L., “Turbulence Modeling Resource,” http://turbmodels.larc.nasa.gov (data posted 
online), and “CFD Validation of Synthetic Jets and Turbulent Separation Control,” 
http://cfdval2004.larc.nasa.gov (data posted online). 
 
 
Wall-resolved LES:
ü Uzun, A. and Malik, M. (AIAA 2017-5308)
Wall-modeled LES:
ü Iyer, P. and Malik, M. (AIAA 2016-3186)
Lattice Boltzmann Methods:
ü Duda, B. and Fares, E. (AIAA 2016-1836)
ü Assess ability of CFD solvers to predict flow separation from a 
smooth body (caused by adverse pressure gradient) as well as 
subsequent reattachment and boundary layer recovery.
1 Greenblatt et. Al. “Experimental Investigation of Separation Control Part 1: Baseline and Steady Suction”. AIAA 
Journal, vol 44, no. 12, pp. 2820-2830, 2006
2 Rumsey C, “Turbulence Modeling Resource”, https://turbmodels.larc.nasa.gov
3 Rumsey C, “CFD Validation f Synthetic Jets and Turbulent Separation Control”, http://cfdval2004.larc.nasa.gov
18
ü Lattice: D3Q27
ü Collision Model: EMRT
ü Synthetic Eddy Method with scaled DNS Flat plate Data at x/c = -3.0
Isocontour of Q-citerion colored by 
normalized streamwise velocity
19
ü Periodic BCs in spanwise direction (Side walls not modeled)
Application of the Lattice Boltzmann Method
Application of the Lattice Boltzmann Method
ü 5 Refinement Levels
ü Refinement ratio of 2:1
ü Level 3 in regions of high vorticity
ü Level 4 on all viscous walls
ü Level 5 from x/c = -0.2 to 1.3
ü 105 million points
ü Spanwise extent = 0.2c
ü ∆+ ≈ 50 in viscous wall units
ü Local as well as adaptive mesh refinement well tested in our Cartesian 
framework.
20
NASA 2-D Hump – Application of Lattice Boltzmann
Streamwise Velocity 
ü Excellent agreement with measurements 
21
NASA 2-D Hump – Application of Lattice Boltzmann
Streamwise Velocity 
22
96 % 
improvement
solver x/c [-] Error [%]
Greenblatt 1.105 -
RANS from TMR 1.26 14.02
DDES 1.34 21.26
DDES + SEM 1.23 11.31
ZDES + SEM 1.11 0.45
NASA 2-D Hump – Application of Lattice Boltzmann
Streamwise Reynolds Stress 
ü Encouraging agreement with experiment for turbulence intensity profiles
23
U/Uref
y/
c
-0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.20.00
0.05
0.10
0.15
Greenblatt et.al.
LAVA ZDES Mode 3 y+ < 1 (11.3M cells)
LAVA LBM y+ < 50 (105M cells)
UZUN Extended Span y+ < 1 (420M cells)
PowerFlow fine y+ < 75 (148.9M cells)
PowerFlow periodic y+ < 75 (148.9M cells)
x/c = 0.65
NASA 2-D Hump – Application of Lattice Boltzmann
24
u’v’/Uref2
y/
c
-0.04 -0.03 -0.02 -0.01 0.000.00
0.05
0.10
0.15
x/c = 1.10
RANS
Improved peak-shear 
stress magnitude
18
Towards Urban Air Mobility (UAM)
High-Fidelity Modeling and Optimization Method Development
NASA Revolutionary Vertical Lift Technology Rotary Project (RVLT)
Picture credit: NASA / Lillian Gipson
Isolated UAS Rotor in Hover Validation
Objective:
ü Validate LAVA for RVLT applications
ü Assess pros and cons of body-
fitted/Cartesian Grid as well as 
Navier-Stokes/Lattice Boltzmann 
approaches
Computational Methodology :
ü Navier-Stokes (NS) URANS solver on 
Structured Overset Grid 
ü Navier-Stokes as well as Lattice 
Boltzmann (LB) on Cartesian Grid
Validation:
ü Propeller Performance
ü Far-field Acoustics 
Closeup of setup in LSAWT facility
LAVA uRANS simulation at 5400 RPM
Experimental Data from Zawodny and Haskin AIAA-2017-3709 26
Isolated UAS Rotor in Hover Validation
B. UAS Vehicle/Component Testing
The mobile MTS will be reconfigured to mount both single rotor-motor systems and full multi-copter UAS
platforms. Figure 12 presents visualizations and component breakdowns of these configurations. In addition
to far-field acoustics, the test setup is also able to be configured for acquiring rotor/vehicle performance data
using multi-axis load cells. Section IV.B provides static performance and acoustic data for a small isolated
rotor in hover conditions as a demonstration of these capabilities.
(a) Small Quad-copter UAS Configuration
Motor-Rotor 
Assembly
Motor Mount
Multi-Axis 
Load Cell
Support Rod
Nose Cone Sting 
Mount
(b) Single Propeller/Rotor (c) Full Vehicle
Figure 12. Visualization of UAS testing configuration and associated hardware in LSAWT.
IV. Preliminary Results
The primary goal of this study is to demonstrate the capabilities of the newly configured NASA Langley
LSAWT for aerodynamic and acoustic testing of small propeller and UAS rotor configurations. The follow-
ing sections document results in the form of small propeller CFD predictions and their incorporation into
the expected LSAWT facility operational limits, and isolated UAS rotor hover measurements. The rotor
measurements are further compared with data acquired in an anechoic chamber on the same tested rotor
and with acoustic predictions performed using the Propeller Analysis System (PAS) of the NASA Aircraft
NOise Prediction Program (ANOPP).9
10 of 17
American Institute of Aeronautics and Astronautics
Do
wn
loa
de
d b
y N
AS
A 
AM
ES
 R
ES
EA
RC
H 
CE
NT
ER
 on
 O
cto
be
r 9
, 2
01
8 |
 ht
tp:
//a
rc.
aia
a.o
rg 
| D
OI
: 1
0.2
51
4/6
.20
17
-37
09
 
ü Experiments conducted at NASA Langley 
LSAWT as well s in the Stru tural Acou ti s 
Loads and Transmission (SALT) anechoic 
chamber. 
ü Motor-Rot r As embly as well as Mount and 
Support structure ot considered in 
simulations.
Zawodny and Haskin
(AIAA-2017-3709)
Rotor Span R 0.1905 [m]
Microphones (M1-M5) 10R
Considered RPM 5400 
27
Experim ntal Data f m Zawodny and 
Haskin AIAA-2017-3709
31
LAVA Cartesian Methods
ü Refinement ratio of 2:1
ü Very Coarse : 40% tip chord (  8lev)
ü Coarse         : 20% tip chord (  9lev)
ü Medium        : 10% tip chord (10lev)
ü Fine              :   5% tip chord (11lev)
Isocontour of Q-criterion colored by Pressure. Simulation on medium Cartesian mesh.
Lattice Boltzmann
(LBM – EMRT)
Navier-Stokes
(NS – WENO6)
32
Lattice Boltzmann Method
Rotor Performance at 5400 RPM  
ü Excellent agreement with experimental measurements
ü Differences (< 1%) well within measurement uncertainty (highlighted in blue) 
33
Lattice Boltzmann Method
Farfield Noise – SPL Spectrum for Observer M1 & M3 
FL
O
W
ü Excellent agreement with BPF1-BPF5 
for M1 (0.0°) microphone location
ü Excellent agreement with 
BPF1 & BPF2 for M3 (45.0°)
ü Different FWH formulations (permeable 
and impermeable) currently under 
investigation
M1 M3
34
Lattice Boltzmann Method
Farfield Noise – Mesh Refinement Study
ü Consistent agreement for BPF1 on all mesh levels, BPF2 more sensitive.
ü Good agreement for BPF 1 even on very coarse mesh.
35
Comparison between the Approaches
ü Consistent prediction using all three approaches 
ü Computational efficiency and complete absence of manual volume mesh 
generation key advantage of LBM
ü Manual meshing efforts increase significantly upon considering installation 
effects (e.g. full Quadcopter or tiltwing urban air taxis)
M1 M3
Summary
36
LAVA Lattice Boltzmann Solver has made significant progress towards becoming a 
work-horse for NASA mission critical applications:
• Ultra-high performance without any compromise in fidelity
• Completely automated workflow without labor intensive mesh generation
Acknowledgments
37
ü This work was partially supported by the NASA ARMD’s Transformational Tools 
and Technologies (T^3), Advanced Air Transport Technology (AATT), 
Revolutionary Vertical Lift Technology (RVLT) projects
ü Computer time provided by NASA Advanced Supercomputing (NAS) facility at 
NASA Ames Research Center
Questions?
Computational Resources for Wall Mounted Hump case
ü All simulations performed on NASA Pleiades Cluster  using Intel Xeon E5-2680v4
39
LAVA curvilinear ZDES Mode 3 : 240 CPUh/CTU
LAVA LBM-LES : 614 CPUh/CTU
2.5x slower
LAVA curvilinear  ZDES Mode 3 : ∆t = 1.8E-5
LAVA LBM-LES : ∆t = 4.6E-7 39x smaller
LAVA curvilinear ZDES Mode 3 : 11.3M grid points
Lattice LBM-LES : 105M grid points
9x more 
Exa Powerflow LBM-VLES1 : 540 CPUh/CTU very close to LAVA-LBM timings
1Duda and Fares. Apllication of a Lattice-Boltzmann Method to the Separated flow over the NASA hump. (AIAA-2016-1836)


Towards NASA's In House Lattice-Boltzmann Solver

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server