Jansson, J.

Krishnasamy, E.

Leoni, M.

English

BCAM's Institutional Repository Data

Adaptive Direct FEM Simulation withUnicorn/FEniCS-HPC for CS1Johan Jansson12, Ezhilmathi Krishnasamy2, Massimiliano Leoni12Collaborators: Cem Degirmenci, Laura Saavedra, Johan Ho↵man, Niclas Jansson,Philipp Schlatter, Ricardo Vinuesa, Jing Gong, Peter VincentComputational Science and Technology CSC, KTH, Stockholm [1]BCAM - Basque Center for Applied Mathematics, Bilbao [2]5th International Workshop on High Order CFD MethodsKissimmee, FL, January 6-7, 2018Aims and outlineAdaptive Direct FEM Simulation (DFS) results with P1 for CS1 together withcomparison to workshop-provided meshes.Preliminary comparison to other leading high-order open source packages: PyFR andNek5000Draft paper available on my home page: http://www.csc.kth.se/~jjanPart of broader program for predictive adaptive DFS high and low Re simulation forindustrial challenge problems.[stall prediction in HiLiftPW3]Automated Computational ModelingFEniCS(-HPC) open source FEM framework for automated solution of general PDE and DirectFEM Simulation (DFS) methodology enables:Automated discretization (generate code for linear system from PDE):r = ( inner ( grad ( u ) , grad ( v ) )   inner (f , v ))⇤dx ) Poisson.cppAutomated error control (incl. parallel adaptive mesh refinement):+ |M(e)|  TOL )with M(e) a goal functional of the computational error e = u   U.Automated modeling of unresolved subscales (i.e. turbulence):(R(U), v) + (hR(U),R(v)) = 0, 8v 2 Vh (ILES: residual-based stabilization, Adaptive DNS/LES)Goal: Autom. generate the solution, mesh and program from PDE(residual) and goal functional M(e) (e.g. drag).Direct FEM Simulation (DFS) methodologyDeveloped over a 20+ year period by Johnson, Ho↵man, Jansson, etc.Realized in the open source Unicorn/FEniCS-HPC software framework (available on BitBucket).Incompressible Navier-Stokes in strong residual form:R(û) =(@t u + (u · r)u   ⌫ u + rp = 0r · u = 0û = (u, p)Weak residual r(û, v̂) = (R(û), v̂)No explicit turbulence modelSpace-time cG(1)cG(1) FEM with Galerkin/least squares stabilization (acting as ILES)r̄(Û, v̂) = r(û, v̂) + ( R(Û), R(v̂)) = 0  = min(h), ⇡ 1, 8v̂ 2 V̂h, Û 2 V̂hDo-nothing duality-based adaptive error control methodDirectly use err repr. M(ê) = r(Û,  ̂) gives error indicator EK = r(Û,  ̂)KAdjoint prob. autom. gen.: r0( ̂, v̂) = M(v̂)Iteratively solve primal and dual problem, refine marked cells.DFS acts as Adaptive DNS/LESLarge time steps by Schur-preconditioned fixed-pointSegregated fixed-point iteration between u and p, gives straightforward preconditioning of linear systems:CG+bjacobi/ILU for continuity and BiCGStab+bjacobi/ILU (PETSc)Schur-complement preconditioning allows large time-steps, up to 100x larger than simple fixed-point.[Ho↵man, Jansson, et. al., 2012 C&F], [Ho↵man, Jansson, et. al. 2016, ECM], [Jansson, et. al. 2018, HiLift, Springer Brief]CS1: Viz of adaptive resultsGoal quantity: drag M(û)Recall err. rep.: M(ê) = ( R(Û),  ̂)Adjoint  ̂ weights error indicators upstream, zero weight downstream of downstreamsphere, residual R(Û) weights turbulent wakes.See extra material for ref on computability by adjoint stability.Time-evolution of CD0 20 40 60 80 100 120 140 160t0.200.250.300.350.400.450.500.55CDCD sphere1expsim iter 21sim iter 22sim iter 23sim iter 24sim iter 25sim iter 260 20 40 60 80 100 120 140 160t0.200.250.300.350.400.450.500.55CDCD sphere2expsim iter 21sim iter 22sim iter 23sim iter 24sim iter 25sim iter 26Time evolution of the drag coe cient for various iterations of ouradaptive procedure.Convergence orderWe hope to contribute an interesting perspective on convergence order inthe setting of adaptive methods.What is the order of convergence of an adaptive h-refinement method?I Generalized length scale h = N1dDOFI Order of convergence e(h) = Chp or log(e(h)) = p log(Ch)I Compute convergence sequence (ei , hi )I Least-squares fit for p gives “e↵ective order of convergence”.Adaptive DNS/LES ) asymptotic regime is DNSWe are interested in e cient computation ) as coarse meshes aspossible typically in ILES regime which is non-smooth (refinementuncovers finer scales).Uniform refinement quasi-optimal for smooth solutions.Adaptive mesh convergence5.0 5.2 5.4 5.6 5.8 6.0log10(vertices)0.360.370.380.390.400.410.420.43CDMesh convergence of CDCD sphere1CD sphere2CD sphere exp 2.3 2.2 2.1 2.0 1.9log10(h) 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5log10(CDerr)CD err sphere1CD err sphere2O(h2.077895)O(h3.482021)Mesh convergence of the drag coe cients of the two spheres. E↵ective order ofconvergence > 3. However, not interesting with asymptotic behavior. Allowingcoarsening could give di↵erent behavior, (Couchman/Galbraith).We see a clear drag reduction for the downstream sphere, consistent with a“slipstream” phenomenon.2% away from experimental reference for upstream sphere.Adaptive mesh convergence4.0 4.5 5.0 5.5 6.0 6.5log10(Work units)0.360.370.380.390.400.410.420.43CDFEniCS-HPC Tandem Spheres - Mesh convergence of CDCD sphere1CD sphere2CD sphere exp 2.3 2.2 2.1 2.0 1.9log10(h) 5 4 3 2 101log10(CDerr)CD err sphere1CD err sphere2O(h2.077895)O(h3.482021)Mesh convergence of the drag coe cients of the two spheres.CS1: Viz of provided Pointwise-Mesh510% away from experimental reference for upstream sphere, no clear drag reductionfor downstream sphere.CS1: Pointwise mesh convergence4.4 4.6 4.8 5.0 5.2 5.4 5.6 5.8 6.0log10(Work units)0.200.250.300.350.400.450.500.55CDFEniCS-HPC Tandem Spheres - Mesh convergence of CDCD sphere1CD sphere2CD sphere expMesh convergence of the drag coe cients of the two spheres, workshop meshes 3-5.10% away from experimental reference for upstream sphere, no clear drag reductionfor downstream sphere.PyFR - Pointwise Mesh1P3PyFR simulations by: Ezhilmathi Krishnasamy (KTH)Main observation: severe timestep restriction dt ⇡ 10 4 ( 3 orders ofmagnitude smaller than Unicorn/FEniCS-HPC). Implies many time stepsand ⇠ 1 order of magnitude higher cost than Unicorn/FEniCS just forMesh1P3). CPU simulations, possibility for 5x speedup with GPU (ref.conversation with Peter Vincent).CS1: PyFR mesh convergence5.6 5.8 6.0 6.2 6.4 6.6 6.8 7.0log10(Work units)0.200.250.300.350.400.450.500.55CDPyFR Tandem Spheres - Mesh convergence of CDCD sphere1CD sphere2CD sphere expMesh convergence of the drag coe cients of the two spheres, PyFR Mesh1 P1-3.No clear mesh convergence, no clear drag reduction for downstream sphere. ⇠ 1 orderof magnitude higher cost than Unicorn/FEniCS just for Mesh1P3). Work in progressto compute on more meshes/orders.Preliminary Nek5000 simulationNek5000 simulation by: Ricardo Vinuesa, Jing Gong and Philipp Schlatter (KTH)ILES (or under-resolved DNS) regime with 1e7 grid points (4e7 DOFs) and dt ⇡ 10 3Work units: 5e6 Wall clock runtime: 13h CD values in progressMOOC - KTH/edX online course on adaptive FEM andFEniCSHigh Performance Finite Element Modeling MOOC supported by KTH MOOCcommitteeOpened on October 17, 2017, already 4500+ students, join the fun!Advanced part 2 on DFS, turbulence and HPC opens mid-March 2018.SummaryI Adaptive DFS in Unicorn/FEniCS-HPC captures relevant qualitative and quantitativeresults to 2% of exp. Parameter-free, no explicit turb. modeling.I Adaptivity and large timesteps allow high e ciency and accuracy: meshes of ca. 1Mvertices, wall clock ⇠1h using 1k cores.I We wish to give perspective on convergence order in setting of adaptive methods. E↵ectiveorder of convergence > 3 with cG(1) (P1).I Open source pays o↵: Unicorn 2017-2018 ⇠2 orders of magnitude faster than Unicorn2012-2014 (developed in-house for a few years), mostly due to Schur formulation enablinglarge time-steps (ref. HiLiftPW2-3).I Verification and validation of do-nothing duality-based adaptive method for low Re turbulentflow. Automated formulation, no jump terms necessary or manual formulation.Preliminary comparisons to leading open source high-order packages:I Accurate Unicorn/FEniCS-HPC simulation ⇠1 order of magnitude cheaper+faster thanpreliminary Nek5000 simulation, drag comparison work in progress.I Accurate Unicorn/FEniCS-HPC simulation ⇠1 order of magnitude cheaper+faster thanPyFR simulations for finest PyFR simulation, severe timestep restriction for PyFR.Future work:I Can an ensemble-approach to parallel-in-time significantly extend parallel strong scaling?I Extension to compressible flow, initial work for model problems already done.I Investigate r-adaptivity, allows coarsening.Acknowledgements:Extra materialParallel-in-timeEnsemble approach of partitions of time interval for blu↵ body, here ↵ = 4.360 2 4 6 8 10 12t0.1500.1550.1600.1650.1700.175cdTime evolution of cdexpsim PIT 00sim PIT 01sim PIT 020 2 4 6 8 10 12t1.551.601.651.701.751.801.85clTime evolution of clexpsim PIT 00sim PIT 01sim PIT 02[Jansson, Ho↵man, preprint, 2017]DFS computability of turbulent flow by adjoint stabilityWe demonstrate computability of 3D turbulent flow by stability of the 3D adjointproblem. We describe this in the invited chapter “Computability and Adaptivity inCFD” in Encyclopedia of Computational Mechanics.We can bound the global error by:|M(û) M(Û)|  CUhkR(Û)k| ̂|H1 (1)We show that kphR(Û)k and |ph ̂|H1 are bounded, which gives computability.3.5 4.0 4.5 5.0 5.5 6.0log10(vertices)0.880.900.920.940.960.981.001.02| h | H1 (  I)Relative mesh sensitivity of | h |H1(  I)0 50 100 150 200 250 300t2.53.03.54.04.55.05.5| h (t)| H1 ( )Time evolution of | h (t)|H1( )Stability of H1-seminorm of adjoint velocity wrt. mesh refinement and timeMSO4SCH2020 project with BCAM, KTH, Atos, EU-maths-in, etc.Objective: Construct an e-infrastructure that provides simple access tosupercomputing with open source mathematical frameworks such asFEniCS-HPC, develop and manage simulations through a web-interfacewith multiple infrastructures (supercomputers, cloud systems, etc.).Enables simple, automatic and optimal management of ensemble andadaptive simulation on a heterogenous set of computer resources.mso4sc.eu

Adaptive Direct FEM Simulation with Unicorn/FEniCS-HPC for CS1

https://bird.bcamath.org/bitstream/20.500.11824/899/1/CS1_3_KTH_Jansson%20copia.pdf

Adaptive Direct FEM Simulation with Unicorn/FEniCS-HPC for CS1

Abstract

Similar works

Full text

Available Versions

BCAM's Institutional Repository Data