Basermann, Achim

Hager, Georg

Kreutzer, Moritz

Röhrig-Zöllner, Melven

Thies, Jonas

Wellein, Gerhard

Institute of Transport Research:Publications

www.DLR.de • Chart 1 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Performance Engineering for Sparse Eigensolvers onHeterogenous ClustersJonas Thies Melven Ro¨hrig-Zo¨llner Moritz KreutzerAchim Basermann Georg Hager Gerhard WelleinGerman Aerospace CenterSimulation and Software Technologyand University of Erlangenproject ESSEXwww.DLR.de • Chart 2 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MotivationMathematical problem• Find 20− 50 eigenpairsAxi = λixiof a large, sparse matrix A• interior or extreme λi• symmetric or general AMemory gap• small memory bandwidth vs.high peak flop rate→ increase the compute intensityRoofline performance model(2x 12 core Haswell EP)14166425610241/4 1 4 16 64PeakbandwidthPeak Flop/sBLAS1 (ddot)DPGFlop/sCompute intensity [Flop / DP element]www.DLR.de • Chart 3 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Block JDQR MethodBlock Jacobi-Davidson correction equation• nb current approximations: Av˜i − λ˜i v˜i = ri , i = 1, . . . , nb• previously converged Schur vectors(q1, . . . , qk)= Q• solve approximately (with Q˜ =(Q v˜1 . . . v˜nb)):(I − Q˜Q˜T )(A− λ˜i I )(I − Q˜Q˜T )xi = −ri i = 1, . . . , nb• use some steps of a block(ed) iterative solver• orthogonalize new directions x1, . . . , xnb (outer subspace iteration)Properties (compared to single-vector method)• usually needs more operations → shunned in practice• more cache-friendly, fewer global operationswww.DLR.de • Chart 4 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Numerical BehaviorBlock size 211.522.533.55 10 20 30 40 50relativeoverhead(#spMVM)# eigenvalues foundAndrewscfd1finan512torsion1ck656cry10000dw8192rdb3200lBlock size 411.522.533.55 10 20 30 40 50relativeoverhead(#spMVM)# eigenvalues foundfrom: Ro¨hrig-Zo¨llner et al. SISC 2015www.DLR.de • Chart 5 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Software I:(General Hybrid and Optimized Sparse Toolkit) provides• intelligent resource management for heterogenous systems• automatic pinning of threads to cores• asynchronous execution of (larger) tasks• some fully optimized kernels for sparse matrix methods• sparse matrix-(multi)vector multiplication (spM(M)VM)• ‘tall and skinny’ matrices in row or column major ordering• target platforms right now: Intel CPUs, Xeon Phi and Nvidia GPUs• programming model: ‘MPI+X’,with X=SIMD intrinsics, OpenMP and CUDAwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 6 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12MPI+X with• System with multiple CPUs(NUMA domains) and GPUs• -np 1: use entire CPU• -np 2: use CPU and first GPU• -np 3: use CPU and both GPUs• -np 4: use one process per socketand one for each GPUOption: distribute problem accordingto memory bandwidth measuredwww.DLR.de • Chart 7 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12What is NOT• a DSL for programming heterogenous hardware• easily portable to platforms other than Intel and Nvidia• easy to integrate in existing code• a mature library=⇒ For implementing iterative solvers we use an interface layer (up next)www.DLR.de • Chart 8 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Software II: PHISTa Pipelined Hybrid-parallel Iterative Solver Toolkit• facilitate algorithm development using• holistic performance engineering• portability and interoperabilityapplication vertical integrationalgorithmspreconditionerscomputational core«abstraction»eigenproblemsetup/applysparseMat mVec sdMatsolver templatesFT strategiesalgo core«interface»kernel interface holisticperformanceengineeringC wrapperadapterwww.DLR.de • Chart 9 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Useful abstraction: kernel interfaceChoose from several ‘backends’ at compile time, to• easily use PHIST in existing applications• perform the same run with different kernel libraries• compare numerical accuracy and performance• exploit unique features of a kernel library (e.g. preconditioners)www.DLR.de • Chart 10 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Cool features of PHISTTask macrosout-of-order execution of code blocks• overlap comm. and comp.• asynchronous checkpointing• ...Consistent random vectorsmake PHIST runs comparable• across platforms (CPU, GPU...)• across kernel libraries• independent of #procs, #threadsPerfCheck:print achieved roofline performance ofkernels after complete run to reveal• deficiencies of kernel lib• implemntation issues of algorithm(strided data access etc.)Special-purpose operations• fused kernels, e.g. computeY = αAX + βY and Y TX• highly accurate core functions, e.g.block orthogonalization in simulatedquad precisionwww.DLR.de • Chart 11 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Sparse matrix-vector multiplication (in a Chebyshev solver)1 64 256 10244 16Number of heterogeneous nodes0.1110100Performance in Tflop/s100% Parallel EfficiencySquare, Weak ScalingBar, Weak ScalingSquare, Strong ScalingWeak and strong scaling(on Piz Daint @ CSCS Lugano)from: Kreutzer et al. IPDPS’15SELL-C-σ sparse matrix storage format for heterogenous systemswww.DLR.de • Chart 12 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12‘Tall & skinny’ kernel performance (V ∈ R10M×40,W ∈ R10M×4)020406080100120SVQB project shrink%ofrooflineperformance10 core Intel(R) IvyBridgeNvidia(R) Tesla K20mC4×4 =W TWW =W ·C4×4 C40×4 =V TWW =W −V ·C40×4V:,1:20 =V ·C40×20⇒ some fallback kernels needed on GPU, further experiments postponedwww.DLR.de • Chart 13 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Strong scaling performanceSetup• non-symmetric matrix from7-point 3D PDE discretization(n ≈ 1.3 · 108, nnz ≈ 9.4 · 108)• find 20 eigenvalues• Ivy Bridge ClusterResults• nb = 2: significantly faster• nb = 4: no further improvement 0100020003000400050008 16 32 64 128nodesnb=1nb=2nb=4www.DLR.de • Chart 14 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Block method faster for various matricesSetup• different large matrices from• Quantum physics• PDE discretization• find 20 outmost eigenvaluesusing (block) Jacobi-Davidson• block size nb = 2 (similar for 4)Results• typically faster by a factor 1.2• less synchronization but largermessages during spMMVM00.20.40.60.811.21.48 16 32 64 128speedupthroughblockingnodesSpinSZ[28]tV[28]Hubbard[16]BosHub[22]MATPDE[16k]MATPDE3D[512]www.DLR.de • Chart 15 > SIAM PP’16 > J. Thies et al. • Performance Engineering > 2016-04-12Further informationand PHIST are developed within the DFG (SPPEXA) fundedproject ESSEX (Equipping Sparse Solvers for the EXa-scale).• project website incl. list of publications:http://blogs.fau.de/essex/• source code: https://bitbucket.org/essex/[ghost|phist]We are happy to collaborate onbuilding blocks, algorithms and applicationsand support ‘friendly users’ !Contact: Jonas.Thies@DLR.de

Performance Engineering for Sparse Eigensolers on Heterogenous Clusters

https://elib.dlr.de/108964/1/presPP16.pdf

Performance Engineering for Sparse Eigensolers on Heterogenous Clusters

Abstract

Similar works

Full text

Available Versions

Institute of Transport Research:Publications