Besseron, Xavier

Gautier, Thierry

Kalé, Laxmikant V.

Zheng, Gengbin

Open Repository and Bibliography - Luxembourg

Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionKaapi / Charm++ preliminary comparisonXavier Besseron1 Thierry Gautier1Gengbin Zheng2 Laxmikant Kalé2(1) MOAIS Project – LIG and INRIA(2) University of Illinois at Urbana-Champaign22-24 June 2010, BordeauxX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 1 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionOutline1 Qualitative comparison2 NUMA execution3 Distributed execution4 Fault-tolerance5 Preliminary ConclusionX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 2 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionObjective: efficiency• 90% of efficiency on 10000 cores ≡ 1000 cores not used• strong scalability (fixed problem size)Comparison of middlewares for exascale architecture• Overhead of parallel programming API• overhead with respect to the execution time of sequential program• Scheduling• quality of scheduling algorithm• Fault tolerance• cost of fault tolerance protocol (coordinated checkpoint/rollback)Cost of developping parallel application• Parallel programming complexity• Add fault tolerance to application• Other criteria (smallest timestep, memory) for futur comparisonsX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 3 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionOverviewCharm++• Library/Compiler (C, C++, Fortran, ...)• Low level interface: Object based (Chares), message driven• High level interface: Structured dagger, Charisma, ...• Production toolKaapi• C++ library, template based, Fortran interface• High level interface: Data flow graph (Athapascan)• At runtime based on active messages + threads + schedulingalgorithms ...• Research softwareX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 4 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionWhich comparison ?Key points• Programming model• Preliminary work: skew due to Charm++/Low level - Kaapi/High level• Futur work: Charisma versus Kaapi• Performance of execution• Overhead with respect to execution time of a sequential program• Multi-core• Cluster and grid• Fault Tolerance capability• Checkpoint / Restart protocols⇒ experimental costsMethodology• "Same benchmark" using both frameworks + sequential version• Experimentation and AnalysisX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 5 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionProgramming models comparisonCharm++ (Low level interface: object based, message driven)• Simple: Model and syntax close to the classic object model++ Good documentation, many examples, stable⇒ Easy to understand, easy to use++ Several "production quality" applications⇒ large scale experiments, including the FT protocol++ Automatic dynamic load balancing⇒ Required for irregular computation+/– Non deterministic execution⇒ Results can depend on the reception order– Explicit communications, required some synchronizations⇒ Source of bugs (race conditions)– Non-transparent fault-tolerance⇒ Requires source code modificationX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 6 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionProgramming models comparisonKaapi (High level interface: macro data flow)• Complex/unusual programming model: macro data flow++ Sequential semantic / Implicit communications andsynchronizations⇒ Once the (sequential) program is written and run on 1 (or 2)processors, very few bugs (in the application source code)++ Automatic dynamic load balancing++ Transparent fault-tolerance⇒ No modification in the source code+/– Deterministic execution⇒ Always the same results– Simple documentation, research software⇒ Harder to understand and to start with ?– Not a language: C++ Template based⇒ Error messages at compilation are very uglyX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 6 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionScheduling of Charm++ program• A Charm++ program⇒ a set of Chares• Scheduler of message queue (priority, ...)• Dynamic Load balancing algorithm• GreedyLB, MetisLB, RecBisectBfLB, NeighborLB, ...• Based on automatic instrumentation of the application by the runtimeX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 7 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionScheduling of Kaapi program• A Kaapi program⇒ a data flow graphKAAPI Schedulerwork stealing + affinity K-ThreadK-ProcessorprocessprocessCPUCPUCPUActive Message over TCP/IP, Myrinet CPU CPUOS schedulerOS schedulerData Flow Graph abstract representationGraphPartitionningX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 8 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionThe benchmark: Poisson3DProblem• Solve Poisson equation on a 3D domain: A× X = B• Parallelized by domain decomposition• Jacobi iterative methodxk+1i =1aii(bi −N∑j=1,j 6=iaij xkj ) for i from 1 to NBenchmark data• X, X’: computation domain & temporary computation domain• A: Jacobi stencil (sparse matrix)• B: right-hand side• S: solution (here the solution is known to check the result)X. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 9 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionSource code• Common code for "sequential" kernel: 671 lines• Same data structure and allocationLines of code (raw values)Charm++ Kaapi(De)serialization 105 125Task encapsulation - 160Interface file (.ci) 50 -Computation Kernel 140 50FT/LB 50 -Total 483 481• Sequential program without decomposition: 50 lines• Sequential program with decomposition: 112 linesX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 10 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionExperiments• NUMA machine "idkoiff"• 8 Opteron 875 dual core, 2.2Ghz, 30GBytes. 8 memory banks• gcc/g++ 4.5. All programs compiled with -O4.• Charm++ 6.2.0 (multicore-linux64.tar.gz)• +p 1: one process• +ppn <n> : number of PEs per process• +setcpuaffinity : useCPU affinity• +excludecore <coreid> : to exclude some core• Kaapi 2.4 (http://kaapi.gforge.inria.fr)• Same mapping of threads onto core for Charm++/Kaapi• using util.thread.cpuset• XKaapi 0.1beta• New C kernel for Kaapi. Currently only for multi-core machine.Timings• Domain size: 4× 106 doubles (100x200x200) split in i × j × ksubdomains• 1 run = average time of one iteration over 50 iterations.• Get the best average over 10 runsX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 11 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionExecution time: 1 core (among 16 cores) 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.161x1x1 1x2x2 2x2x2 1x4x4 2x4x4 4x4x4 4x8x8One−iteration time (s)DecompositionCharm++ SMPKaapiX−KaapiSeq with decompositionSequentialX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 12 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionExecution time: 8 cores (among 16 cores) 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.451x1x1 1x2x2 2x2x2 1x4x4 2x4x4 4x4x4 4x8x8One−iteration time (s)DecompositionCharm++ SMPKaapiX−KaapiSequential• Few parallelism⇒ slowdown due to activity of the threadsX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 13 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionExecution time: 16 cores (among 16 cores) 0 0.1 0.2 0.3 0.4 0.5 0.61x1x1 1x2x2 2x2x2 1x4x4 2x4x4 4x4x4 4x8x8One−iteration time (s)DecompositionCharm++ SMPKaapiX−KaapiSequential• Same slowdownX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 14 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionDistributed execution: experimentsExecution platform• Two experiments on Grid’5000 clusters:• Cluster Griffon at Nancy : 64 nodes with 8 cores, ie 512 cores• Cluster GDX at Orsay : 283 nodes with 2 cores, ie 566 cores• Gigabit ethernet networkSame Poisson 3D benchmark• Weak scalability: domain size = 4× 106 doubles per core• Two cases for Kaapi and Charm++:• 1 process per node (named SMP)• 1 process per core• Charm++ 6.1.3 net-linux-x86_64[-smp]• Kaapi (git version 03/01/2010, branch besseron/master)X. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 15 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionDistributed execution: cluster GriffonWeak scalability: 4× 106 doubles per core, 4 subdomains per coreUp to 512 cores (64 nodes of cluster Griffon of Nancy) 0 0.1 0.2 0.3 0.4 0.5 0  100  200  300  400  500  600One−iteration execution time (s)Number of coresKaapiKaapi SMPCharmCharm SMPSequentialX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 16 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionDistributed execution: cluster GDXWeak scalability: 4× 106 doubles per core, 4 subdomains per coreUp to 566 cores (283 nodes of cluster GDX of Orsay) 0 0.1 0.2 0.3 0.4 0.5 0  100  200  300  400  500  600One−iteration execution time (s)Number of coresKaapiKaapi SMPCharmCharm SMPSequentialX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 16 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionCheckpoint protocols in Charm++ and KaapiCharm++Double blocking coordinated checkpoint on disk• double disk ≡ 2 copies: local and on a remote buddy processor• ⇒ tolerate only 1 failure in worst case• global consistency requires an explicit barrier in the codeKaapiBlocking coordinated checkpoint on disk• 1 copy per process on a Kaapi checkpoint server• ⇒ assume that checkpoint servers are stable,• global consistency is ensured by the checkpoint protocolX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 17 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionExperimental conditions3D Poisson Benchmark• Cluster Griffon at Nancy : 80 nodes with 8 cores, ie 640 cores• Domain size = 106 doubles per core• Charm++ net-linux-x86_64-syncft (git version on 04/12/2010)with three different load-balancers: NullLB, GreedyLB, OrbLB• Kaapi with FT support (git version on 04/12/2010, branch besseron/master)Remote buddy processors / Kaapi checkpoint servers• Located on the next node (according to the node list)Charm++ PEs Remote buddy processorKaapi processors Checkpoint servergriffon-01 −→ griffon-02......griffon-79 −→ griffon-80griffon-80 −→ griffon-01X. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 18 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionCheckpoint time80 nodes with 8 cores, ie 640 coresDomain size = 106 doubles per coreKaapi202 MB/nodeCharm++ NullLB455 MB/nodeKaapi202 MB/nodeCharm++ NullLB455 MB/nodeCharm++ GreedyLB455 MB/node0510152025 5,0 s  5,0 s11,0 s27,1 s 3,9 s14,6 sTotal = 16,0 sTotal = 27,1 sTotal =  5,4 sTotal =  3,9 sTotal = 14,6 sCheckpoint time (s)Send dataWait for acknowledgmentCheckpoint to disk Checkpoint to memoryX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 19 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionRestart protocols in Charm++ and KaapiScenario• Domain size = 106 doubles per core• 1 failure⇒ restart on 79 nodes = 632 cores⇒ load balancingRestart protocols• Charm++ : Global restart + load balancing• Failed processes restart from their remote checkpoint• Alive processes restart from their local copy• Kaapi : Global restart + rescheduling• All processes restart from their remote checkpoint• Rescheduling: static partitionning + data redistributionX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 20 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionRestart and load balancing time1 failure⇒ restart on 79 nodes with 8 cores, ie 632 coresKaapi Charm++ OrbLB Charm++ GreedyLB Charm++ NullLB024682,1 s 2,2 s 2,1 s1,0 s4,6 s6,2 s1,7 s3,3 sTotal = 7,0 s Total = 6,8 sTotal = 8,4 sTotal = 1,0 sTime (s)Global restartLoad balancingStatic partitionningData redistributionX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 21 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionConclusion of this preliminary work (1/2)Two different levels• Charm++: low level API• Charisma will be the next candidate• Kaapi: high level API• Also recursive applications + work-stealing• Heterogeneous architecture (CPUs + GPUs)Execution performance• NUMA: new XKaapi implementation performs well• comparison with NUMA features [Pousa, Méhaut (INRIA), Gioachin,Mei, Kalé (UIUC)] for Charm++• Distributed execution: SMP Charm++ execution is betterX. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 22 / 23Qualitative comparison NUMA execution Distributed execution Fault-tolerance Preliminary ConclusionConclusion of this preliminary work (2/2)Fault tolerance• Checkpoint is the most costly part of the protocols• Charm++ tolerates only 1 failure in worst case without stable servers• Kaapi tolerates the failure of one cluster but required "stable servers"⇒ Stable memory issues• diskless checkpointing: k-duplication, erasure encoding, locality• Restart• Global restart is simple, but lost work can be large⇒ Partial restart in Kaapi (based on the data flow graph)X. Besseron, T. Gautier, G. Zheng, L. Kalé Kaapi / Charm++ preliminary comparison 23 / 23

Kaapi / Charm++ preliminary comparison

http://orbilu.uni.lu/bitstream/10993/39972/1/talk_2010_jlpc.pdf

Kaapi / Charm++ preliminary comparison

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg