We craft a few scenarios for the execution of sequential and parallel jobs on future generation machines. Checkpointing or migration, which technique to choose

Cappello, Franck

Casanova, Henri

Robert, Yves

arXiv

HAL-ENS-LYON

Checkpointing vs. Migration for Post-PetascaleMachinesFranck Cappello, Henri Casanova, Yves RobertTo cite this version:Franck Cappello, Henri Casanova, Yves Robert. Checkpointing vs. Migration for Post-Petascale Machines. [Research Report] 2009. <inria-00437201>HAL Id: inria-00437201https://hal.inria.fr/inria-00437201Submitted on 30 Nov 2009HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Checkpointing vs. Migration for Post-Petascale MachinesFranck Cappello Henri Casanova Yves RobertNovember 2009Research Report LIP-2009-32EXTENDED ABSTRACTAbstractWe craft a few scenarios for the execution of sequential and parallel jobs on future generationmachines. Checkpointing or migration, which technique to choose?1 IntroductionFrom fault-tolerance to resilience [1, 2]. Large machines are subject to failures. Applications will faceresource faults during execution. Fortunately, failure prediction is there to help. For instance, the systemwill receive an alarm when a disk or CPU becomes unusually hot. In that case, the application mustdynamically do something to prepare for, and recover from, the expected failure. The goal is to comparetwo well-known strategies:• Checkpointing: purely local, but can be very costly• Migration: requires availability of a spare resourceFinally, we assess the cost of periodically checkpoint parallel jobs in the absence of failure prediction.2 Notations• C: checkpoint save time (in minutes)• R: checkpoint recovery time (in minutes)• D: down/reboot time (in minutes)• M : migration time (in minutes)• N : total number of cluster nodes• µ: the mean time between failures (e.g., 1/λ if the failures are exponentially distributed)Obviously, the checkpointing/migration comparison makes sense only if M < C +D + R, otherwisebetter use the faulty machine as its own spare. Techniques such as live migration [3] allow for migratingwithout any disk access, thereby dramatically reducing migration time.13 Sequential jobs3.1 CheckpointingWe checkpoint just in time before the failure. Each resource is unavailable during C+D+R time-steps,and this happens every µ time-steps in average. Hence the global throughput isρcp =µµ+ C +D +R×N3.2 MigrationLet us assume we keep m of the N nodes as spares. We need to ensure that we are never short of aspare machine. We encounter a problem in the execution if there are more than m resources that areengaged in migration or rebooting. The probability that, at a given time, a machine is not migrating orrebooting is:u =µµ+M +D,and that it is migrating or rebooting is:v =M +Dµ+M +D.Therefore, the probability that we do not encounter a problem is:success(m) =m∑k=0(Nk)uN−kvk .So we need to find the good percentage of spare machines, say m = α(ε)N , that “guarantees” asuccessful execution with probability at least 1− ε. Unfortunately, the expression for success(m) doesn’tallow for solving the success(m) ≥ 1− ε equation analytically. It must therefore be solved numerically.Note that(Nk)≥ (N/k)k. Therefore,success(m) ≥m∑k=0(N/k)kuN−kvk ,which may be a bit easier to use for numerically solving the equation, and leads to an overestimation ofthe number of spares for achieving a probability of success 1− ε.Given m spares, the global throughput isρm =µµ+M× (N −m)Remark 1. When there is a problem with migration, it does mean that the execution fails, because wecannot find a spare to replace a machine that goes down, and at that moment, it is too late to checkpoint.4 Parallel jobs4.1 DistributionThe number of processors required by typical job obeys a strange distribution, which is a two-stagelog-uniform distribution biased to powers of two, see [5]. We assume something similar but simpler:• let N = 2Z for simplicity• the probability that a job is sequential is α0 = p1 ≈ 0.25• otherwise, the job is parallel, and the probability that it uses 2j processors is independent of j andequal to αj = (1− p1)× 1Z for 1 ≤ j ≤ Z = log2N2We assume a steady-state utilization of the whole platform, where all processors are active all thetime, and where the proportion of jobs using any given number of processors remains constant. At anytime-step, the expectation of the number of jobs that use 2j processors exactly is βj for 0 ≤ j ≤ Z. Theexpectation of the total number of jobs running is K. We have:K =Z∑j=0βj (1)βj = αjK for 0 ≤ j ≤ Z (2)N =Z∑j=02jβj (3)We deriveNK=Z∑j=02jαj = p1 +1− p1ZZ∑j=12j = p1 +1− p1Z(2N − 2)hence the value of K, and then that of all the βj .4.2 CheckpointingIf a job uses two processors, then the expected interval time between failures is µ/2. This is because theminimum of two identical exponential laws is exponential with a doubled parameter. More generally,let’s call µk the mean of the minimum of 2k i.i.d. variables. If the variables are exponentially distributed,with scale parameter λ, then µk = 1/(λ2k). If the variables are Weibull, with scale parameter λ andshape parameter a, then µk = λΓ(1 + 1/(a2k)).For 0 ≤ k ≤ Z, there are βk×2k processors running jobs with 2k parallel tasks, hence whose expectedinterval time between failures is µk. The throughput is given as:ρcp =Z∑k=0βk × 2k ×µkµk + C +D +R.For the exponential distribution, this becomes:ρcp =Z∑k=0βk × 2k ×1λ1λ+ 2k(C +D +R).4.3 MigrationThe probability of running OK is the same as for independent jobs:success(m) =m∑k=0(Nk)uN−kvk .Because there are only N −m machines ”really” available, we scale the throughput by the factor (N −m)/N). The global throughput now becomesρm =(Z∑k=0βk × 2k ×µµ+ 2kM)× N −mN5 Numerical ResultsIn this section we present numerical results to understand the impact of checkpointing vs. migrationunder a number of scenarios, both in the ”all sequential” case and in the ”parallel jobs” case. All resultsare in percentage improvement of migration over checkpointing (negative or positive values).All results use the following values:3Table 1: ”Today” scenario: C = 25, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Numbers of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 2.75 (65) 2.65 (73) 2081.37 (65) 2080.30 (73)1 day 217 2.96 (386) 2.93 (406) 2760.06 (386) 2759.62 (406)220 3.03 (2732) 3.02 (2786) 3200.37 (2732) 3200.20 (2786)214 0.31 (16) 0.27 (20) 1196.77 (16) 1196.45 (20)1 week 217 0.40 (73) 0.39 (81) 2158.28 (73) 2158.15 (81)220 0.43 (437) 0.42 (458) 2824.48 (437) 2824.42 (458)214 -0.02 (3) -0.04 (5) 136.28 (3) 136.25 (5)1 month 217 0.00 (8) 0.00 (10) 609.60 (8) 609.59 (10)220 0.01 (27) 0.01 (32) 1575.36 (27) 1575.35 (32)214 -0.02 (2) -0.02 (2) 14.81 (2) 14.81 (2)1 year 217 -0.00 (3) -0.00 (4) 97.57 (3) 97.57 (4)220 0.00 (6) -0.00 (9) 471.29 (6) 471.29 (9)• µ = 1 day, 1 week, 1 month, 1 year;• N = 10, 000, 100, 000, 1, 000, 000;• ε = 10−4, 10−6.and with particular values of C = R, M , and D in the following scenarios.5.1 Scenario ”today”• C = R ∈ [20, 30]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (32GB on a 10Gbps net)Results in Table 1 for particular values in the above ranges.5.2 Scenario ”2011 HD”• C = R ∈ [5, 10]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 2 for particular values in the above ranges.5.3 Scenario ”2011 SSD”• C = R ∈ [4, 6]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 3 for particular values in the above ranges.4Table 2: ”2011 HD” Scenario: C = 7.5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 0.34 (65) 0.25 (73) 773.24 (65) 772.81 (73)1 day 217 0.55 (386) 0.52 (406) 995.63 (386) 995.46 (406)220 0.62 (2732) 0.61 (2786) 1131.29 (2732) 1131.23 (2786)214 -0.03 (16) -0.08 (20) 458.73 (16) 458.59 (20)1 week 217 0.05 (73) 0.04 (81) 796.68 (73) 796.63 (81)220 0.08 (437) 0.08 (458) 1012.44 (437) 1012.42 (458)214 -0.03 (3) -0.06 (5) 50.04 (3) 50.02 (5)1 month 217 -0.01 (8) -0.01 (10) 236.64 (8) 236.64 (10)220 0.00 (27) -0.00 (32) 595.00 (27) 595.00 (32)214 -0.02 (2) -0.02 (2) 4.86 (2) 4.86 (2)1 year 217 -0.00 (3) -0.01 (4) 35.06 (3) 35.06 (4)220 -0.00 (6) -0.00 (9) 182.61 (6) 182.61 (9)Table 3: ”2011 SSD” scenario: C = 5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.00 (65) -0.10 (73) 563.73 (65) 563.40 (73)1 day 217 0.21 (386) 0.17 (406) 719.04 (386) 718.91 (406)220 0.27 (2732) 0.26 (2786) 811.69 (2732) 811.64 (2786)214 -0.08 (16) -0.13 (20) 337.65 (16) 337.55 (20)1 week 217 0.00 (73) -0.01 (81) 580.07 (73) 580.03 (81)220 0.03 (437) 0.03 (458) 730.30 (437) 730.28 (458)214 -0.03 (3) -0.06 (5) 35.92 (3) 35.90 (5)1 month 217 -0.01 (8) -0.01 (10) 174.29 (8) 174.28 (10)220 -0.00 (27) -0.00 (32) 436.32 (27) 436.32 (32)214 -0.02 (2) -0.02 (2) 3.40 (2) 3.40 (2)1 year 217 -0.00 (3) -0.01 (4) 25.00 (3) 25.00 (4)220 -0.00 (6) -0.00 (9) 134.17 (6) 134.17 (9)5Table 4: ”2011 Flash” scenario: C = 1.5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.48 (65) -0.58 (73) 245.48 (65) 245.31 (73)1 day 217 -0.28 (386) -0.31 (406) 306.01 (386) 305.95 (406)220 -0.21 (2732) -0.22 (2786) 339.91 (2732) 339.89 (2786)214 -0.15 (16) -0.20 (20) 150.13 (16) 150.07 (20)1 week 217 -0.07 (73) -0.08 (81) 252.08 (73) 252.06 (81)220 -0.04 (437) -0.04 (458) 310.19 (437) 310.18 (458)214 -0.04 (3) -0.06 (5) 14.76 (3) 14.75 (5)1 month 217 -0.01 (8) -0.01 (10) 76.90 (8) 76.90 (10)220 -0.00 (27) -0.00 (32) 192.75 (27) 192.75 (32)214 -0.02 (2) -0.02 (2) 1.33 (2) 1.33 (2)1 year 217 -0.00 (3) -0.01 (4) 10.15 (3) 10.15 (4)220 -0.00 (6) -0.00 (9) 58.63 (6) 58.63 (9)5.4 Scenario ”2011 Flash”• C = R ∈ [1.5, 2]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 4 for particular values in the above ranges.5.5 Scenario ”2011 Flash” + Faster Reboot• C = R ∈ [1.5, 2]• D ∈ [0, 0.5]• M ∈ [.51.5] (64GB on a 20Gbps net)Results in Table 5 for particular values in the above ranges.5.6 Scenario ”2015”• C = R ∈ [0, .15]• D ∈ [0, .5]• M ∈ [.5, 1.5] (128GB on a 40Gbps net)Results in Table 6 for particular values in the above ranges.5.7 Summary• Sequential jobs: forget migration• Parallel jobs: prefer migration, until checkpointing costs dramatically reduce (in proportion ofmigration costs)6Table 5: ”2011 Flash + Faster Reboot” scenario: C = 1.5, D = 0.25, M = 1. Percentage improvementof migration over checkpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.21 (30) -0.27 (35) 131.39 (30) 131.32 (35)1 day 217 -0.08 (155) -0.10 (168) 161.38 (155) 161.35 (168)220 -0.04 (1024) -0.05 (1056) 177.39 (1024) 177.38 (1056)214 -0.09 (9) -0.12 (12) 80.95 (9) 80.92 (12)1 week 217 -0.03 (33) -0.04 (39) 134.46 (33) 134.45 (39)220 -0.01 (174) -0.01 (188) 163.10 (174) 163.10 (188)214 -0.02 (2) -0.04 (3) 7.52 (2) 7.51 (3)1 month 217 -0.01 (5) -0.01 (7) 40.93 (5) 40.93 (7)220 -0.00 (14) -0.00 (17) 103.70 (14) 103.70 (17)214 -0.01 (1) -0.02 (2) 0.67 (1) 0.66 (2)1 year 217 -0.00 (2) -0.00 (3) 5.14 (2) 5.14 (3)220 -0.00 (4) -0.00 (6) 30.95 (4) 30.95 (6)Table 6: ”2015” scenario: C = 0.05, D = 0.25, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.41 (30) -0.47 (35) -47.52 (30) -47.54 (35)1 day 217 -0.28 (155) -0.30 (168) -55.58 (155) -55.58 (168)220 -0.24 (1024) -0.25 (1056) -58.81 (1024) -58.81 (1056)214 -0.12 (9) -0.15 (12) -28.92 (9) -28.94 (12)1 week 217 -0.06 (33) -0.07 (39) -48.25 (33) -48.25 (39)220 -0.04 (174) -0.04 (188) -55.84 (174) -55.84 (188)214 -0.02 (2) -0.04 (3) -2.25 (2) -2.25 (3)1 month 217 -0.01 (5) -0.01 (7) -13.47 (5) -13.48 (7)220 -0.00 (14) -0.00 (17) -37.62 (14) -37.62 (17)214 -0.01 (1) -0.02 (2) -0.20 (1) -0.21 (2)1 year 217 -0.00 (2) -0.00 (3) -1.52 (2) -1.52 (3)220 -0.00 (4) -0.00 (6) -9.91 (4) -9.91 (6)76 Impact of failure predictionIn this section we deal with the case where no failure prediction is available. The idea is to checkpointperiodically. This raises two questions:1. How to determine the optimal period?2. What is the impact on platform throughput?Question 1 has received some attention in the literature for uni-processor jobs. Let T be the period,i.e. the time between two checkpoints, let C be the checkpoint duration time, and µ the expected intervaltime between failures. We compute W , the expected percentage of time lost, or “wasted”, as in [6]:W =CT+T2µ(4)The first term in the right-hand side of Equation 4 is by definition, because there are C time-stepsdevoted to checkpointing every T time-steps. The second term accounts for the loss due to failures andis explained as follows: every µ time-steps, a failure occurs, and we lose an average of T/2 time-steps.Note that because the checkpoint and failure rates are independent, the quantity T/2 does not dependupon the failure distribution (Poisson, Weibull, etc). W is minimized for Topt =√2Cµ. This is Young’sapproximation [7]. The corresponding minimum waste is Wmin =√2Cµ.Equation 4 does not account for recovery time R after each failure. A more accurate expression isthe following:W =CT+T2+R+Dµ(5)Now in the right-hand side we state that every µ time-steps, a failure occurs, and we lose an average ofT2+R+D time-steps. W is minimized for the same value Topt =√2Cµ as before, but the correspondingminimum waste becomesWmin =R +Dµ+√2Cµ(6)Note that this is different from the first-order approximation given by Daly [4, equations (10) and(12)] because we target the steady-state operation of the platform rather than the optimization of theexpected duration of a given job.It turns out that Wmin may become larger than 1 when µ gets very small, a situation which is morelikely to happen with jobs requiring many processors. In that case the application is not progressing anymore. To solve for Wmin ≤ 1 in Equation 6, we let ν = 1√µ and deriveν2(R +D) + ν√2C − 1 ≤ 0We get Wmin ≤ 1 if ν ≤ νb (hence µ ≥ 1/ν2b ) withνb =−√2C +√2C + 4(R+D)2(R+D).In all cases, the minimum waste ismin(Wmin , 1)6.1 Independent jobsWe simply write that the throughput isρ = (1−Wmin)N8Table 7: Yield ρ/N for C = D +R = 1 and p1 = 0.25. Parallel jobs with p1 = 0.25.N Yield (µ = 1 month) Yield (µ = 1 year)28 90.8% 97.5%211 69.9% 92.6%214 13.5% 76.3%217 01.7% 22.1%220 00.2% 02.8%6.2 Parallel jobsWe assume the same distribution of parallel jobs as in Section 4.1, and we keep the same notations K(number of jobs), βk for 1 ≤ k ≤ Z = log2N (number of jobs of size 2k), and µk (expected interval timebetween failures for a job using 2k processors).With 2k processors we use µk instead of µ in Equation 5 to derive the minimum wasteWmin(k). Thethroughput becomesρ =Z∑k=0(1−Wmin(k))2kβk6.3 Numerical ResultsHere is a typical result for parallel jobs:• C = D = R = 1• µ = 1 month or 1 year• p1 = 0.25Results in Table 7 for particular values of N .7 ConclusionNew software/hardware techniques are needed in order to reduce checkpoint, recovery, and migrationtimes. This is a condition for parallel jobs to execute at a satisfying rate on future massively parallelmachines.As for migration, we point out another requirement, namely being able to rely on accurate failurepredictions.Another direction is to design ”self-fault-tolerant” algorithms (e.g. asynchronous iterative algorithms)whose execution can progress in the presence of local faults. Also, replication techniques should beinvestigated: despite the resource costs induced by duplicating the same tasks on different processors,replication can dramatically increase the reliability of the whole application.Most likely, parallel jobs will be deployed on large-scale machines through a mix of all previoustechniques (checkpointing, migration, replication, self-tolerant variants).References[1] F. Cappello. Fault Tolerance in Petascale/ Exascale Systems: Current Knowledge, Challenges andResearch Opportunities. Int. Journal of High Performance Computing Applications, 23(3):212–226,2009.[2] F. Cappello, A. Geist, B. Gropp, L. Kale, B. Kramer, and M. Snir. Toward Exascale Resilience. Int.Journal of High Performance Computing Applications, 23(4):374–388, 2009.9[3] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. LiveMigration of Virtual Machines. In Proc. of the 2nd Symp. on Networked Systems Design and Imple-mentation, pages 273–286. USENIX, April 2005.[4] J. T. Daly. A higher order estimate of the optimum checkpoint interval for restart dumps. FutureGeneration Computer Systems, 22(3):303–312, 2004.[5] U. Lublin and D. Feitelson. The Workload on Parallel Supercomputers: Modeling the Characteristicsof Rigid Jobs. Journal of Parallel and Distributed Computing, 63(11):1105–1122, 2003.[6] J. Wingstrom. Overcoming The Difficulties Created By The Volatile Nature Of Desktop Grids ThroughUnderstanding, Prediction And Redundancy. PhD thesis, University of Hawai‘i at Manoa, 2009.[7] J. W. Young. A first order approximation to the optimum checkpoint interval. Communications ofthe ACM, 17(9):530–531, 1974.10

Checkpointing vs. Migration for Post-Petascale Machines

INRIA a CCSD electronic archive server

HAL Id: inria-00437201https://hal.inria.fr/inria-00437201Submitted on 30 Nov 2009HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Checkpointing vs. Migration for Post-PetascaleMachinesFranck Cappello, Henri Casanova, Yves RobertTo cite this version:Franck Cappello, Henri Casanova, Yves Robert. Checkpointing vs. Migration for Post-PetascaleMachines. [Research Report] 2009. ￿inria-00437201￿Checkpointing vs. Migration for Post-Petascale MachinesFranck Cappello Henri Casanova Yves RobertNovember 2009Research Report LIP-2009-32EXTENDED ABSTRACTAbstractWe craft a few scenarios for the execution of sequential and parallel jobs on future generationmachines. Checkpointing or migration, which technique to choose?1 IntroductionFrom fault-tolerance to resilience [1, 2]. Large machines are subject to failures. Applications will faceresource faults during execution. Fortunately, failure prediction is there to help. For instance, the systemwill receive an alarm when a disk or CPU becomes unusually hot. In that case, the application mustdynamically do something to prepare for, and recover from, the expected failure. The goal is to comparetwo well-known strategies:• Checkpointing: purely local, but can be very costly• Migration: requires availability of a spare resourceFinally, we assess the cost of periodically checkpoint parallel jobs in the absence of failure prediction.2 Notations• C: checkpoint save time (in minutes)• R: checkpoint recovery time (in minutes)• D: down/reboot time (in minutes)• M : migration time (in minutes)• N : total number of cluster nodes• µ: the mean time between failures (e.g., 1/λ if the failures are exponentially distributed)Obviously, the checkpointing/migration comparison makes sense only if M < C + D + R, otherwisebetter use the faulty machine as its own spare. Techniques such as live migration [3] allow for migratingwithout any disk access, thereby dramatically reducing migration time.13 Sequential jobs3.1 CheckpointingWe checkpoint just in time before the failure. Each resource is unavailable during C +D +R time-steps,and this happens every µ time-steps in average. Hence the global throughput isρcp =µµ + C + D + R× N3.2 MigrationLet us assume we keep m of the N nodes as spares. We need to ensure that we are never short of aspare machine. We encounter a problem in the execution if there are more than m resources that areengaged in migration or rebooting. The probability that, at a given time, a machine is not migrating orrebooting is:u =µµ + M + D,and that it is migrating or rebooting is:v =M + Dµ + M + D.Therefore, the probability that we do not encounter a problem is:success(m) =m∑k=0(Nk)uN−kvk .So we need to find the good percentage of spare machines, say m = α(ε)N , that “guarantees” asuccessful execution with probability at least 1− ε. Unfortunately, the expression for success(m) doesn’tallow for solving the success(m) ≥ 1 − ε equation analytically. It must therefore be solved numerically.Note that(Nk)≥ (N/k)k. Therefore,success(m) ≥m∑k=0(N/k)kuN−kvk ,which may be a bit easier to use for numerically solving the equation, and leads to an overestimation ofthe number of spares for achieving a probability of success 1 − ε.Given m spares, the global throughput isρm =µµ + M× (N − m)Remark 1. When there is a problem with migration, it does mean that the execution fails, because wecannot find a spare to replace a machine that goes down, and at that moment, it is too late to checkpoint.4 Parallel jobs4.1 DistributionThe number of processors required by typical job obeys a strange distribution, which is a two-stagelog-uniform distribution biased to powers of two, see [5]. We assume something similar but simpler:• let N = 2Z for simplicity• the probability that a job is sequential is α0 = p1 ≈ 0.25• otherwise, the job is parallel, and the probability that it uses 2j processors is independent of j andequal to αj = (1 − p1) × 1Z for 1 ≤ j ≤ Z = log2 N2We assume a steady-state utilization of the whole platform, where all processors are active all thetime, and where the proportion of jobs using any given number of processors remains constant. At anytime-step, the expectation of the number of jobs that use 2j processors exactly is βj for 0 ≤ j ≤ Z. Theexpectation of the total number of jobs running is K. We have:K =Z∑j=0βj (1)βj = αjK for 0 ≤ j ≤ Z (2)N =Z∑j=02jβj (3)We deriveNK=Z∑j=02jαj = p1 +1 − p1ZZ∑j=12j = p1 +1 − p1Z(2N − 2)hence the value of K, and then that of all the βj .4.2 CheckpointingIf a job uses two processors, then the expected interval time between failures is µ/2. This is because theminimum of two identical exponential laws is exponential with a doubled parameter. More generally,let’s call µk the mean of the minimum of 2k i.i.d. variables. If the variables are exponentially distributed,with scale parameter λ, then µk = 1/(λ2k). If the variables are Weibull, with scale parameter λ andshape parameter a, then µk = λΓ(1 + 1/(a2k)).For 0 ≤ k ≤ Z, there are βk×2k processors running jobs with 2k parallel tasks, hence whose expectedinterval time between failures is µk. The throughput is given as:ρcp =Z∑k=0βk × 2k ×µkµk + C + D + R.For the exponential distribution, this becomes:ρcp =Z∑k=0βk × 2k ×1λ1λ+ 2k(C + D + R).4.3 MigrationThe probability of running OK is the same as for independent jobs:success(m) =m∑k=0(Nk)uN−kvk .Because there are only N − m machines ”really” available, we scale the throughput by the factor (N −m)/N). The global throughput now becomesρm =(Z∑k=0βk × 2k ×µµ + 2kM)×N − mN5 Numerical ResultsIn this section we present numerical results to understand the impact of checkpointing vs. migrationunder a number of scenarios, both in the ”all sequential” case and in the ”parallel jobs” case. All resultsare in percentage improvement of migration over checkpointing (negative or positive values).All results use the following values:3Table 1: ”Today” scenario: C = 25, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Numbers of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 2.75 (65) 2.65 (73) 2081.37 (65) 2080.30 (73)1 day 217 2.96 (386) 2.93 (406) 2760.06 (386) 2759.62 (406)220 3.03 (2732) 3.02 (2786) 3200.37 (2732) 3200.20 (2786)214 0.31 (16) 0.27 (20) 1196.77 (16) 1196.45 (20)1 week 217 0.40 (73) 0.39 (81) 2158.28 (73) 2158.15 (81)220 0.43 (437) 0.42 (458) 2824.48 (437) 2824.42 (458)214 -0.02 (3) -0.04 (5) 136.28 (3) 136.25 (5)1 month 217 0.00 (8) 0.00 (10) 609.60 (8) 609.59 (10)220 0.01 (27) 0.01 (32) 1575.36 (27) 1575.35 (32)214 -0.02 (2) -0.02 (2) 14.81 (2) 14.81 (2)1 year 217 -0.00 (3) -0.00 (4) 97.57 (3) 97.57 (4)220 0.00 (6) -0.00 (9) 471.29 (6) 471.29 (9)• µ = 1 day, 1 week, 1 month, 1 year;• N = 10, 000, 100, 000, 1, 000, 000;• ε = 10−4, 10−6.and with particular values of C = R, M , and D in the following scenarios.5.1 Scenario ”today”• C = R ∈ [20, 30]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (32GB on a 10Gbps net)Results in Table 1 for particular values in the above ranges.5.2 Scenario ”2011 HD”• C = R ∈ [5, 10]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 2 for particular values in the above ranges.5.3 Scenario ”2011 SSD”• C = R ∈ [4, 6]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 3 for particular values in the above ranges.4Table 2: ”2011 HD” Scenario: C = 7.5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 0.34 (65) 0.25 (73) 773.24 (65) 772.81 (73)1 day 217 0.55 (386) 0.52 (406) 995.63 (386) 995.46 (406)220 0.62 (2732) 0.61 (2786) 1131.29 (2732) 1131.23 (2786)214 -0.03 (16) -0.08 (20) 458.73 (16) 458.59 (20)1 week 217 0.05 (73) 0.04 (81) 796.68 (73) 796.63 (81)220 0.08 (437) 0.08 (458) 1012.44 (437) 1012.42 (458)214 -0.03 (3) -0.06 (5) 50.04 (3) 50.02 (5)1 month 217 -0.01 (8) -0.01 (10) 236.64 (8) 236.64 (10)220 0.00 (27) -0.00 (32) 595.00 (27) 595.00 (32)214 -0.02 (2) -0.02 (2) 4.86 (2) 4.86 (2)1 year 217 -0.00 (3) -0.01 (4) 35.06 (3) 35.06 (4)220 -0.00 (6) -0.00 (9) 182.61 (6) 182.61 (9)Table 3: ”2011 SSD” scenario: C = 5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.00 (65) -0.10 (73) 563.73 (65) 563.40 (73)1 day 217 0.21 (386) 0.17 (406) 719.04 (386) 718.91 (406)220 0.27 (2732) 0.26 (2786) 811.69 (2732) 811.64 (2786)214 -0.08 (16) -0.13 (20) 337.65 (16) 337.55 (20)1 week 217 0.00 (73) -0.01 (81) 580.07 (73) 580.03 (81)220 0.03 (437) 0.03 (458) 730.30 (437) 730.28 (458)214 -0.03 (3) -0.06 (5) 35.92 (3) 35.90 (5)1 month 217 -0.01 (8) -0.01 (10) 174.29 (8) 174.28 (10)220 -0.00 (27) -0.00 (32) 436.32 (27) 436.32 (32)214 -0.02 (2) -0.02 (2) 3.40 (2) 3.40 (2)1 year 217 -0.00 (3) -0.01 (4) 25.00 (3) 25.00 (4)220 -0.00 (6) -0.00 (9) 134.17 (6) 134.17 (9)5Table 4: ”2011 Flash” scenario: C = 1.5, D = 2.5, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.48 (65) -0.58 (73) 245.48 (65) 245.31 (73)1 day 217 -0.28 (386) -0.31 (406) 306.01 (386) 305.95 (406)220 -0.21 (2732) -0.22 (2786) 339.91 (2732) 339.89 (2786)214 -0.15 (16) -0.20 (20) 150.13 (16) 150.07 (20)1 week 217 -0.07 (73) -0.08 (81) 252.08 (73) 252.06 (81)220 -0.04 (437) -0.04 (458) 310.19 (437) 310.18 (458)214 -0.04 (3) -0.06 (5) 14.76 (3) 14.75 (5)1 month 217 -0.01 (8) -0.01 (10) 76.90 (8) 76.90 (10)220 -0.00 (27) -0.00 (32) 192.75 (27) 192.75 (32)214 -0.02 (2) -0.02 (2) 1.33 (2) 1.33 (2)1 year 217 -0.00 (3) -0.01 (4) 10.15 (3) 10.15 (4)220 -0.00 (6) -0.00 (9) 58.63 (6) 58.63 (9)5.4 Scenario ”2011 Flash”• C = R ∈ [1.5, 2]• D ∈ [1.5, 5]• M ∈ [.5, 1.5] (64GB on a 20Gbps net)Results in Table 4 for particular values in the above ranges.5.5 Scenario ”2011 Flash” + Faster Reboot• C = R ∈ [1.5, 2]• D ∈ [0, 0.5]• M ∈ [.51.5] (64GB on a 20Gbps net)Results in Table 5 for particular values in the above ranges.5.6 Scenario ”2015”• C = R ∈ [0, .15]• D ∈ [0, .5]• M ∈ [.5, 1.5] (128GB on a 40Gbps net)Results in Table 6 for particular values in the above ranges.5.7 Summary• Sequential jobs: forget migration• Parallel jobs: prefer migration, until checkpointing costs dramatically reduce (in proportion ofmigration costs)6Table 5: ”2011 Flash + Faster Reboot” scenario: C = 1.5, D = 0.25, M = 1. Percentage improvementof migration over checkpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.21 (30) -0.27 (35) 131.39 (30) 131.32 (35)1 day 217 -0.08 (155) -0.10 (168) 161.38 (155) 161.35 (168)220 -0.04 (1024) -0.05 (1056) 177.39 (1024) 177.38 (1056)214 -0.09 (9) -0.12 (12) 80.95 (9) 80.92 (12)1 week 217 -0.03 (33) -0.04 (39) 134.46 (33) 134.45 (39)220 -0.01 (174) -0.01 (188) 163.10 (174) 163.10 (188)214 -0.02 (2) -0.04 (3) 7.52 (2) 7.51 (3)1 month 217 -0.01 (5) -0.01 (7) 40.93 (5) 40.93 (7)220 -0.00 (14) -0.00 (17) 103.70 (14) 103.70 (17)214 -0.01 (1) -0.02 (2) 0.67 (1) 0.66 (2)1 year 217 -0.00 (2) -0.00 (3) 5.14 (2) 5.14 (3)220 -0.00 (4) -0.00 (6) 30.95 (4) 30.95 (6)Table 6: ”2015” scenario: C = 0.05, D = 0.25, M = 1. Percentage improvement of migration overcheckpointing. Number of required spares in parentheses.Sequential Jobs Parallel Jobsµ N ε = 104 ε = 106 ε = 104 ε = 106214 -0.41 (30) -0.47 (35) -47.52 (30) -47.54 (35)1 day 217 -0.28 (155) -0.30 (168) -55.58 (155) -55.58 (168)220 -0.24 (1024) -0.25 (1056) -58.81 (1024) -58.81 (1056)214 -0.12 (9) -0.15 (12) -28.92 (9) -28.94 (12)1 week 217 -0.06 (33) -0.07 (39) -48.25 (33) -48.25 (39)220 -0.04 (174) -0.04 (188) -55.84 (174) -55.84 (188)214 -0.02 (2) -0.04 (3) -2.25 (2) -2.25 (3)1 month 217 -0.01 (5) -0.01 (7) -13.47 (5) -13.48 (7)220 -0.00 (14) -0.00 (17) -37.62 (14) -37.62 (17)214 -0.01 (1) -0.02 (2) -0.20 (1) -0.21 (2)1 year 217 -0.00 (2) -0.00 (3) -1.52 (2) -1.52 (3)220 -0.00 (4) -0.00 (6) -9.91 (4) -9.91 (6)76 Impact of failure predictionIn this section we deal with the case where no failure prediction is available. The idea is to checkpointperiodically. This raises two questions:1. How to determine the optimal period?2. What is the impact on platform throughput?Question 1 has received some attention in the literature for uni-processor jobs. Let T be the period,i.e. the time between two checkpoints, let C be the checkpoint duration time, and µ the expected intervaltime between failures. We compute W , the expected percentage of time lost, or “wasted”, as in [6]:W =CT+T2µ(4)The first term in the right-hand side of Equation 4 is by definition, because there are C time-stepsdevoted to checkpointing every T time-steps. The second term accounts for the loss due to failures andis explained as follows: every µ time-steps, a failure occurs, and we lose an average of T/2 time-steps.Note that because the checkpoint and failure rates are independent, the quantity T/2 does not dependupon the failure distribution (Poisson, Weibull, etc). W is minimized for Topt =√2Cµ. This is Young’sapproximation [7]. The corresponding minimum waste is Wmin =√2Cµ.Equation 4 does not account for recovery time R after each failure. A more accurate expression isthe following:W =CT+T2+ R + Dµ(5)Now in the right-hand side we state that every µ time-steps, a failure occurs, and we lose an average ofT2+R+D time-steps. W is minimized for the same value Topt =√2Cµ as before, but the correspondingminimum waste becomesWmin =R + Dµ+√2Cµ(6)Note that this is different from the first-order approximation given by Daly [4, equations (10) and(12)] because we target the steady-state operation of the platform rather than the optimization of theexpected duration of a given job.It turns out that Wmin may become larger than 1 when µ gets very small, a situation which is morelikely to happen with jobs requiring many processors. In that case the application is not progressing anymore. To solve for Wmin ≤ 1 in Equation 6, we let ν = 1√µ and deriveν2(R + D) + ν√2C − 1 ≤ 0We get Wmin ≤ 1 if ν ≤ νb (hence µ ≥ 1/ν2b ) withνb =−√2C +√2C + 4(R + D)2(R + D).In all cases, the minimum waste ismin(Wmin , 1)6.1 Independent jobsWe simply write that the throughput isρ = (1 − Wmin)N8Table 7: Yield ρ/N for C = D + R = 1 and p1 = 0.25. Parallel jobs with p1 = 0.25.N Yield (µ = 1 month) Yield (µ = 1 year)28 90.8% 97.5%211 69.9% 92.6%214 13.5% 76.3%217 01.7% 22.1%220 00.2% 02.8%6.2 Parallel jobsWe assume the same distribution of parallel jobs as in Section 4.1, and we keep the same notations K(number of jobs), βk for 1 ≤ k ≤ Z = log2 N (number of jobs of size 2k), and µk (expected interval timebetween failures for a job using 2k processors).With 2k processors we use µk instead of µ in Equation 5 to derive the minimum waste Wmin(k). Thethroughput becomesρ =Z∑k=0(1 − Wmin(k))2kβk6.3 Numerical ResultsHere is a typical result for parallel jobs:• C = D = R = 1• µ = 1 month or 1 year• p1 = 0.25Results in Table 7 for particular values of N .7 ConclusionNew software/hardware techniques are needed in order to reduce checkpoint, recovery, and migrationtimes. This is a condition for parallel jobs to execute at a satisfying rate on future massively parallelmachines.As for migration, we point out another requirement, namely being able to rely on accurate failurepredictions.Another direction is to design ”self-fault-tolerant” algorithms (e.g. asynchronous iterative algorithms)whose execution can progress in the presence of local faults. Also, replication techniques should beinvestigated: despite the resource costs induced by duplicating the same tasks on different processors,replication can dramatically increase the reliability of the whole application.Most likely, parallel jobs will be deployed on large-scale machines through a mix of all previoustechniques (checkpointing, migration, replication, self-tolerant variants).References[1] F. Cappello. Fault Tolerance in Petascale/ Exascale Systems: Current Knowledge, Challenges andResearch Opportunities. Int. Journal of High Performance Computing Applications, 23(3):212–226,2009.[2] F. Cappello, A. Geist, B. Gropp, L. Kale, B. Kramer, and M. Snir. Toward Exascale Resilience. Int.Journal of High Performance Computing Applications, 23(4):374–388, 2009.9[3] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. LiveMigration of Virtual Machines. In Proc. of the 2nd Symp. on Networked Systems Design and Imple-mentation, pages 273–286. USENIX, April 2005.[4] J. T. Daly. A higher order estimate of the optimum checkpoint interval for restart dumps. FutureGeneration Computer Systems, 22(3):303–312, 2004.[5] U. Lublin and D. Feitelson. The Workload on Parallel Supercomputers: Modeling the Characteristicsof Rigid Jobs. Journal of Parallel and Distributed Computing, 63(11):1105–1122, 2003.[6] J. Wingstrom. Overcoming The Difficulties Created By The Volatile Nature Of Desktop Grids ThroughUnderstanding, Prediction And Redundancy. PhD thesis, University of Hawai‘i at Manoa, 2009.[7] J. W. Young. A first order approximation to the optimum checkpoint interval. Communications ofthe ACM, 17(9):530–531, 1974.10

Checkpointing vs. Migration for Post-Petascale Machines

Abstract

Similar works

Full text

Available Versions

HAL-ENS-LYON

INRIA a CCSD electronic archive server

Hal-Diderot

HAL-CentraleSupelec