Energy minimization of parallel applications considering thermal distributions among the processor cores is an emerging challenge for current and future generations of many-core computing systems. This paper proposes an adaptive energy minimization approach that hierarchically applies dynamic voltage\slash frequency scaling (DVFS), thread-to-core affinity and dynamic concurrency controls (DCT) to address this challenge. The aim is to minimize the energy consumption and achieve balanced thermal distributions among cores, thereby improving the lifetime reliability of the system, while meeting a specified power budget requirement. Fundamental to this approach is an iterative learning-based control algorithm that adapts the VFS and core allocations dynamically based on the CPU workloads and thermal distributions of the processor cores, guided by the CPU performance counters at regular intervals. The adaptation is facilitated through modified OpenMP library-based power budget annotations. The proposed approach is extensively validated on an Intel Xeon E5-2630 platform with up to 12 CPUs running NAS parallel benchmark applications

Al-Hashimi, Bashir

Das, Anup K.

Merrett, Geoff V.

Shafik, Rishad Ahmed

Yang, Sheng

Southampton (e-Prints Soton)

Thermal-aware Adaptive Energy Minimizationof OpenMP Parallel ApplicationsRishad A. Shafik, Anup K. Das, Sheng Yang, Geoff V. Merrett & Bashir M. Al-HashimiSchool of ECS, University of Southampton, SO17 1BJ, UK, e-mail: {ras1n09,akd1g13,gvm,sy2u12,bmah}@ecs.soton.ac.ukAbstract—This paper proposes an adaptive energy minimizationapproach that hierarchically applies DVFS, thread-to-core affinityand dynamic concurrency controls (DCT) to minimize the energyconsumption and improve lifetime reliability through balanced ther-mal controls, while meeting a specified power budget requirement.Fundamental to this approach is an iterative learning-based controlalgorithm that adapts the VFS and core allocations dynamically basedon the CPU workloads and thermal distributions, guided by theCPU performance counters at regular intervals. The adaptation isfacilitated through modified OpenMP library-based power budgetannotations. The proposed approach is extensively validated on anIntel Xeon E5-2630 platform with up to 12 CPUs running NASparallel benchmark applications.Index Terms—Lifetime reliability, voltage/frequency scaling.I. INTRODUCTIONModern processors typically exhibit increased power density,operating temperatures and their variations. Such high power densityand temperatures accelerate the device wearout mechanisms throughelectromigration, dielectric breakdown, etc. Hence, thermal-awareenergy minimization is a key design challenge [1].Parallel programming model is a key contributor to the continuingperformance growth of current and future generations of many-coreapplications. OpenMP is one such programming model, consideredas the de facto standard of shared memory multiprocessing [5]. Toachieve OpenMP-based dynamic adaptation between energy and per-formance trade-offs, dynamic voltage/frequency scaling (DVFS) isa major runtime control knob [14]. Dynamic concurrency throttling(DCT) is another effective software knob, which selects the numberof concurrent threads during runtime to trade-off performance forenergy consumption. Over the years, various approaches usingDVFS and DCT separately and also synergestically have beenshown; examples include [1], [3], [9] etc.Existing OpenMP-based energy minimization approaches forparallel applications have the following limitations. Firstly, existingapproaches [3], [9], [12] employ DVFS and/or DCT using offlinetraining to learn the system architecture and control parameters,thus demonstrating poor scalability. Secondly, these approaches [9],[12] do not consider control of thermal hotspots and cycling causedby uneven workloads and operating frequencies of the processingcores. Other approaches [11] use POSIX thread-based runtimeimplementations that require substantial application re-design andare not feasible for OpenMP parallel applications. This paperaddresses the above limitations through a novel thermal-awareadaptive energy minimization approach. Section II further detailsthe approach, Section III validates the effectiveness of the approachand finally, Section IV concludes the paper.II. PROPOSED ENERGY MINIMIZATION APPROACHFig. 1 shows the proposed adaptive energy minimization approachorganized in three steps, highlighting the interactions betweenapplication, runtime and hardware. In the first step, power budgetannotation is incorporated in the application codes. This annotation,defined within and compiled by the modified OpenMP library(libgomp), communicate the overall power budget requirement tothe runtime. The runtime, which consists of the OpenMP library andOS routines, uses this power budget to guide the online DCT andpower budgeting control step at regular time intervals (we denotethis as ∆TDCT ). This is then followed by the DVFS control stepat smaller regular intervals (we denote this as ∆TDV FS)), guidedby the monitored performance counters. The proposed energyminimization steps are further detailed in the following.Application #pragma omp budget(cur_budget)RuntimeHardwareMonitor Perf. counterControls(DCT+DVFS)Modified OpenMP runtime (libgomp) libraryStep 2Step 3Online thread profilingApply DCT/Affinity for concurrent threads/coresApply allocation algorithm for thread budget allocationStep 1Predict workloadApply ILC for DVFS control for given power budgetMonitor workload, power and adjust errorsInterconnectCore 0 Core 1 Core 2 Core nDCT control interval (ΔT DCT)DVFS control interval (ΔT DVFS = ΔT DCT / N)Fig. 1. Proposed energy minimization approachA. Step 1: Power Budget AnnotationThe power budget annotation in the application codes iscarried out to enable energy minimization and control ap-plication performance (Fig. 1). This is done through anno-tating the beginning of the master thread code (the mainfunction, not enclosed by any other OpenMP annotations) by#pragma omp budget(cur_budget). The budget is in-troduced to pass the current application power budget (Pbudget) ofcur_budget (in mW) to the OpenMP runtime library.B. Step 2: Online DCT and Thread Budgeting ControlWith the specified application power budget, DCT and threadbudget controls are applied in the following three phases:1) Online Thread Profiling: To enable online thread workloadprofiling, an additional timer-based thread is introduced in theOpenMP runtime library with a period of ∆TDCT . At each ∆TDCT ,the affinity of each computing thread (i.e. the core that is executingthe thread) is obtained with a reference to the given processorto prevent any preemption and movement to another processorusing get_cpu(). At this time, three thread and core metrics areevaluated through the performance counters using LIKWID [10]:(a) per thread average workload (Wd) asWd =1N∆TDCT∑t=0Wdt , d = 0 : D (1)where Wdt is the thread workload at each ∆TDV FS interval.(b) per thread average power consumption (Pd) asPd =1N∆TDCT∑t=0Pdt , (2)where Pdt is the observed power consumption at each ∆TDV FSinterval due to chosen DVFS (see Section II-B3).(c) per core temperature standard deviation (std(Tempc)) asstd(Tempc) =√√√√ 1N∆TDCT∑t=0(Tempct − Tempc)2. (3)Upon evaluation of these metrics, the timer thread returns theprocessor reference back through put_cpu() and relinquishesthe CPU and moves at the back of the thread queue, which allowsfor other CPUs to execute. The timer thread is statically assignedthe same affinity as the main master thread.2) DCT and Thread Affinity Control: With the evaluated metrics,the DCT control is applied with an aim of improving the systemperformance, while meeting the given application power budget.Moreover, the thread affinity is also managed with an aim ofreducing the thermal cycles caused by uneven workloads and theirDVFS controls.Algorithm 1 shows the DCT and affinity control algorithmused to carry out such optimizations. As can be seen, with theAlgorithm 1 Iterative DCT and Affinity control algorithmRequire: Wd, Pd, Tempc and std(Tempc)1: Calculate total power consumption, P =∑Dd=1 Pd2: if P less than Pbudget then3: Increase number of parallel threads: D=D+1; D ≤ C4: else if P more than Pbudget then5: Decrease number of parallel threads: D=D-1; D ≥ 26: end if7: for each core: c=1 to C do8: Sort the cores in ascending standard deviation9: end for10: for each core: c′=1 to C do11: Swap thread affinity between cores c′ and C − c′12: end forobserved average power consumption per thread Pd, the totalpower consumption (P ) is initially found out (line 1). If P isless than Pbudget, the number of parallel threads (D) is increasedto improve system performance within the power budget (lines2-3); otherwise, if P is higher than Pbudget, D is decreased toreduce power consumption (lines 4-5). Decreasing the numberof parallel threads reduces number of parallel computing coresand hence cuts down the core leakage power consumptions at theexpense of reduced application speedup. The temperature standarddeviations are then sorted in ascending order (lines 8-10). Withinthe sorted list, the thread affinity of the first core is swapped withthe last core, the thread affinity of the second core is swappedwith the second last and so on (lines 10-12). Such affinity controlhas the advantage of more even thermal distribution among thecores. Since the DVFS control and memory bandwidth is affected,it, however, can affect the parallel application performance withincreased thread synchronization times, which is controlled in thelower level hierarchy through DVFS controls at smaller intervals(∆TDV FS = 1N ∆TDV FS).3) Thread Power Budget Allocation: With the given DCT andaffinity controls, the thread power budgeting is reviewed andupdated at each ∆TDCT interval. For the new threads joiningdue to DCT controls in Algorithm 1, the newly allocated powerbudget for d′-th thread (Pd′budget ) is initially assigned asPd′budget =1DPbudget. (4)However, for the existing threads the power budget is updated basedon the online profiled workloads in Section II-B1. The updatedpower budget of the d-th thread (Pdbudget ) is given asPdbudget =Wd(Pbudget −∑d′ Pd′budget)∑D′d=1 Wd, (5)where D′ is the number of profiled threads (D′ <= D) andPd′budget is the power budget of the newly joined thread.C. Step 3: DVFS ControlWith the allocated per thread power budgets, the energy min-imization is carried out through DVFS based on the predictedworkloads at regular intervals (Fig. 1). To effectively predict thetime-varying workload at each interval, exponentially weightedmoving average (EWMA) is used as the prediction scheme, similarto [13]. Using this scheme, the predicted workload at the tthinterval, Wˆdt (in CPU cycles), is given by [13] asWˆdt = ωWdt−1 +E∑i=1(1− ω)iWdt−i , (6)where Wdt and Wi are the previous observed workloads (in CPUcycles) at the tth and ith decision epochs, 1 ≤ i ≤ E, ω is themoving average coefficient and E is the window size (ω and Eare evaluated empirically for higher prediction accuracy). Basedon the predicted workload Wˆdt in (6) the operating frequency isdetermined by the iterative learning control (ILC) function asfdt = fdt−1 −∆fkdtK1; if Edt−1 ≈ 0 (7)fdt = fdt−1 −∆fK1Edt−1 , otherwise (8)where fdt−1 is the previous operating frequency, ∆f is thefrequency differential, K1 is ILC constant defining frequencyscaling steps, Edt−1 is the power budget error incurred due toprevious control actions and kdt is given bykdt = exp [K2(Wˆdtfmax ×∆t − 1)], (9)where K2 is a constant, ∆t is the time interval and fmax is themaximum processor core frequency in the system. The powerbudget error (Edt−1 , in %) in (7) is evaluated asEdt−1 =1Pdbudgett−1∑i=1(Pdi − Pdbudget), (10)where Pdi is the observed thread power consumption at the i-thinterval. The ∆f and fmax values can be obtained from the OS(in the case of Linux, from sysfs variable).Energy minimization through ILC is implemented as an addi-tional timer-based thread in the OpenMP runtime library with aperiod of ∆t. This thread is statically assigned the same affinityas the main master thread (i.e. thread 0).III. EXPERIMENTAL RESULTSTo validate the effectiveness of the proposed approach, anumber of experiments are carried out on an Intel Xeon E5-2630 [7] platform, which has a total of 12 cores, organizedin two sockets with 6 cores each. Each core operates at aminimum frequency of 1.2 GHz (at Vdd = 0.98V ) and a maximumfrequency of 2.6 GHz (at Vdd = 1.35V ); there are also thirteenother intermediate frequencies increasing in 0.1 GHz steps. NASapplication benchmarks of class B (medium) and C (large) areexecuted on Linux kernel version 2.6.32. The average performanceand energy consumption are recorded over 100 executions of theapplications using the LIKWID [10]. The lifetime reliability inmean time to failure (MTTF) is estimated using the model in [8].Fig. 2.(a) and (b) show the the execution times and energyconsumption of the applications for a given power budget of 30Wfor four different approaches: the proposed approach, Linux’s050100150200250300350400Execution times (seconds)(a)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.C  ProposedLinux OndemandMatthew et al.Cochran et al.02000400060008000100001200014000Energy (Joules)(b)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.C  ProposedLinux OndemandMatthew et al.Cochran et al.012345678910Estimated MTTF (years)(c)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.C  ProposedLinux OndemandMatthew et al.Cochran et al.Fig. 2. Comparative (a) performance (seconds), (b) energy consumption (Joules), and (c) lifetime reliability (MTTF in years)50556065707580859095100Temperature, deg C(a)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.CProposed50556065707580859095100Temperature, deg C(b)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.COndemand50556065707580859095100Temperature, deg C(c)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.CMatthew et al.50556065707580859095100Temperature, deg C(d)bt.Bbt.Ccg.Bcg.Cdc.Bep.Bep.Cft.Bft.Cis.Bis.Clu.Blu.Cmg.Bmg.Csp.Bsp.Cua.Bua.CCochran et al.Fig. 3. Comparative average temperatures and their deviationsondemand governor [14], and the approaches proposed in [9]and [1]. From the figures two observations can be made. Firstly,depending on the nature of the computation of the application,the execution times and energy consumptions vary dynamically.The applications bt (class C) and sp (class C) exhibit the highestexecution times and corresponding energy consumptions, while mgand is show the lowest execution times and energy consumptions.Secondly, due to power budget constrained energy minimization theproposed approach outperforms the other approaches in terms ofthe energy consumptions (by up to 15%). The ondemand governorand the approach proposed in [9] consisently outperforms andconsumes more energy than the proposed approach due to powerbudget unaware energy minimizations. Although the approachproposed in [1] is power budget aware but it depends largely onoffline training driven look up table for online adaptation, whichcan scale poorly due to variable thread synchronization times [12].The proposed approach achieves energy minimization by up to 6%due to online DCT and power budget adaptation (Section II).Fig. 2.(c) shows the lifetime reliability comparisons of theapproaches. As can be seen, the proposed approach offers betterlifetime reliabilities based on the average, peak and thermal cyclebased model proposed in [8]. The lifetime reliability improvementscan be explained by the average temperatures and their deviationsshown in Fig. 3. As can be seen, the proposed approach offersbetter control on the average temperatures and the thermal cyclesdue to adpaptive DCT and DVFS controls (Section II).IV. CONCLUSIONSAn adaptive energy minimization approach for parallel applica-tions is proposed. The adaptation is facilitated through OpenMP-based power budget annotations in the applications, defined inthe modified OpenMP runtime library. Using the power budgetannotations, the proposed approach achives thermal-aware energyminimization through hierarchical controls using DCT, threadaffinity and DVFS to minimize energy consumption for the givenpower budget considering the thermal distributions among processorcores. The proposed approach is validated on a many-core platformrunning various benchmark applications, showing up to 15%reduced energy compared to the existing approaches.REFERENCES[1] R. Cochran et al.. Pack & Cap: Adaptive DVFS and thread packing underpower caps. 44th MICRO, pp.175–185. ACM, 2011.[2] M. Etinski et al.. Understanding the future of energy-performance trade-offvia DVFS in HPC environments. in JPDC, 72(4), pp.579–590, 2012.7[3] A.K. Porterfield et al.. Power Measurement and Concurrency Throttling forEnergy Reduction in OpenMP Programs. in PDPS, pp.884–891. 2013.[4] D. Brodowski and N. Golde. Linux CPUFreq–CPUFreq governors. LinuxKernel.[5] OpenMP. [Online]: http://www.openmp.org/[6] NAS Parallel Benchmarks. [Online]: www.nas.nasa.gov[7] Intel Xeon E5-2630. Intel R© Xeon R© Processor E5-2630 Family (15M Cache,2.3GHz) [Online]: http://ark.intel.com/products/64593/[8] J. Srinivasan et al.. The Case for Lifetime Reliability-Aware Microprocessors.In SIGARCH Comput. Archit. News, 32(2), pp.276–281, March, 2004.[9] C-M. Matthew et al.. Prediction models for multi-dimensional power-performance optimization on many cores. ICPAC, 2008.[10] J. Treibig et al.. LIKWID: Lightweight Performance Tools. Ch. in Competencein High Performance Computing, pp.165–175. Springer, 2012.[11] A.K. Das et al. Reinforcement learning-based inter-and intra-applicationthermal optimization for lifetime improvement of multicore systems. in DAC,pp.1–6, June, 2014.[12] Y. Hwang, and K. Chung. Dynamic power management technique for multicorebased embedded mobile devices. in IEEE TII, 9(3), pp.1601–1612, 2013.[13] S. Sinha, J. Suh, B. Bakkaloglu and Y. Cao. Workload-Aware NeuromorphicDesign of the Power Controller. in IEEE JESTCS, 1(3), pp.381–390, 2011.[14] V. Pallipadi and A. Starikovskiy. The ondemand governor. Proceedings ofthe Linux Symposium, pp. 215–229, 2006.

Thermal-aware adaptive energy minimization of open MP parallel applications

Abstract

Similar works

Full text

Available Versions

Southampton (e-Prints Soton)