Advanced Cyberinfrastructure Research & Education Facilitators Virtual Residency 2015 hosted at the University of Oklahoma (OU) and available live via videoconferencing Sunday May 31 - Saturday June 6 2015 for 
Computational & Data-enabled Science & Engineering Track: Lecture: Benchmarking and Tuning by Brett ZimmermanPresented at the Advanced Cyberinfrastructure Research & Education Facilitators Virtual Residency 2015, June 2, 2015.ACI-REF Workshop

The University of Oklahoma

The  University of Oklahoma Supercomputing Center for Education & Research (OSCER)N

Zimmerman, Brett

SHAREOK repository

Benchmarking  and  TuningACI-­REF  WorkshopBrett  ZimmermanUniversity  of  Oklahoma  – IT/OSCERint  power(int  x,  int  y){int  result;;if  (y  <  0)  {result  =  0;;}  else  {for  (result  =  1;;  y;;  y-­-­)result  *=  x;;}return  result;;}int  power  (int  x,  int  y)  {int  result;;if  (y<0)  {result  =  0;;}  else  {for  (result  =  1;;  y;;  y-­-­)result  *=  x;;}return  result;;}int  power  (int  x,  int  y){int  result;;if  (y<0)  {result  =  0;;}else{for  (result=1;;  y;;  y-­-­){result  *=x;;}}return  result;;}ABCDon'tl Randall  Munroe  – http://xkcd.com/1513/Complementary  ProcessesSystemsProcessesResource  Utilizationl CPUl Memory  (cache,  RAM;;  space,  bandwidth)l Disk  (  intentional  as  well  as  swap  and  paging)l Network  (bandwidth,  latency)A  bottleneck  always exists.Benchmarking  and  Tuningl Systems− Measure  the  performance  characteristics  of  the  system− Adjust  the  system  to  accommodate  a  given  usel Hardware  adaptationl System  tunablesl Processes− Measure  the  resource  usage  of  a  given  piece  of  code− Adjust  the  code  to  make  efficient  use  of  the  systemUnitsl SI  vs.  IEC  unitsl KiB  210  (1024)           KB  103  (1000)  l MiB  220  (1048576)            MB  106  (1000000)l GiB  230  (1073741824) GB  109 (1000000000)l TiB  240  (1099511627776) TB    1012 (1000000000000)Some  Groundwork  – Queueingl L  =  λW− Units  of  work  within  the  system  is  equal  to  the  product  of  the  arrival  rate  of  the  units  of  work  and  the  time  the  unit  spends  in  the  systemBroad  Optimization  Approachesl Reducing  Visit  Counts− Amortize  fixed  overhead  by  aggregating  operations− Not  doing  stuffl Reducing  Wait  time− Reducing  Overhead− Using  multiple  queues− Space  tradeoffs/Caching− Efficiencyl Overriding  concerns− The  first  priority  for  tuning  is  stabilityOptimization  Overviewl Outside  measurement  of  resourcesl Code  review− Algorithm− Implementationl Compilerl 80/20  Rulel Time/space  tradeoffsl Effect  of  the  Storage  HierarchyThe  Storage  Hierarcyl ~2  ns  -­ L1l ~5  ns  -­ L2l ~14  ns  -­ L3l ~60  ns  -­ RAMl ~3  ms  -­ DiskDetermining  System  Performancel Published  specifications− Processor− Bus− Memoryl Known  Benchmarks− Network  (ping,  netperf,  qperf,  OSU  Micro)− IO  (Iozone,  Fio)  − Global  (HPL)Coarse  Timing  Datal /usr/bin/time  rather  than  shell  builtinTOTAL RUN TIME: 0 days 0 hours 2 minutes 15 seconds 805 msecCommand being timed: "Orca/3.0.1/bin/orca /home/zim/orca6.inp"User time (seconds): 107.59System time (seconds): 1.88Percent of CPU this job got: 80%Elapsed (wall clock) time (h:mm:ss or m:ss): 2:15.82Average shared text size (kbytes): 0Average unshared data size (kbytes): 0Average stack size (kbytes): 0Average total size (kbytes): 0Maximum resident set size (kbytes): 599168Average resident set size (kbytes): 0Major (requiring I/O) page faults: 219Minor (reclaiming a frame) page faults: 598418Voluntary context switches: 22746Involuntary context switches: 1629Swaps: 0File system inputs: 642448File system outputs: 787584Socket messages sent: 0Socket messages received: 0Signals delivered: 0Page size (bytes): 4096Exit status: 0Measuring  System  Usagel Crontab− */10  *  *  *  *      /usr/lib64/sa/sa1  1  1  -­S  XALLl Sysstat  packagel Reports  paging,  IO  usage,  per  block  device  usage,  interrupt  counts,  power  management,  network  utilization,  cpu  utilization,  run-­queue  length,  memory  utilization,  swap  space,  inode  and  dentry  cache,  switching  l e.g.  sar  -­n  EDEV  will  display  per-­device  network  errors  for  each  network  device  on  the  systemSystem  Tunablesl /proc− net− vm− kernel− fsl proc.sys.net.ipv4.neigh.default.gc_thresh2l Documentation− Kernel-­doc  package− /usr/share/doc/kerne-­version/Documentation/sysctll ltrace  -­Sfc  example% time     seconds  usecs/call     calls      function------ ----------- ----------- --------- --------------------50.03    2.159365     2159365         1 __libc_start_main46.43    2.004238       35162        57 strcpy0.63    0.027280          69       393 strlen0.43    0.018509       18509         1 getaddrinfo0.26    0.011290          68       166 malloc0.25    0.010619          68       156 free0.20    0.008802          68       128 realloc0.20    0.008774         165        53 SYS_open0.20    0.008771          67       129 ferror0.20    0.008723          69       126 fgets0.19    0.008099        8099         1 open0.09    0.003855          68        56 memcpy0.09    0.003750        1250         3 fclose0.09    0.003712         100        37 SYS_close0.06    0.002697        2697         1 connect0.06    0.002650         662         4 SYS_connectgprofl Compile  with  profilingl Run  code− Normal  output− Side  effect:  profiling  data  written  to  gmon.outl Report  profiling  data− gprof  –line  –flat-­profile  area-­serial  gmon.out%   cumulative   self              self     total           time   seconds   seconds    calls  Ts/call  Ts/call  name    0.00      0.00     0.00   200000     0.00     0.00  function_to_integrate (area_under_curve.c:139 @ 400b30)0.00      0.00     0.00        1     0.00     0.00  input_arguments (area_under_curve.c:55 @ 4008a8)0.00      0.00     0.00        1     0.00     0.00  sum_intervals (area_under_curve.c:108 @ 400a48)Perf  stat  for  Array  walkperf stat -Bd array_inner Performance counter stats for 'array_inner':4175.666892 task-clock                #    1.000 CPUs utilized          7 context-switches          #    0.002 K/sec                  7 cpu-migrations            #    0.002 K/sec                  1,872 page-faults               #    0.448 K/sec                  11,577,144,187 cycles                    #    2.773 GHz                     [89.99%]3,943,947,255 stalled-cycles-frontend   #  34.07% frontend cycles idle    [89.99%]1,445,711,602 stalled-cycles-backend    #   12.49% backend  cycles idle    [79.98%]19,770,949,724 instructions              #    1.71  insns per cycle        #    0.20  stalled cycles per insn [89.99%]4,361,497,083 branches                  # 1044.503 M/sec                   [89.99%]8,346,798 branch-misses             #    0.19% of all branches         [90.01%]3,844,013,042 L1-dcache-loads           #  920.575 M/sec                   [90.01%]81,717,861 L1-dcache-load-misses     #    2.13% of all L1-dcache hits   [90.01%]3,871,347 LLC-loads                 #    0.927 M/sec                   [90.01%]2,255,846 LLC-load-misses           #  58.27% of all LL-cache hits    [90.00%]4.176491482 seconds time elapsedperf stat -Bd array_outer Performance counter stats for 'array_outer':13440.776165 task-clock                #    1.000 CPUs utilized          18 context-switches          #    0.001 K/sec                  5 cpu-migrations            #    0.000 K/sec                  1,872 page-faults               #    0.139 K/sec                  37,238,405,421 cycles                    #    2.771 GHz                     [89.99%]29,860,712,853 stalled-cycles-frontend   #  80.19% frontend cycles idle    [90.00%]23,981,968,848 stalled-cycles-backend    #  64.40% backend  cycles idle    [80.00%]19,791,761,293 instructions              #    0.53  insns per cycle        #    1.51  stalled cycles per insn [90.00%]4,369,050,335 branches                  #  325.059 M/sec                   [90.00%]8,363,924 branch-misses             #    0.19% of all branches         [90.00%]3,851,175,888 L1-dcache-loads           #  286.529 M/sec                   [90.00%]563,864,367 L1-dcache-load-misses     #  14.64% of all L1-dcache hits   [90.00%]18,290,787 LLC-loads                 #    1.361 M/sec                   [90.00%]16,310,806 LLC-load-misses           #  89.17% of all LL-cache hits    [90.00%]13.442679796 seconds time elapsedPerf  Countersl perf  packagel perf  list  – List  available  counters  and  tracepointsl perf  top  – System  level  profiling− /proc/sys/kernel/perf_event_paranoid  – Allow  unpriviledged  users  to  collect  performance  counter  data-­1  not  paranoidl perf  stat  – Run  a  command  and  gather  statsSystemTapl Allows  fine-­detail  monitoring  of  the  kernell systemtap,  systemtap-­runtime,  kernel-­debuginfo,  kernel-­debuginfo-­common-­arch,  kernel-­devell stap  -­v  -­e  'probe  vfs.read  {printf(“read\n”);;exit()}'l Systemtap  automates  adding  instrumentation  modules  to  the  running  kernell stapdev  – privileged  stap  users.    Effective  rootl Stapusr  – can  use  staprun  to  run  modules  in  /lib/modules/version/system-­tap/SystemTap  Exampleglobal wevent,reventprobe vfs.write.return {wevent[execname()] += $return}probe vfs.read.return {revent[execname()] += $return}probe timer.s(10) {printf("___READS___\n")foreach(exe in revent- limit 10)printf("%15s: %d bytes\n",exe,revent[exe])printf("\n")printf("___WRITES___\n")foreach(exe in wevent- limit 10)printf("%15s: %d bytes\n",exe,wevent[exe])}___READS___dd: 335304334 bytesirqbalance: 288890 bytescrond: 57767 bytessadc: 52016 bytesdate: 13990 bytessshd: 11952 bytesunix_chkpwd: 11934 bytessa1: 11495 bytesscreen: 5963 bytessystemd-journal: 5112 bytes___WRITES___dd: 335300191 bytessshd: 12205 bytessadc: 11228 bytesscreen: 9077 bytesstapio: 3558 bytessystemd-logind: 2102 bytesping: 1831 bytesauditd: 1783 bytessystemd: 472 bytesgdbus: 392 bytes

Benchmarking and tuning

https://shareok.org/bitstream/11244/15470/3/acirefvirtres2015_talk_benchmarking_tuning_20150602.pdf

Benchmarking and tuning

Abstract

Similar works

Full text

Available Versions

SHAREOK repository