psort \ue8 stato il pi\uf9 veloce software di ordinamento per macchine di classe PC dal 2008 al 2011 (benchmark Pennysort, http://sortbenchmark.org) e un suo adattamento per cluster ha migliorato il record per il benchmark datamation di quasi un ordine di grandezza nel 2011. Il rapporto tecnico ufficiale si trova sul sito sortbenchmark.org (che cataloga i pi\uf9 efficienti software di ordinamento per varie categorie di task/hardware - originariamente mantenuto dal premio Turing Jim Gray) all'URL http://sortbenchmark.org/psort_2011.pdf -- Ulteriori dettagli si possono trovare nelle pubblicazioni:
P. Bertasi, M. Bressan, E. Peserico. psort, yet another fast stable sorting software, ACM Journal of Experimental Algorithmics, vol. 16, 2011 -- 
P. Bertasi, M. Bonazza, M. Bressan, E. Peserico. Datamation: a quarter of a century and four orders of magnitude later. Proc. of IEEE CLUSTER 201

Bertasi, Paolo

Bogo, Federica

Bressan, Marco

PESERICO STECCHINI NEGRI DE SALVI, Enoch

psort è stato il più veloce software di ordinamento per macchine di classe PC dal 2008 al 2011 (benchmark Pennysort, http://sortbenchmark.org) e un suo adattamento per cluster ha migliorato il record per il benchmark datamation di quasi un ordine di grandezza nel 2011. Il rapporto tecnico ufficiale si trova sul sito sortbenchmark.org (che cataloga i più efficienti software di ordinamento per varie categorie di task/hardware - originariamente mantenuto dal premio Turing Jim Gray) all'URL http://sortbenchmark.org/psort_2011.pdf -- Ulteriori dettagli si possono trovare nelle pubblicazioni:
P. Bertasi, M. Bressan, E. Peserico. psort, yet another fast stable sorting software, ACM Journal of Experimental Algorithmics, vol. 16, 2011 -- 
P. Bertasi, M. Bonazza, M. Bressan, E. Peserico. Datamation: a quarter of a century and four orders of magnitude later. Proc. of IEEE CLUSTER 201

BERTASI, PAOLO

BOGO, FEDERICA

BRESSAN, MARCO

PESERICO STECCHINI NEGRI DE SALVI, ENOCH

Archivio istituzionale della ricerca - Università di Padova

psort 2011 – pennysort, datamation, joulesort ∗Paolo Bertasi, Federica Bogo, Marco Bressan, Enoch PesericoUniv. Padova, Italy - psort@dei.unipd.itAbstractThis memo reports the results of our psort (general purpose) sortingsoftware on a number of hardware configurations. “Vanilla” psort sorted10GB for 2122 joules on a Nokia N900 smartphone with an extra Kingston16GB flash drive (joulesort 10GB Daytona+Indy benchmark). It sortedover 286.2 GB for 0.01$ of computing time on a PC equipped with an AMD2.7GHz Sempron processor, 4 GB of RAM and 5 Samsung SpinPoint F4HD322GJ drives (pennysort Daytona benchmark); on the same hardware,a hand tailored version of psort sorted over 334.4 GB for 0.01$ (pennysortIndy benchmark). Finally, a cluster version of psort sorted 100 MB in40.5 milliseconds from disk to disk (and in 55 milliseconds from platterto platter) on a 16 PC cluster with 96 Samsung SpinPoint F3 HD103SJdrives and 10Gb/s Ethernet (datamation benchmark).1 IntroductionThis memo describes the psort entry for the 2011 sorting benchmarks – morespecifically pennysort (both Indy and Daytona), datamation, and joulesort(10GB Daytona). Section 2 briefly describes the structure of psort (for a morein-depth description see [3, 2]). Section 3 describes the pennysort benchmarkresults. Section 4 describes the datamation benchmark results. Section 5 de-scribes the joulesort benchmark results.2 psortpsort is a fast, stable external sorting software originally designed for PC classplatforms. The experiments described in this memo show, however, that itcan also run surprisingly well on small embedded platforms and can be easilyextended to run on small-to-medium sized PC clusters. Its architecture is de-scribed in detail in [2, 3]; however, in this section we briefly review its internals.In a nutshell, psort is a “classic” (deterministic and stable) merge-basedsorter; it sorts files viewed as sequences of (at most 264) records of arbitrary∗This work was support in part by project AACSE, and in part by Samsung through agenerous equipment donation.1size (up to 264 bytes), according to an arbitrary infix, by first sorting individual“runs” of data approximately the size of the main memory and then mergingthose runs into a single sorted run. psort is entirely merge-based even in thefirst phase, using k−way mergesort (with adaptive k) to sort the initial runs.Note how this is a marked difference from most other Pennysort competitors,that first distribute the records into small buffers according to a (usually 16bit long) prefix of the key and then sort those buffers according to a variety ofalgorithms, sometimes randomized. The rationale behind psort ’s purely merge-based implementation is providing a deterministic behavior: this guaranteesstability and high performance even against adversarially chosen inputs. psortcan use disks independently (as described e.g. in [4]), or as a single RAIDlogical volume. Furthermore, psort can efficiently exploit multicore processors;however, with the record footprint of the sorting benchmarks (10 byte keys with90 bytes of payload), a single core always appears more than adequate for itsneeds.3 PennysortThe pennysort benchmark involves sorting as many 100 byte records as possible(according to the first, random 10 bytes), in 0.01$ of computer time (assumingthe cost of a machine is amortized over three years). “Indy” software can bespecifically optimized for the pennysort record format and key distribution;“Daytona” software must be general purpose sorting software.This section describes our hardware, filesystem and OS choices for the pen-nysort benchmark (Subsection 3.1), the “Indy” optimizations – taylored to thehardware and the specific key distribution of the pennysort benchmark (Sub-section 3.2) – and the actual test results for both the Daytona and the Indyversion of psort (Subsection 3.3).3.1 Hardware, filesystem and OSHardware – at least the parts critical to efficient external sorting – has notconsiderably improved in terms of price-performance in the last two years. Diskperformance has increased somewhat; RAM and motherboard performance lessso; the processor critical statistics (size and access speed of the various levelsof cache) have remained almost unchanged. On the other hand, prices haveslightly increased across the board.After various tests, we selected as the best disk candidates the HitachiHDS721050CLA362 500GB and the Samsung SpinPoint F4 HD322GJ 320GB.The Hitachi disks had on their side an extremely low price of 39$, and a slightlylarger size of 500GB (meaning speed remained closer to the top speed achiev-able on the outer rim throughout the sort). The Samsung disks were, however,some 15−25% faster on the outer rim and, perhaps more importantly, appearedmore reliable (exhibiting far less performance variation over time and betweendifferent disks).2We paired our disks with the cheapest processor we could find (a SempronSargas 140 2.7 GHz), an ASRock 880 gm-le motherboard, and 2 banks of 2GBDDR3 1333 MHz 8-8-8-24 RAM.We installed Gentoo GNU/Linux on our system (compiled with Huge Pagessupport), and created a number of XFS RAID 0 (stripe size: 128KB) parti-tions, starting from the (faster) portion of the disk closer to the outer rim; eachpartition was just large enough to hold our input and output. Note how thissize was different for the Daytona and Indy tests (see below); the number ofpartitions was also different – 2 for the Indy test and 3 for the Daytona test (forthe latter, the new 2009 rules forbid overwriting the input). [2] motivates ourchoice of XFS as a filesystem. We also disabled any “smart” disk scheduling.We compiled psort using gcc version 4.4.5.3.2 Indy optimizationsIn an effort to push psort as far as it can go when hand-optimized for a specificmachine and architecture, we retooled it into an Indy version optimized forthe pennysort benchmark. We hard-coded all parameters (record and sort size,key position and length etc.), and re-coded the first phase so as to actuallydistribute records into bins according to the initial 16 bits of the key beforeactually sorting each bin. This latter sort uses “standard” psort merge-basedcode, with one catch: it only operates on the first 8 bytes of each key. Becauseof the random distribution of the keys, it is extremely unlikely for two keys toactually coincide on the first 8 bytes; we leave the check that this is really thecase to the second phase of the sort, where the computational pressure is lower.We actually have code in place to deal with identical key prefixes and guaranteethe final output is sorted, but it never gets invoked for the current sort sizes.3.3 Pennysort resultsA 451.36$ cost yielded a 2096 seconds time budget (see figure 3.3). In this time,“vanilla” psort sorted and merged 91 runs of 31457280 records each, for a totalof 2862612480 records (of 100 bytes each) and a Daytona sort size of slightlymore than 286.2 GB. This was achieved starting with the input (generated bythe gensort software from the sort benchmark site) on the innermost (slowest)partition, writing the temporary output of the first phase on the outermost(fastest) partition, from which it was read during the second phase to producea final output on the middle partition. For checking purposes, the sum of the32 bit CRCs of all records was 55 : 50 : 23 : c7 : f7 : de : b3 : cc.In the same time, Indy psort sorted 116 runs (of 28835840 records of 100bytes each), for a sort size of slightly more than 334.4 GB. This was achievedpositioning the input on the faster, outermost partition, overwriting it with thetemporary output at the end of the first phase, and writing the final outputinto a second, slower partition immediately closer to the spindle. For checkingpurposes, the sum of the 32 bit CRCs of all records was 63 : b0 : 3d : 80 : 31 :9f : 72 : fa.3Figure 1: The newegg price listing for our hardware. Note that we are nottaking into account any instant savings for the final price calculation44 DatamationDatamation is the “original” sorting benchmark [1]: benchmarks such as pen-nysort, minutesort, joulesort [5, 7] etc. are all derived from it, and maintain itsrecord format (a 10 byte key followed by an incompressible 90 byte payload).Datamation involves sorting 1 million such records in as little time as possible.When datamation was first proposed as a benchmark of transaction processingpower, over twenty years ago, the sorting time was of the order of an hour. Tenyears ago, it had already dropped to less than a second, becoming a benchmarkof a system’s responsiveness rather than of its sustained throughput.Still, responsiveness is an extremely important characteristic for many datamanagement systems (webserver-linked databases, online game servers, auto-matic trading systems etc.) and one that none of the other sorting benchmarkscurrently addresses. Thus, while the focus of datamation may have shifted, itstill remains a crucial sorting benchmark [6].4.1 Hardware, filesystem and OSWe tested psort on a 16 PC Linux cluster provided by Univ. Padova ProjectAACSE. Each machine is equipped with 6 Samsung Spinpoint F3 HD103SJ(1TB) drives, generously donated by Samsung. These drives, exhibiting a peaktransfer rate close to 150 MB/s, were the fastest 7200 rpm drives in mid 2010,when the cluster was set up. Each node runs a Nehalem core i7 950 CPU at3.07Ghz, supported by an ASUS P6T SE motherboard with 3 banks of 4 GBtri-channel DDR3 running at 1.6 GHz. The cluster is connected by a dedicated10Gb/s Ethernet through Myricom 10G-PCIE-8B-S cards and a Fujitsu XG2600switch.The operating system is Gentoo Linux – with a 2.6.36 kernel, patched withCon Kolivas’s (CK) kernel patch set. The drives are configured as a RAID 0with 1024 KB stripe size, with maximum (2048 blocks) readahead, and XFS asa filesystem. On sequential reads, they saturate the ICHR10 southbridge of themotherboard when reaching the value of approximately 680 MB/s.Perhaps more important for the datamation benchmark is the disk access la-tency. Each disk takes approximately 8.3 milliseconds to complete a revolution.According to specifications, the disk head can typically access a given track inless than 5 milliseconds “on average”. In theory, if one could align a read insuch a way that it was all positioned on a single track (over 1.2 MB of data onthe outer rim), this would entail reads of up to 1.2 MB per disk in less than 15millisecond. However, such positioning is extremely difficult to achieve (due tothe variable number of bytes per track) and in practice, we have seen slightlylonger access times to read or write 1 MB on each disk (over 20 milliseconds).On the other hand, transfering data to/from the hard disk on-board cache isdefinitely faster: about 2− 3 milliseconds plus a time proportional to the trans-fer time at the peak bandwidth (i.e. about 11−12 milliseconds to transfer 6MBfrom or to a RAID of 6 disks).54.2 Cluster psortLike virtually all cluster sorting software, cluster psort reads samples data fromeach node, determines a partition of the (sorted) data between the various nodes,sends each record to the appropriate destination node, and finally sorts andwrites the record to disk. Data can actually be partitioned into a number ofsubsets that is a multiple of the number of nodes; this allows data that reacheseach node to be already partially sorted, minimizing computational pressure andallowing the destination node to pipeline network I/O, computation, and diskI/O. In the case of the datamation benchmark, to minimize latency we replacedthe portion of the code that determines the partition of the sorted data on asimple, static partition based on the random distribution of the keys; data werepartitioned into 256 sets, with 16 sets assigned to each node.The psort process is launched through command-line on one of the 16 nodes.Every node in the cluster listens on a specified UDP port for commands fromthe rest of the cluster, executing them through a fork() and an exec() –essentially a very minimal, but “general purpose” shell that allows the masternode to create psort processes on the rest of the cluster. After reading andpartitioning the data from its local RAID, each of the 16 nodes goes through15 phases. During the ith phase, the jth node sends to the ((i+ j)modn)th node“its” records – and signals to the ((j − 1)modn)th node that it has done so,allowing it to enter the next phase. This simple barrier mechanism is crucial toavoid expensive collisions on the network.Once every node has received all the records partitioned in 16 sets (with allrecords in each set smaller than all records in the next) it sorts and sends todisk each set. After this, each node signals the master node it has finished; themaster process terminates after receiving such an acknowledgement from everynode.4.3 Datamation resultspsort was compiled with gcc version 4.4.4, with flags -O3 -mtune=native -march=native-funroll-loops -funsafe-loop-optimizations.To minimize the execution time, we placed the data on a small partitionclose to the outer rim of each disk, and before launching psort, we read thedata once. This ensured that all the data was already in the disk onboard cachewhen we launched psort (there was slightly more than 1MB of data per disk).Similarly, each disk was configured with hdparm -W1 to ensure that every writecommand returned as soon as the data was on the onboard cache of the disk,and not necessarily on the physical platter.Out of 200 trial runs, the minimum was slightly below 40.5 millisecondsand the median slightly below 40.8 (see figure 4.3). This “cache-to-cache” timeis approximately the interval between two consecutive TV frames, and, to thebest of our knowledge, over an order of magnitude less than the last recordeddatamation result [6].While the onboard cache is part of the disk – and thus the “cache-to-cache”600.050.10.150.240.4 40.6 40.8 41 41.2 41.4 41.6 41.8frequencyexecution time (ms)datamation cache-to-cacheFigure 2: Datamation time distribution, sorting from and to the disk onboardcachetime is the disk-to-disk time of the Datamation benchmark, we also measuredthe “platter-to-platter” sort time, by flushing the disk onboard caches (withdummy reads) before launching psort, and by configuring the disks with hdparm-W0 to ensure that no write command would return before the data was on theplatter. This led to a somewhat larger variance in the sort times; out of 200 trialruns, the fastest took slightly less than 55 milliseconds to complete, the median57 (see figure 4.3). For checking purposes, the sum of the 32 bit CRCs of allrecords was 7 : a1 : 9c : ff : 46 : 74 : 38, and 0 duplicate keys were encounteredwhen the output was (automatically) verified to be in sorted order.5 JoulesortTo test its versatility, we also decided to test “vanilla” psort on a computingplatform that was completely different from that for which it was originally700.050.10.150.20.2554 56 58 60 62 64 66frequencyexecution time (ms)datamation platter-to-platterFigure 3: Datamation time distribution, sorting from and to the physical platter8designed: namely, a Nokia N900 smartphone. Surprisingly, psort managed tosort 10GB of data for slightly less energy than the 2010 joulesort record.5.1 Hardware, filesystem and OSThe Nokia N900 is based on an ARM 7 Cortex A8 processor running at 600MHz, Maemo 5 Linux, and an internal 32GB flash drive. We found that wehad about 100MB of memory space available for running psort. We added a16GB Kingston class 10 flash drive in the Nokia N900 expansion slot. We usedext3 as a filesystem for the internal card and vfat for the microSDHC. We crosscompiled psort with the Scratchbox toolchain.5.2 Test setupWe tested the total energy consumption with a Yokogawa WT 130 wattmeter(0.5% RDG + 0.3% MG error), sampling at 200 millisecond intervals.Interfacing the Nokia N900 with the wattmeter was not easy – the mainmeans of communication of the Nokia is its wireless system, which can drain aconsiderable amount of energy, distorting the measurement. Instead, we optedto have the Nokia N900 launch psort after a brief sleep period, emitting abeep immediately before beginning the sort, and a second beep immediatelyafter completing it. To emit the beep, the sound system must be activated,creating a small energy spike that can be detected before the actual beep isemitted. Thus, the wattmeter starts measuring on the energy spike, is verifiedto be already measuring when the first beep is emitted, and stops measuringimmediately after the second beep. This provides an upper bound to the totalenergy consumption (introducing an error that is still probably below 0.1% -less than 2 seconds, over more than 40 minutes of sorting time).5.3 Joulesort resultsWe measured time and energy of 5 different psort trial runs, with all the sub-systems of the Nokia N900 not instrumental to the sort (screen, wireless etc.)turned off. The average time was 2583 seconds. The average energy was 2122joules (with an instrument error of at most 17 joules) and a standard deviationof 2.22 joules.S1 time (s) S1 energy (Wh) S2 time (s) S2 energy (Wh) total energy (Wh) total energy (J)1430 0.305 1152 0.2842 0.5892 21211435 0.3060 1151 0.2839 0.59 21241438 0.3067 1149 0.2834 0.5901 21241430 0.305 1150 0.2837 0.5887 21191430 0.305 1152 0.2842 0.5892 21219References[1] A measure of transaction processing power. Datamation, 31(7):112–118,1985.[2] P. Bertasi, M. Bressan, and E. Peserico. psort, yet another fast stable sortingsoftware. In SEA, pages 76–88, 2009.[3] P. Bertasi, M. Bressan, and E. Peserico. psort, yet another fast stable sortingsoftware – in press. Journal of Experimental Algorithmics – in press, 2011.[4] R. Dementiev and L. Kettner. Stxxl: Standard template library for xxl datasets. In In: Proc. of ESA 2005., pages 640–651.[5] J. Gray. A measure of transaction processing 20 years later. CoRR,abs/cs/0701162, 2007.[6] F. I. Popovici, J. Bent, B. Forney, A. Arpaci-dusseau, and R. Arpaci-dusseau.Abstract datamation 2001: A sorting odyssey, 2002.[7] S. Rivoire, M. A. Shah, P. Ranganathan, and C. Kozyrakis. Joulesort: abalanced energy-efficiency benchmark. In SIGMOD Conference, pages 365–376, 2007.10

psort

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Padova