A PC-based parallel computer for medium/large scale lattice QCD simulations
is suggested. The Eotvos Univ., Inst. Theor. Phys. cluster consists of 137
Intel P4-1.7GHz nodes. Gigabit Ethernet cards are used for nearest neighbor
communication in a two-dimensional mesh. The sustained performance for
dynamical staggered(wilson) quarks on large lattices is around 70(110) GFlops.
The exceptional price/performance ratio is below $1/Mflop.Comment: 3 pages, 2 figures, Lattice2002(machines

Alfieri

Christ

Csikor

Di Pierro

Eicker

Fodor

G. Pappa

Gottlieb

Lu¨scher

S.D. Katz

Z. Fodor

English

arXiv

A PC-based parallel computer for medium/large scale-lattice QCD 
simulations is suggested. The Eotvos Univ., Inst. Theor. Phys. 
cluster consists of 137 Intel P4-1.7GHz nodes. Gigabit Ethernet 
cards axe used for nearest neighbor communication in a two-
dimensional mesh. The sustained performance for dynamical 
staggered(wilson) quarks on large lattices is around 70(110) 
GFlops. The exceptional price/performance ratio is below 
$1/Mflop

Fodor, Z

Katz, SD

Papp, G

ELTE Digital Institutional Repository (EDIT)

arXiv:hep-lat/0209115v1  12 Sep 20021ITP-Budapest 588, DESY 02-141A scalable PC-based parallel computer for lattice QCDZ. Fodora, S.D. Katzb∗ and G. PappaaInstitute for Theoretical Physics, Eötvös University, Pázmány P. 1/A, H-1117 Budapest, HungarybDeutsches Elektronen-Synchrotron DESY, Notkestr. 85, D-22607, Hamburg, GermanyA PC-based parallel computer for medium/large scale lattice QCD simulations is suggested. The Eötvös Univ.,Inst. Theor. Phys. cluster consists of 137 Intel P4-1.7GHz nodes. Gigabit Ethernet cards are used for nearestneighbor communication in a two-dimensional mesh. The sustained performance for dynamical staggered(wilson)quarks on large lattices is around 70(110) GFlops. The exceptional price/performance ratio is below $1/Mflop.1. IntroductionThe most powerful computers for lattice gaugetheory are industrial supercomputers or specialpurpose parallel computers (see e.g. [1,2,3]). It ismore and more accepted that e.g. off-the-shelf PCsystems can be used to build parallel computersfor lattice QCD simulations [4,5,6].Single PC hardware has excellent price/perfor-mance ratios. For recent review papers see [7,8].We present our experiences and benchmark re-sults on a scalable system, which uses nearest-neighbor communication through Gigabit Ether-net (GigE) cards. The communication is fastenough (consuming 40% of the total time intypical applications) and cheap enough (30% ofthe total price is spent on the communication).The system can sustain ≈100 Gflops on today’smedium/large lattices. Its price/performance ra-tio is below $1/Mflops for 32-bit applications(twice as much for 64-bit applications).Already in 1999 a report was presented [5] onthe PC-based parallel computer project at theEötvös University, Budapest, Hungary. A ma-chine was constructed with 32 PCs arranged ina three-dimensional 2×4×4 mesh. Each nodehad two special, purpose designed communica-tion cards providing communication to the sixneighbors. We used the “multimedia extension”instruction set of the processors to increase theperformance. The communication bandwidth (16∗on leave from Institute for Theoretical Physics, EötvösUniversity, Pázmány P. 1/A, H-1117 Budapest, HungaryMbit/s) at that time was only enough for bosonicsimulations. Here we report on a system whichneeds no hardware development and has two or-ders of magnitude larger communication band-width through GigE cards.2. HardwareEach node consists of an Intel KD850GB moth-erboard, Intel-P4-1.7GHz processor, 512 MBRDRAM, 100 Mbit Ethernet card, 20.4 GB IDEHDD and four SMC9452 GigE cards for thetwo dimensional inter-node communication. The100 Mbit Ethernet cards with switches are usedfor job management only.We have 137 PC nodes. A single machine isused for controlling. Two smaller clusters with4 nodes are used for development. The remain-ing 128 machines can be used as one cluster with128 nodes (or two clusters with 64 nodes or fourclusters with 32 nodes) for mass production.The best number of connected nodes for agiven lattice can be determined by optimizing thesurface to volume ratio of the local sub-lattice.Changing the node topology needs reconnectingcables, which can be done easily in a few minutes.In April, 2002 the price of one node includingthe 100 Mbit Ethernet switches is $380 (see [9]).The four GigE cards with cables cost additional$160 for each node. The power consumption ofthe nodes (140W each) requires a cooling systemwhich costs around $13 per node. Thus, the to-tal node (PC+communication+cooling) price is2Figure 1. Full QCD performance results on a 64-node cluster. The node and the total performanceas a function of the spatial extension Ls. The lat-tice sizes are chosen to be L3s× 2Ls. The dottedline represents the $1/Mflops value. Wilson andstaggered quarks are both indicated. The ma-chine size (the number of communicating nodes)for a given lattice volume is shown by an insertedscale.about $553.The key element of a cost effective design is anappropriate balance between communication andthe performance of the nodes. We spent morethan twice as much on the bare PC ($393 includ-ing cooling) than for the Gigabit communication($160). These numbers are in strong contrastwith Myrinet based PC systems, for which thehigh price of the Myrinet card exceeds the priceof such a PC by a factor of two.3. SoftwareThe main operating system of the cluster isSuSE Linux 7.1, being installed on each node.Job management is done by the “main” computerthrough the 100 Mbit Ethernet network. We de-veloped a simple job-management utility to dis-tribute jobs on the nodes and collect the results.To take advantage of the Gigabit communi-cation from applications (e.g. C, C++ or For-tran), a simple C library was written using theFigure 2. Sustained staggered performance of a4-node (2×2) system as a function of the local lat-tice volume V=L3s×Ls/2 (lower panel). Squaresshow the performance of a myrinet, triangles thatof a gigabit sytem. The one-node performance isalso shown by stars. Communication is of no useif the one-node performance is better than the4-node one. Thus, the useful region is given bylocal lattices 83×4 or larger. In this useful regionthe relative difference between the performancesof the myrinet and gigabit systems (upper panel)is always less than 12%.standard Linux network interface. Currently, weare using a standard socket based communica-tion with Transmission Control Protocol (TCP)widely used in network applications. The typicalbandwidth that we can reach in QCD applicationsis around 400 Mbit/s in contrast to the theoreti-cal 1000 Mbit/s. Writing a lower level driver forthe gigabit cards which would increase the band-width is in progress. The functions of the librarycan be used to open and close the communcationchannels and transfer data between the neighbors.4. PerformanceFigure 1 gives a summary of our benchmarkruns. Single precision is used for local variables(gauge links and vectors, see [10]) and dot prod-ucts are accumulated in double precision. Usingdouble precision gauge links and vectors reduces3the performance by approximately 50%.For benchmarking we started from the MILCcode [10]. To increase the performance of thecode we modified it by three different techniques.First of all, we used the “multimedia extension”(sse) instructions of the Intel-P4 processors Aswe pointed out it in 1999 [5] this capability canaccelerate the processor by a large factor. Werewrote almost the whole conjugate gradient partof the program to assembly (including also loopsover the lattice and not only elementary matrixoperations). This way we obtained a speedup fac-tor of ≈2. (see similar results of e.g. Ref. [8,11]).Secondly, an important speedup was obtainedby changing the data structure as suggested by S.Gottlieb [12]. In the original “site major” conceptall the physical variables of a given site are storedin one structure and the lattice is an array of thesestructures. Instead of this concept one should use“field major” variables. The set of a given typeof variable of the different sites are collected andstored sequentially. This increases the numberof cache hits. Similarly to Ref. [12] this changeleads to a speedup factor of ≈2.The third ingredient of our improvement was abetter cache management by extensive use of the“prefetch” instruction.One of the most obvious features of Figure 1 isthe sharp drop of the performance when one turnson the communication. The most economic solu-tion is to turn on the communication only if thememory is insufficient for the single-node mode.Smaller local lattices can be also studied by usingthe communication (for instance for thermaliza-tion or parameter tuning); however, in these casesthe communication overhead increases somewhat.The performance as a function of the local latticevolume is shown on Fig. 2 (we also made a di-rect comparison between our architecture and aMyrinet system at DESY, Hamburg). Two ordersof magnitude smaller local lattice means factor of≈2 drop in the performance. Thus, for local lat-tices as small as 83 × 4 the performance is stillacceptable. This indicates clearly the scalabilityof this architecture. Note that the performanceof a Myrinet based system is only 5-10% betterthan that of the GigE based system.5. ConclusionWe reportde on the status of our PC-basedparallel computer project for lattice QCD (formore details see ref. [13]). Nearest-neighbor com-munication is implemented in a two-dimensionalmesh using Gigabit Ethernet cards. This ar-chitecture presents a good compromise betweencomputation and communication. The satu-rated sustained performances on large lattices arearound 0.5(0.8) GFlops/node for staggered (Wil-son) fermions, which gives a price/performanceratio better than $1.0(0.7)/Mflop.Acknowledgment. We thank F. Csikor andZ. Horváth for their continuous help. This workwas supported by Hungarian Science FoundationGrants under Contract Nos. OTKA-T37615/-T34980/T29803/M37071/OMFB1548/OMMU-708.REFERENCES1. R. Alfieri et al., hep-lat/0102011.2. http://www.rccp.tsukuba.ac.jp.3. http://phys.columbia.edu/∼cqft.4. N. Eicker et al., Nucl. Phys. (Proc. Suppl.) 83(2000) 798 [hep-lat/9909146].5. F. Csikor et al., Comp. Phys. Comm. 134(2001) 139 [hep-lat/9912059].6. X.Q. Luo et al., cs.DC/0109004.7. N. Christ, Nucl. Phys. (Proc. Suppl.) 83(2000) 111; S. Gottlieb, Nucl. Phys. (Proc.Suppl.) 94 (2001) 833.8. M. Lüscher, Nucl. Phys. Proc. Suppl. 106(2002) 21.9. http://www.pricewatch.com10. http://physics.indiana.edu/∼sg/milc.html.11. M. Di Pierro, hep-lat/0110116.12. S. Gottlieb, hep-lat/0112026, hep-lat/0112038, hep-lat/0112039.13. Z. Fodor, S. D. Katz and G. Papp, hep-lat/0202030.

A scalable PC-based parallel computer for lattice QCD

Crossref

A scalable PC-based parallel computer for lattice QCD

Abstract

Similar works

Full text

Available Versions

ELTE Digital Institutional Repository (EDIT)

Crossref