Dimou, M P

Györgyi, G

Kaim, J

Segal, B

CERN Document Server

26 August, 1996 CERN CN Report 96-13 OCR Outputposed by the CORE system.The conclusion is that HIPPI is a highly viable solution for the demanding data-rate requirementsin the Computer Centre.very high speed 10 km serial I-HPPI link connecting the NA48 experiment to the Meiko CS2 machineperformance tape staging to large numbers of FDDI—based tape servers, as well as the support of aA later phase of the tests involved two HIPPI to FDDI "GigaRouters". These permit highand high performance tape drives on the IBM SP/2.UltraNet and show the performance of HIPPI used for tape staging between SGI Power Challengesthe feasibility of HIPPI as a high bandwidth interconnect, compare the performance of HIPPI toThis paper presents the results of tests, conducted at the CERN Computer Centre, trying to provephase out UltraNet.Power Challenges), provide a high speed tape staging link for powerful machines and allow CORE tois to complement FDDI as a high bandwidth interconnect between powerful machines (especially SGIPerformance Parallel Interface, was recently introduced in the CERN Computer Centre. I—HPPI’s rolehigh data rates and redundancy. The newest component of this Network - HIPPI, the HighThe "Hybrid Backplane Network" consists of FDDI and UltraNet, which work in parallel providingscene, relies on a scalable "Hybrid Backplane Network" that can support the appropriate data rates.CORE, the Centrally Operated RISC Environment, which dominates the CERN central computingAbstractGabor Gyori (OPAL)Ben Segal, Jacek Kaim & Maria Dimou (CN-PDP/N S)HIPPI in the CERN Comouter CentreCERN-CN-96-013S ee Qs (S flCN / 96 / 13Lk I I(L€R1u-ew»9€>-·©1?>CERN CN Report 96-13 26 August, 1996 OCR Outputby varying the buffer size and the number of concurrent TCP streams. The tests were performedbetween two hosts nmning multiple concurrent TCP streams. Different measurements were obtainedgoal of these tests was to determine the maximum achievable performance of this network mediumA number of memory—to-memory pure TCP transfer tests were first conducted over UltraNet. The2. UltraNet pure TCP testsFig.2 Test machine specificationsMalka, CS2 10oMI—Iz32 tlaubla I Ross, I-Iv¤arsI¤AI=tcIBM, sI>/2 s7MI—Iz64 I IBM, Pawarz20OMl-lzSGI, Challenge XL I 12 I MIPS, R4400200Ml-IzSGI, Challenge XL I 12 I MIPS, FI4400200MI—IzSGI, Challenge XL | 20 I MIPS, R4400200MHzSGI, Challenge XLIS I 8 I MIPS, R44OOtime of the tests.Below is a table containing the technical specifications of the machines involved in the tests at theFig.1 HIPPI setup in the CERN Computer CentreSG' Ch’="'€"99 XL SGI Challenge xt IBM sp/2cemsp2shift8SMU... ......4;. .. r... ..I-IIPPI switchSerial I—III>I¤I to CERN North Area..§;:`iiii. ````''' I ::· mj;=itSGI Challenge XLIS SGI Challenge XL Meiko CS2cemcs2shiIt2 shI1t3parallel interfaces using a l6Xl6 non-blocking PHPPI switch from Avaika.four SGI Challenges, the IBM SP/2 and the Meiko CS2. These machines were interconnected byAt the time the tests were performed the HIPPI setup in the CERN Computer Centre consisted oftherefore of great importance.shipped on the "Hybrid Backbone Network". Good performance of RFIO transfers over I-HPPI iscurrently used by the CORE system, and transfers using it constitute more than 90% of the dataTCP and allows secure bulk data transfers. RFIO is of special interest in this context, since it isto-memory RFIO test. RFIO is the Remote File I/O protocol developed at CERN, which runs on top ofTwo different connection modes were used - a memory-to-memory pure TCP test and a memorytransfer statistics are presented.and amount of data to be transferred can be selected. The transfer gets initiated and at the end of theto the server using the socket interface. Parameters like buffer size, socket buffer size, user buffer sizemachine and the client part on the other machine to be tested. The client establishes a TCP connectionThe tests were performed with small c1ient—server applications. The server part is started on one1. IntroductionOCR OutputHIPPI inthe CERN Computer Centre26 August, 1996 CERN CN Report 96-13 OCR OutputCPUs each. . The kernel buffer size allocated to sockets (socket buffering) was set again to 512kB.The tests were performed between the same two Silicon Graphics Challenge XL machines with 12measurements were obtained by varying the buffer size and the number of concurrent TCP streams.performance of the network medium running multiple concurrent TCP streams. Differentnetwork medium. The goal of these tests was again to determine the maximum achievableThe same memory-to-memory pure TCP transfer tests were now repeated using HIPPI as the2. HIPPI pure TCP testsstream, the performance reaches a level close to the maximum.buffer size, which is the buffer size used by RFIO in production, and with more than one parallelconcurrent streams, and the buffer size has a substantial impact on the overall performance. At 128kBThis graph shows clearly that the performance changes only slightly with the number ofFig. 3 Results of memory-to-memory pure TCP test over UItraNetBuffer size (kB)=·= 14 B1024 · r` 5um”°°‘OD 2 dfstreams2.04.0E 6.0m 8.010.012.014.0Buffer size: 1024 kB#streams: 3Max speed: 12.3 MB/snnUltraNet trander speeds pure TCP (SGI to SGI)58152965 4344 4979 5223 54501706 3551 4718906690758194 8457 84312560 4172 5354 6257 72461011910168 9943 97049164 9260 99473657 5993 763310609 1076110659 106078818 10802 110286023 11541 11533110101114511038 10946 1101911150 11245 10907 110606826112941165310811 1079311264 11161 11183 10781 111068533113561144211740 1155510240 11444 11165 11294 11072 112851195212191 1217111986 12280 12032 11928 11810 117411024010‘·‘·‘ 1‘’2:1:3:55:5:i:i:¢:§:¤:¤:¤:¢:¢i‘total bandwidth of the concurrent streams in kB/s.number of concurrent TCP streams and the rows represent the buffer sizes in kB. The values are theIn the tables and graphs below the results of these tests are presented. The columns represent theallocated to sockets (socket buffering) was set to 512kB.between two Silicon Graphics Challenge XL machines with 12 CPUs each. The kemel buffer sizeHIPPI inthe CERN Computer Centre26 August, 1996 OCR Output CERN CN Report 96-13intemal IBM SP/2 fast switch. The setup is described with the figure below:Challenge to an IBM SP/2 gateway node - sp04l. From there the transfer is routed to sp023 via theserver (SGI Challenge) and sp023 — a tape server (IBM SP/2 node). The transfer goes from the SGIA number of tests were done to measure the performance of transfers between shift7 - a CPU4. HIPPI RFIO tests to tape server (3590)buffer size of 128kB - the standard RFIO buffer size, the performance is acceptable.RFIO ACK messages due to thc TCP Nagle algorithm (discussed further below). Fortunately at thepicks up again to reach a new peak. This behaviour is thought to be due to delayed transmission ofpeak, to descend again with increasing buffer size up to 512kB. Only at 1024kB the performancesize. Below the nominal 128kB, the performance is disastrous. At 128kB the performance reaches aWhat can be noted here is the extraordinary sensitivity of RFIO with respect to the chosen bufferFig. 5 Results of memory-to-memory RFIO test over HIPPI&1tfersize(kB)_ ___ g 3 § @ $ SF2v N' '|"'“0.0 2 #streams5.010.015.0%20.025.01030.035.040.045.050.0Buffer size: 1024 kB#streams: 10Max speed: 49.8 MB/sB socket butterinHiPPI trander weeds using RFIOITCP (SGI to SGI)476 555 639 71980 1 60 238 31 7 396942 1260 1437160 480 638 754 1117 1595320960 1 920 2240 2546 2792 3200320 640 1 280 15923766225094 28632 31703 35428 396506736 1 1807 1 6422 2082426700 29300 30643 323817757 11868 1 601 7 1 9050 21171 246842410919958 25336 277052723 6964 9427 1 2297 14750 1 71 594867034304 38652 39936 46203 4979710666 1 7767 23763 292371 10`‘:` E ‘‘‘‘ ·:-FS:¤!‘¤"Z"·5:i·‘i":·2¥:>¥*¤·2i‘F:"·:·: ‘-:-:¥ =:=:=:2=§¤§:¥=:;:2=3§:=:=:i=S&=R;E=:=:=?=;=;=QZgZ;Z;2;1;Z;2;>Z-2·C%I%`.{·§Z ' Z·2·i·2·Z·2·C·3· .;.;.;,;.3.;.;.;.·2;J;2;Z·i·>';'-}'~.;§g>Z·Z·Z·Z·Z·Z·5 z·:·1·:·c·2·:;:·:E:;:·2-:Rt::~'E·§:EE¢:·:¥:·:¢:¢:¤:¥:¤:¥:i:·simple RFIO program called “grtio" was linked with the Ultra libraries.performed between the same two Silicon Graphics Challenge XL machines through I—HPPI. TheSome measurements were done with a simplitied version of RFIO software. The tests were3. HIPPI RFIO testsOCR OutputHIPPI in the CERN Computer CentreCERN CN Report 96-13 OCR Output 26 August, 1996other direction show very similar results.size of staged files. The transfers were done using a single stream. Once again tests performed in thewriting test between shit"t7 and spO23. The size of the transferred tile was 200MB, which is a typicalOther RFIO tests were performed to verify the buffer size sensitivity. The following is a RFIOIt can be noted that the gateway node supports the load somewhat better.parallel HIPPI streams the CPU consumption on sp023 reaches 90%, thus almost saturating the node.The graphs show clearly that the bottleneck resides in the tape server node. When processing threeFig. 7 Results of memory·to-memory RFIO test over HIPPI to tape serverBuffer size: 128kB#s tr eam s1 2 a0.0SPshi"? sp041-O;-gi #streams2.0%0.04.0%20.0iaol?40.0aol?60.0·¥10.0+80.012.0100.016.0(on trander nodes)(SGI to SP2)CPU consumptionHiPPI trander speeds pure TCP90.047.314.8 52.572.043.3 46.514.240.510.4 28.5 27.3%i&i%S§O23€<i%i"’‘`’ f ‘:`i‘‘spO23. Tests performed in the other direction show very similar results.presented below are a geometrical mean of several measurements of transfers between shift7 andinteresting factor is the CPU consumption when running multiple concurrent streams. The results128kB - the standard RFIO buffer size. Since both the SP/2 nodes are single CPU machines anqueue-size on spO23 is set to 200kB. The amount of transferred data is 512MB. The buffer size isThe t`1rst test to bc performed was a pure TCP writing test between shift7 and sp023. The receiveFig. 6 Setup of mem0ry·t0-memory RFIO test over HIPPI to tape server.Tape SewerHiPPI gatewaysea chan xu. Gnge BM,m '.BM;id "° 6·‘$3;°$?‘IntemalHi PPI IBM SP/2 {HIPPI inthe CERN Computer Centre26 August, 1996 OCR Output CERN CN Report 96-13is a summary of the results:performance could also be improved by increasing the "'I`ape blocksize" from 32kB to 128kB. Belowroutes we could either use FDDI or the combination of HIPPI and the IBM SP/2 intemal switch. Thean SGI Challenge XZLIS (shift2), which has both HIPPI and FDDI attachments. By switching theIBM SP/2 nodes attached to both the intemal switch and FDDI. The disk server used in the tests wastests were done writing data to NTP tapes and then reading it back again. The NTP tape servers aresoftware show that the combination of HIPPI and IBM SP/2 internal switch outperforms FDDI. TheseThe correction of all these problems is not yet finished, but some tests performed with improved200ms for small messages such as RFIO ACK’s)(while trying to increase the amount of data sent inside large MTU packets, it introduces a latency ofproblems caused by the "Nagle Algorithm" in TCP/IP(128kB gives very bad performance when reading, 1MB fixes the problem)• problems in the RFIO software caused by too small client buffer size for the switch(too small gives bad performance, too big causes crashes because of MBUF exhaustion)problems with the choice of the IP window size(nodes with different AIX versions could not communicate through the switch)problems during transition from AIX version 3 to version 4physical problems with the switch itselfthat the problems were caused by:got strange results that we could not explain. Later with the help of IBM engineers we have found outFor a time the switch was ridden by many problems. During tests performed on the switch we haveA11 the nodes in the IBM SP/2 machine are connected using a fast, proprietary, internal switch.5. IBM SP/2 internal switch testsconsumption on the involved nodes does not change greatly with changing buffer size.This test confirms again the typical RFIO buffer size sensitivity. It can also be noted that CPUFig. 8 Results of RFIO test over HIPPI to tape sewer with changing buffer sizeBuffer slze1024 512 256 128 Buffer size5.65.8 128_ 1024 512 2566.0shift?6.2spO236.4106.6U ·g 15 ¤`6»87.0727.4(on transfer nodes)(reading SGI->SP2)CPU consumptionHiPPl transfer speeds RFIO23.26.52 16.212823.11 6.6256 6.3923.116.86.7451226.220.57.601024·===·:‘= tm ‘‘=‘HIPPI inthe CERN Computer CentreCERN CN Report 96-13 26 August, 1996 OCR OutputBelow is the logical design of the interconnect (Figure ll).GigaRouter arld directly to FDDI clients via the standard connection (the CORE DEC GigaSwitch).This was taken into account by providing the DLT Tape Servers with dual access: to HIPPI via thecourse other client machines without I·HPPI interfaces must also be able to access the DLT Servers.FDDI bandwidth bottlenecks are created even when all DLT tape drives on all Servers are active. OfFDDI rings, which are subnets of the main Class C CORE FDDI network. The design ensures that noconnected only via FDDI) through HIPPI. The DLT Servers are distributed evenly over 4 independentthe SGI Challenge and DEC TurboLaser to access a set of about 16 DLT Tape Servers (themselvesThis idea is currently used in the CERN Computer Centre to allow its most powerful machines likeCORE.concurrent streams to FDDI attached machines than would be possible via its direct FDDI interface tobandwidth than FDDI (80OMb/s vs 100Mb/s) a multiprocessor machine can thus run many morethat only have interfaces on FDDI. Since a IHPPI interface allows an eight times higher totalmachines like the SGI Challenges to access through HIPPI a large set of service machines in COREThe idea behind the use of this first CERN GigaRouter was to allow powerful multiprocessorFig.1O HIPPI setup in the CERN Computer Centre with addition of a GigaRouterSG" C'“"€“9° XL sel challenge xt IBM sl¤12cemsp2shift8Shit?GigaRouter'It_ Serial HiPPl to CERN North AreaQ i,.mm¥§.,. ____ _eiFDDI ringsswitcha enge a enge Gl o SGI Ch II XLIS SGI Ch Il XL M `k CS2cemcs2shitt3shitt2connection onto multiple FDDI rings.non-blocking, media independent intemal crossbar switch which allows to route for example a I-HPPIThe GigaRouter can route between different media, including FDDI and IHPPI. It uses a high speed,Later on in the IHPPI project at CERN a new device was introduced, namely a Netstar GigaRouter.6. Introduction of a GigaRouterFig.9 NTP test results3.3M B/sHIPPI & SP/2 switch 128kB128kB 4.6MB/sHIPPI & SP/2 switchHIPPI & SP/2 switch 32kB 2.9M B/s2.slvl B/SsakeFDDI‘‘‘‘‘‘‘’HIPPI in the CERN Computer Centre26 August, 1996 CERN CN Report 96-13 OCR Outputrates between 5 and 10 MB/s being produced for extended periods.PI interfaces. The NA48 experiment is already using this link in production mode, with real datathroughput is expected to reach 40 MB/s when firmware improvements are made to the GigaRouterimplementation. A fourth FEW and 2 more CS2 nodes will soon be connected, and the link4KB packets/s, which is known to be the current limit for the current GigaR0uter HIPPITCP streams distributed among 3 FEWs and 2 CS2 nodes. This corresponds to approximately 5000Initial tests of the link throughput have shown that 19 MB/s can be achieved today using 4 parallelthe CS2). This setup is described in the figure below (Figure 12).necesssary data rate, so the FDDI solution is used, connecting to 4 independent FDDI interfaces onconsiderably less in 1996. (Currently the CS2’s HIPPI performance is not adequate to handle thef`1rst GigaRouter for this purpose. The required sustained data rate in 1997 is 20 MB/s, butmachines) either by HIPPI or via another set of 4 dedicated FDDI rings that have been added to theconnection, over 10 km long, to the Computer Center where it can enter the CS2 (or other COREparallel FDDI rings to a second CERN Gigakouter. From there it is sent down the serial HIPPIgenerated on the NA48 FDDI-attached "Front-End Workstations (FEWs)" is transferred throughareas to the Meiko CS2 machine in the Computer Centre for processing and archiving. Datalink allows NA48 physicists to transfer large amounts of on-line data from their experimental and testover a single pair of monomode fibres). As a leading part of the Central Data Recording project, thisRecently a connection to the CERN North Area has been achieved using serial HIPPI (running7. HIPPI connection to the CERN North Areamachines in the case HIPPI or the GigaRouter is unavailable.directly via FDDI (GigaSwitch). This arrangement gives us a fallback solution for these powerfulto CORE FDDI, and can thus access the DLT Servers either through HIPPI/FDDI (Gigakouter) orNote that (although not shown in Hg.ll) the HIPP1·attached machines are also connected directlyRg.11 Logical interconnect for DLR access through HIPPI and FDDI.’Et t I.,_._rTu rbo Lase rRouter Servers SwitchGiga Q; Giqa(~1O rings)CORE FDDIHIPPI inthe CERN Computer CentreCERN CN Report 96-13 26 August, 1996 OCR Outputthe kleiko CS2.ent tronments. Finally we thank Bernd Panzer (CN-PDP) for help in mnning tests between NA48 andtheir help in understanding and overcoming RFIO problems encountered in the HIPPI and SP2 switchltr km serial link to the North Area. We also thank Felix Hassine (CiN—PDP) and Tim Bell (IBM) for·(`lLi<N-ECP) in setting up the HIPPI infrastructure, both in the CERN Computer Centre and for theThe authors wish to acknowledge the essential contributions of Arie van Praag and Patrick Donnat9. AcknowledgementsCt lltli network media, making it a full member of CORE’s "Hybrid Backbone Network".(`entre. expanding its network capabilities. This allows us to integrate HIPPI tightly with the other.-\re.t through serial HIPPI, have been introduced into the I-IIPPI project in the CERN ComputerNew developments, like the addition of two GigaRouters and a 10km link to the CERN Northe~tat~hshed.hrnitttnon of the tape server node and an upper limit of three concurrent transfer streams has beennet·.t ark can be performed, achieving good transfer rates. The bottleneck has been found to be CPUl·urthermore, it has been demonstrated that tape transfers using I-IIPPI and the IBM SP/2 internalNeu ertheless, the RFIO protocol is currently being modified to mimimise its effects.used; fortunately, at the buffer size at which RFIO normally operates, this is not extremely serious.to be an unfortunate interaction of the Nagle algorithm and the RFIO protocol when ACK packets aretnotltfications to the software. A buffer size sensitivity of RFIO has however been discovered, believedlt has also been shown that acceptable speeds can be obtained over HIPPI using RFIO, without anythan the highest transfer rate achieved over UltraNet.only 8.5MB/s lower than the nominal maximum transfer speed for the medium, and by far highertransfer rate of 91.5MB/s can be achieved over HIPPI between SGI hosts using pure TCP, which isThe I-IIPPI tests described in this paper have been a success. It has been shown that an aggregate8. ConclusionsFig.12 Connection to CERN North Area through serial HIPPI.’$5é=S;i;£°°‘.,.r,.r .,r...,....._r..r;...e,;.....;.,r,.r.... . r,.,...;_4 . .r...,...,..,.,.r...r,:...r . .r.e...r..r.......r.r.......r..,*HIPPI in the CERN Computer Centre

HIPPI in the CERN computer centre

Abstract

Similar works

Full text

Available Versions

CERN Document Server