魅力ある大学院教育イニシアティブ：実践IT力を備えた高度情報学人材育成プログラ

三浦 信一

岡本 高幸

魅力ある大学院教育イニシアティブ：実践IT力を備えた高度情報学人材育成プログラムtechnical repor

Tsukuba Repository  (Tulips-R)

高性能クラスタにおけるEthernetを利用した高性能マルチパス・マルチリンクネットワークシステムの開発著者 三浦 信一, 岡本 高幸雑誌名 2006年度CSテクニカルレポート・システム開発型研究プロジェクト特集号発行年 2006その他のタイトル A High-performance Network System withMulti-Path / Multi-Link Ethernet forHigh-Performance PC ClustersURL http://hdl.handle.net/2241/104471Ethernet† †EthernetPC HPCRI2N/UDPVFREC-NetEthernet2 1 LinuxRI2N/Drv2A High-performance Network Systemwith Multi-Path / Multi-Link Ethernet for High-Performance PC ClustersSHIN’ICHI MIURA † and TAKAYUKI OKAMOTO †Ethernet is the most popular interconnection network to be used on various PC clusters.Especially, Gigabit Ethernet provides a very high cost/performance ratio as the most usedcommodity network. However, it is difficult to utilize it on a large scale HPC clusters becauseits absolute bandwidth and scalability is limited. To solve this problem, we have been de-veloping both RI2N/UDP and VFREC-Net systems for wide bandwidth, high scalability andhigh dependability for large scale HPC clusters. In this research we implemented a specialdevice driver on Linux system, named RI2N/Drv. Through the performance evaluation, weconfirmed that the new implementation provides both higher bandwidth and low latency thanthe old systems.1.PC PCHPCHPC PCHPC†Graduate School of Systems and Information Engineering, Univer-sity of TsukubaEthernetTreeMyrinet1) InfiniBand2) SAN (System AreaNetwork) HPCSAN Eth-ernetEthernetRI2N3)VFREC-Net4)22 1PC2RI2N/Drv4 5RI2N/Drv672.RI2N3)VFREC-Net4)23),4)2.1 RI2NRI2N (Redundant Interconnection with InexpensiveNetwork)RI2N( 1)node A node B node Cswitch 1 switch 2 switch 3NIC1 RI2N/UDPuser threadAPI librarynetworkNICNIC NICNICmulti-linkUDP socketRI2N thread2 RI2N/UDPRI2N RI2N/UDP3)RI2N/UDP 2 RI2N/UDPsocketAPI RI2N/UDPTCP/IPUDP/IP UDP/IPRI2N/UDPUDP/IPTCP/IPRI2N/UDPTCP/IP socket APITCP/IP Open MPI5) PCMPI2.2 VFREC-Net--Gigabit Ethernet GbE- GbE10GbEnode-A node-B node-C node-DVLAN ID = 2VLAN ID = 13 VLANGbEGbE 10GbELayer-2( L2)L2IEEE 802.3ad6)PCTreetagged-VLAN7)VLAN IDEthernetVLAN8) VLAN3 VLANVLANEthernetVLANEthernet HPCEthernetVLANPCVFREC-Net (VLAN-based Flexible,Reliable and Expandable Commodity Network)4)VFREC-Net EthernetTCP/IPUDP/IPSW SW SW SWSW SW SW SWP P P P P P P P P P P P P P P P4 VLAN Based Fat TreeVFREC-NetSANEthernet4 Fat TreeEthernet ( SANVBFT (VLANBased Fat Tree)8) )2.3RI2N/UDPVFREC-NetRI2N/UDP -VFREC-Net -2RI2N/UDP UDP/IPVFREC-Net2VFREC-NetRI2N/UDP29)RI2N/UDP•• socketThreadVFREC-Net機能で 実現されたネットワーク(1)VFREC-Net機能で 実現されたネットワーク(2)対称なネットワークRI2N機能を用いて 複数のリンクを 同時に利用5VFREC-Net RI2N/UDP EthernetRI2N/UDP2RI2N/UDP3.VFREC-Net RI2N/UDP5RI2N/Drv RI2N/DrvRI2N VFREC-NetVFREC-NetVBFT( 4)RI2N3.1RI2N/UDPAPIRI2N/DrvOSsocket APIアプリケーションUDP TCPIPR-NIC P-NICPCL2 スイッチVLANの設定L2 スイッチVLANの設定実装部分6VFREC-NetTCP/IPRI2N/Drv VFREC-NetRI2N VFREC-NetTCP/IPLinux EthernetTCP/IP RI2N/Drv3.26OSIIPEthernetOS( P-NIC ) RI2N/DrvEthernet P-NIC IPP-NIC Ethernet( R-NIC )3.2.121 1IPP-NICPC・IPアドレスから使用するNICをP-NICに決定・R-NICをround-robinで選択→ノード間バンド幅の向上R-NICR-NICPCIPP-NICR-NICR-NIC独立したネットワーク独立したネットワーク7n R-NIC Ethernet( 7) n R-NICEthernetEthernet RI2N/DrvP-NIC(e.g. IP)Ethernet MTUR-NICn Ethernet() nEthernetR-NICR-NICEthernetR-NICEthernetR-NIC Eth-ernetEthernet (e.g.IP) (e.g. TCP) EthernetL2 スイッチVLANの設定L2 スイッチVLANの設定L2 スイッチVLANの設定L2 スイッチVLANの設定IPP-NICPC・IPアドレスから使用するNICをP-NICに決定・あて先にあわせてVIDを付与→スイッチ間経路を指定するIPP-NICPCR-NICIPP-NICPCR-NICIPP-NICPCR-NICスイッチ間リンクの競合を避けることができる→バイセクションバンド幅の向上8TCPTCP RI2N/DrvVFREC-Net ( 8)VFREC-Net(2.2 )4)3.2.2RI2N/Drv R-NICTCPEthernetTCPTCP WindowR-NICR-NIC9 2P-NICPCR-NICR-NICPCアプリケーションP-NICR-NICR-NICアプリケーション他のR-NICに比べて到着パケット数が極端に少なくなる→故障と判断通信時（できるだけ早く故障を検出したい）P-NICPCR-NICR-NICPCアプリケーションP-NICR-NICR-NICアプリケーションハートビートが長時間到着しない→故障と判断無通信時（ゆっくりでも確実に検出したい）デーモンデーモン9R-NICR-NICR-NIC EthernetR-NICEthernetR-NICEthernet(e.g.)R-NIC1CPU Intel Xeon 3 0GHz 1-way (Hyper Thread ON)Memory 1 0 Gbytes DDR2/400MHz (Dual Channel)NIC Intel PRO/1000MT Dual Ports (MTU=8000)PCI-X 64bit/133MHzOS Linux Kernel 2 6 19Compiler GCC ver 4 12Normal Ethernet a – b 121 Mbyte/seca – e 121 Mbyte/secRI2N/UDP a – b 203 Mbyte/seca – e 203 Mbyte/secRI2N/Drv a – b 246 Mbyte/seca – e 246 Mbyte/secRI2N/Drv3.34.RI2N/Drv1 8RI2N/DrvRI2N/DrvEthernet TCP/IPRI2N/UDP104.1(a – b)Ethernet 2(a – e)2Normal Ethernet 1(a) Normal Ethernetで用いるネットワーク1 LinkSwitch Aa b c d e f g hSwitch B(c) RI2N/Drvで用いるネットワーク4 Links a b c d e f g hSwitch B4 Links Switch DSwitch ASwitch C(b) RI2N/UDPで用いるネットワーク1 LinkSwitch Aa b c d e f g hSwitch B1 LinkSwitch C Switch D10Gigabit Ethernet(125Mbyte/sec)RI2N/UDPRI2N/Drv250Mbyte/secRI2N/UDP203Mbyte/sec RI2N/UDP UDP/IP245Mbyte/sec RI2N/UDPRI2N/DrvRI2N/UDP246Mbyte/sec250Mbyte/secRI2N/UDPOSTCP/IPRI2N/DrvRI2N/UDP4.2211, 1220 KbyteRI2N/Drv MTU(8000 byte) EthernetMTU1 Ethernet2 R-NIC 0 100 200 300 400 500 600 700 800 0  10  20  30  40  50  60latency [microsecond]message size [Kbyte]RI2N/DrvEthernetRI2N/UDP11 (a – b) 0 100 200 300 400 500 600 700 800 0  10  20  30  40  50  60latency [microsecond]message size [Kbyte]RI2N/DrvEthernetRI2N/UDP12 (a – e)RI2N/UDPEthernet30–50 µsecMTURI2N/Drv 2 R-NICEthernetRI2N/UDP MTU 2Ethernet20 KbyteNIC2102104106NIC1101103105107OK10099ACKACKACK一度に読めるだけ読むACKACKACKそのためどちらを先に読んでも3つのACKが返る→(無駄な)再送開始13 TCP RI2N/DrvRI2N/DrvRI2N/Drv5. TCPRI2N/DrvRI2N/DrvTCP13R-NIC NIC1 NIC2TCP 100101 NIC1NIC OSEthernet103,105,107NIC1 Linux TCP3 Ethernet103,105,107TCPNIC2 102,104,106 EthernetOSACK103,105,107 31023 ACKACK TCP 3ACK 0 100 200 300 400 500 600 700 800 0  10  20  30  40  50  60latency [microsecond]message size [Kbyte]RI2N/DrvEthernetRI2N/UDP14 ( a – b ) MTU=1500Ethernet MTU(MTU)Ethernet MTU( 14)TCPAck3 6R-NICEthernet 1 R-NICR-NICEthernet6.RI2N/DrvTCPTCPRI2N/Drv5RI2N/Drv7.EthernetIEEE 802.3ad6) OSIEEE 802.3ad HPCHPC IEEE 802.3adHPCPM/Ethernet10) PM/Ethernet Eth-ernetPM/EthernetNICAPI TCP/IPPM socket APIHPC MPI Open-MPI5) OpenMPIMPITCP/IP8.RI2N/UDP VFREC-NetPCRI2N/Drv2246 Mbyte/secTCP/IPTCP/IPRI2N/DrvEthernetPCIT1) Myricom, inc.: Myrinet.http://www myri.com/.2) InfiniBand Trade Association: InfiniBand,http://www.infinibandta.org/.3) , , , ,: Ethernet PCRI2N/UDP,(HPCS2007), , pp. 41–48 (2007).4) , , , , :VFREC-Net VLAN,, Vol. 47, pp. 35–45(2006).5) Gabriel, E., Fagg, G. E., Bosilca, G., Angskun, T.,Dongarra, J. J., Squyres, J. M., Sahay, V., Kambadur,P., Barrett, B., Lumsdaine, A., Castain, R.H., Daniel,D. J., Graham, R. L. and Woodall, T. S.: Open MPI:Goals, Concept, and Design of a Next GenerationMPI Implementation, Proceedings, 11th EuropeanPVM/MPI Users’ Group Meeting, Budapest, Hun-gary, pp. 97–104 (2004).6) IEEE: P802.3ad Link Aggregation Task Force.http://grouper.ieee.org/groups/802/3/ad/.7) IEEE: 802.1Q - Virtual LANs.http://www.ieee802.org/1/pages/802.1Q.html.8) , , , ,, : VLANL2 Ethernet ,, Vol. 45,No. SIG06(ACS6), pp. 35–44 (2004).9) , , , , :tagged-VLAN PC,, , pp. 25–30 (2006).10) Sumimoto, S. and Kumon, K.: PM/Ehernet-kRMA:A High Performance Remote Memory Access Facil-ity Using Multiple Gigabit Ethernet Cards, 3rd IEEEInternational Symposium on Cluster Computing andthe Grid (CCGrid 2003), IEEE Computer Society,pp. 326–333 (2003).

A High-performance Network System with Multi-Path / Multi-Link Ethernet for High-Performance PC Clusters

Abstract

Similar works

Full text

Available Versions

Tsukuba Repository (Tulips-R)