International audienceWe present a new communication subsystem for high speed networks featuring an extendable packet optimization engine mixing several communication flows. Optimizations are parameterized by the capabilities of the underlying network drivers, and are triggered by the network cards when they become idle. The database of predefined strategies can be easily extended

Aumage, Olivier

Brunet, Elisabeth

Namyst, Raymond

INRIA a CCSD electronic archive server

HAL Id: inria-00110773https://hal.inria.fr/inria-00110773Submitted on 31 Oct 2006HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Short Paper : Dynamic Optimization of Communicationsover High Speed NetworksElisabeth Brunet, Olivier Aumage, Raymond NamystTo cite this version:Elisabeth Brunet, Olivier Aumage, Raymond Namyst. Short Paper : Dynamic Optimization of Com-munications over High Speed Networks. The 15th IEEE International Symposium on High Perfor-mance Distributed Computing (HPDC-15), Jun 2006, Paris/France, France. ￿inria-00110773￿Short Paper : Dynamic Optimization ofCommunications over High Speed Networks ∗Elisabeth BRUNET Olivier AUMAGEINRIA - LABRI, Université Bordeaux 1,TALENCE, FRANCERaymond NAMYSTAbstractWe present a new communication subsystem for highspeed networks featuring an extendable packet optimizationengine mixing several communication flows. Optimiza-tions are parameterized by the capabilities of the underly-ing network drivers, and are triggered by the network cardswhen they become idle. The database of predefined strategiescan be easily extended.1 IntroductionTo maintain the performance level of today’s highspeed networks (such as QUADRICS, MYRINET or IN-FINIBAND), communication libraries combine a vari-ety of techniques and mechanisms to select how tosend a given packet the best way. These techniquesinclude PIO and DMA transfer modes, eager, rendez-vous and remote memory access protocols, zero-copy vsby-copy packet handling, etc. Libraries also have todecide whether to aggregate a packet with other pack-ets to reduce the number of network requests –at thecost of additional processing– or to send the packetalone to avoid waiting for another one to merge with(or to benefit from a pipeline effect) or even to usea gather/scatter request. All these decisions mustbe consistent with the capabilities of the underly-ing network drivers. The second aspect that makescommunication optimization so important is that to-day’s parallel applications tend to use complex con-glomerates of multiple communication middlewaressuch as CORBA, JAVA RMI or DSM, increasing thenumber of concurrent communication flows betweenprocessing nodes. Nowadays communication hard-ware/software feature NIC virtualization capabilitiesthat provide transparent multiplexing over a singleNIC. It would thus be tempting to simply do a one-to-one mapping of communication flows onto virtu-alized NICS. A much powerful approach, though, is∗This work is supported by the French Ministery of Research andthe Regional Council of Aquitaine.to consider network multiplexing units as networkingresources to be put in common into a pool managedby a software packet scheduler. Then, the one-to-onemapping is now only one mere scheduling policy (thatcould be selected as a fallback, for instance) amongmany other possible ones.In fact, the role of a modern communication libraryis to both optimize and schedule all the data transfers be-tween processes onto the available network resources.2 Towards a Dynamic Message SchedulerIncluding the application activity as one input pa-rameter of the optimization process and aggregatingpackets from multiple flows means introducing non-determinism in the underlying communication flows.By having a global control on the network multiplex-ing resources, a scheduler may assign some of theseresources to different classes of traffic (assigning dif-ferent channel to large synchronous sends, put/gettransfers and control/signalling messages) and helpthe receiver in sorting out the incoming packets. Sucha scheduler may also perform dynamic load balancingon multiple resources, multiple NICs, or even NICsfrom multiple technologies. Finally, the schedulermay also choose to dynamically change the assign-ment of networking resources to traffic classes, thusselecting different policies, as the needs of the appli-cation evolve during the execution.The MADELEINE library [1] was designed to per-form well not only with regular communicationschemes –commonly encountered with MPI-like pro-gramming environments– but also with programmingmodels involving irregular communication schemessuch as RPC or DSM. Other related works [3, 6] havesimilar optimization features. At the time, though,middleware integrators such as PADICO [2] or code-coupling applications were not widely used: thereusually was a single middleware (such as GLOBALARRAYS [5], PM2 [4], or a DSM) between the applica-tion and the network library. Consequently, this previ-ous version of Madeleine was not designed to performcross-flow optimization and its design was limited todeterministic flow manipulations.3 The Madeleine Optimization EngineThe job of the packet scheduler being to reorganizepackets, the more information it is fed with, the betterit can optimize the communication flow. One valuablesource of information is this communication flow it-self: if the scheduler accumulates packets before mak-ing a decision, it decreases the risk to take wrong de-cisions and also widens the possibilities of packet re-ordering. An example of a wrong decision is to send asmall packet just before another small packet becomesavailable to send, incuring two network transactionswhere an aggregated one would have been better. Forthe scheduler to build its own pool of lookahead pack-ets, we propose to synchronize the optimizing processwith the NIC(s) activity instead of the application re-quests. Figure 1 shows this idea: the application sim-ply enqueues packets into a list and immediately re-turns to computing. The scheduler is not activatedeach time the application submits a new packet, butrather when one of the NICs becomes idle. While theNIC is busy sending a packet, the scheduler simplyaccumulates a backlog of packets. If the NIC neverstays busy long enough for packets to accumulate, thescheduler may send packets as they become available,as a regular communication library would do, or mayartificially delay them for a short time to increase thepotential of interesting aggregations (in a TCP NA-GLE’s algorithm fashion).Nevertheless, the very nature of packets encoun-tered in the range of applications targeted by such acommunication library must be taken into account.Those applications are built on top of a stack of mid-dlewares. As result, most communication requestswill come from these middlewares and will containextra control data implementing specific protocols.Far from being simple sequences of bytes, such re-quests are indeed structured messages with one ormore fragments expressing what the message carriesor requests, and one or more other fragments beingthe actual data or request arguments. These messageinternal dependencies are expressed by the applica-tion and middlewares through the Madeleine API pre-sented in [1]. They are taken into account as limitingfactors –or constraints– by the scheduler while estimat-ing the value of a given packet reordering operation.We have implemented this scheduler following the3-layered architecture depicted on Figure 1. The opti-mizing layer in the middle monitors NICs activity andmakes sure that they are kept adequately busy with ad-RDV PIO Put/get RDV PIO Put/get... ......Myrinet NIC Myrinet NIC......Quadrics NICNetwork...TransferLayerApplicationMadeleineLayerLayerWaiting packsOPTIMIZER − SCHEDULERRDV PIO Put/getMad. Driver/ElanOptimizingCollectSenderMadeleine Driver/MXFigure 1. Architecture proposalequately scheduled communication requests.4 Conclusion and Future WorkWe propose a new approach to optimize communi-cations over high speed networks with a new designof MADELEINE. Its main originality is its powerfuland dynamic optimization engine, which is portable,multithreaded, and easily extendable. A beta versionon top of MX/Myrinet (available at http://gforge.inria.fr/projects/pm2) already exhibits significantimprovements over the previous software in manycases. Most noticeably, the aggregation of eager seg-ments collected from several independent communi-cation flows brings huge performance gains.In the near future, we intend to experiment with dif-ferent packet lookahead window sizes. We also planto study how to bound the number of data rearrange-ments the optimizer has to evaluate so as to determinethe best combination of optimization techniques.References[1] O. Aumage, L. Bougé, A. Denis, L. Eyraud, J.-F. Méhaut,G. Mercier, R. Namyst, and L. Prylli. A portable and effi-cient communication library for high-performance clus-ter computing. Cluster Computing, 2002.[2] A. Denis, C. Pérez, and T. Priol. Padicotm: An openintegration framework for communication middlewareand runtimes. Future Generation Computer Systems, 2003.[3] W. Gropp, E. Lusk, N. Doss, and A. Skellum. Ahigh performance, portable implementation of the MPImessage passing interface standard. Technical ReportANL/MCS-TM-213, Argonne Nat. Lab., 1996.[4] R. Namyst and J.-F. Méhaut. PM2: Parallel Multi-threaded Machine; A computing environment for dis-tributed architectures. In Parallel Computing, 1995.[5] J. Nieplocha, R. Harrison, and R. Littlefield. Global ar-rays: a portable ”shared-memory” programming modelfor distributed memory computers. In SC’94.[6] T. Takahashi, S. Sumimoto, A. Hori, H. Harada, andY. Ishikawa. PM2: High performance communicationmiddleware for heterogeneous network environments.In SC’00.

Short Paper : Dynamic Optimization of Communications over High Speed Networks

We present a new communication subsystem for high speed networks featuring an extendable packet optimization engine mixing several communication flows. Optimizations are parameterized by the capabilities of the underlying network drivers, and are triggered by the network cards when they become idle. The database of predefined strategies can be easily extended

BRUNET, Elisabeth

AUMAGE, Olivier

NAMYST, Raymond

Oskar Bordeaux

We present a new communication subsystem for high speed networks featuring an extendable packet optimization engine mixing several communication flows. Optimiza-tions are parameterized by the capabilities of the underly-ing network drivers, and are triggered by the network cards when they become idle. The database of predefined strategies can be easily extended. 

Short Paper : Dynamic Optimization of Communications over High Speed Networks

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

Oskar Bordeaux

CiteSeerX