Cavin, David

Pedone, Fernando

Schiper, André

Urbán, Péter

Infoscience - École polytechnique fédérale de Lausanne

Weak Ordering Oracles for Failure Detection-Free SystemsFernando PedoneyAndre SchiperzPeter UrbanzDavid CavinzyHewlett-Packard Laboratories | Software Technology LaboratoryPalo Alto, CA 94304, USAfernando pedone@hp.comzEcole Polytechnique Federale de Lausanne (EPFL) | Faculte I&CCH-1015 Lausanne, Switzerlandfandre.schiper, peter.urban, david.caving@ep.ch1 MotivationAgreement abstractions, such as consensus, atomicbroadcast, and generic broadcast [2], are importantbuilding blocks in distributed systems subject toprocessor failures. Atomic broadcast, for example,has been used to build many fault-tolerant systems,such as highly-available databases. Atomic broad-cast guarantees that if a message is broadcast to agroup of processors and one of these processors de-livers the message, then all processors also deliverthe message | a property known as agreement ; andif two processors deliver the same two messages,they do so in the same order | a property knownas total order.Asynchronous systems. A protocol implemen-tation is as general as the underlying assumptionsabout its system model. In this sense, the asyn-chronous model of computation is a very generalway to formalize distributed systems. It basicallymakes no assumptions about the time it takes totransmit messages over network links and for pro-cessors to perform their computations. The asyn-chronous model is very appealing as any solutionthat can be applied in it can also be applied in astronger model (e.g., the synchronous model, whereprocessors can estimate the time it takes for otherprocessors to execute their programs).Nevertheless, one fundamental result about dis-tributed systems subject to processor failures statesthat atomic broadcast cannot be solved in a pureasynchronous model. The basic reason stems fromthe fact that in pure asynchronous systems, proces-Research supported by the Swiss National Science Foun-dation NCCR MICS project and the CSEM Swiss Center forElectronics and Microtechnology, Inc., Neucha^tel.sors cannot tell a very slow processor (i.e., a pro-cessor that takes a long time to reply to the others)from a processor that has failed.To circumvent this impossibility result, currentproposals have extended the asynchronous modelwith further assumptions, such as failure detectors.Failure detectors. Failure detectors provideprocessors with information about processors fail-ures [1]. Failure detectors can make mistakes, thatis, they can suspect processors that have not failedand not suspect processors that have failed. Severalclasses of failure detectors have been proposed [1].Failure detectors are usually implemented withsome timeout mechanism and even the weakest fail-ure detector requires some timeout calibration tosatisfy its properties. Moreover, even though pro-tocols based on failure detectors can cope withwrong processor suspicions, their performance ishurt when suspicions are wrong. But more impor-tantly, failure detectors have to deal with the fol-lowing dilemma:the failure detector dilemma. On theone hand, timeout values should be large inorder to maximize performance (by avoidingwrong suspicions). On the other hand, timeoutvalues should be small in order to ensure a fastreaction to failures.2 Weak Ordering OraclesInstead of failure detectors, we propose to extendthe asynchronous system with weak ordering ora-cles. Weak ordering oracles capture the behaviorof network multicast in state-of-the-art local-areanetworks: if processors send multicast messages inc2002 IEEE. Appeared in Proc. of the 2002 International Conference on Dependable Systems and Networks (DSN),supplemental volume, pages B-32 and B-33.a local-area network, there is a good chance thatsome of the messages will be received by all pro-cessors in the same order. To illustrate this fact,Figure 1 shows the results of measurements per-formed in a cluster of 12 PCs connected by a 100Mbit/s Ethernet. Each host broadcasts messages toall other hosts using IP multicast. When messagesare broadcast with a period greater than 0.14 mil-liseconds, very few messages are received out of or-der (only about 5%).Message interval (ms)Outofordermessages(%)0.160.140.120.10.080.060.040.021009080706050403020100Figure 1: Spontaneous total order propertyBriey (see [3] for more details), our oracle is de-ned by the primitives W-ABroadcast(r;m) (queryto broadcast message m) and W-ADeliver(r;m)(delivery of m). The integer parameter r groupsmessages with the same r value. The weak order-ing property holds for r if there exists a message msuch that each process delivers W-ADeliver(r;m)rst (among all W-ADeliver(r; )).To illustrate this property, consider three pro-cesses, p1, p2, and p3, executing the followingqueries to the oracle (for brevity, we denote nextW-ABroadcast(r;m) by B(r;m)): p1executes B(0;m1); B(1;m2); B(2;m3), p2executes B(0;m4); B(1;m5); B(2;m6), p3executes B(0;m7); B(1;m8); B(2;m9),and assume the following sequences ofW-ADeliver(r;m) (for brevity, we denote nextW-ADeliver(r;m) by D(r;m)): on p1: D(0;m1); D(1;m2); D(0;m4); D(2;m3);D(0;m7); etc. on p2: D(0;m4); D(0;m1); D(1;m5); D(0;m7);D(2;m3); etc. on p3: D(0;m4); D(0;m7); D(2;m3); D(1;m8);etc.The weak ordering property holds for r = 2 (m3isthe rst message with r = 2 delivered by p1, p2, andp3), but does not hold for either r = 0 or r = 1.We dene the Weak Atomic Broadcast (WAB)oracle as the oracle that fullls the weak orderingproperty innitely often.Consensus and atomic broadcast can be solvedusing the WAB oracle [3]. Actually consensus canbe solved with an oracle weaker than the WAB or-acle [3]. When using weak ordering oracles, atomicbroadcast algorithms and other agreement-relatedalgorithms do not suer from the failure detectordilemma. Since they do not rely on failure de-tectors, there is no tuning of timeouts involved inthe execution of an atomic broadcast. As a result,there is no notion of reaction time to failures withthese protocols. The performance of algorithms us-ing weak ordering oracles is as good in the presenceof failures as in the absence of failures.3 ExperimentsWe have implemented and conducted several ex-periments using our atomic broadcast algorithm.In the setup used to obtain Figure 1, and for arange between 10 and 250 messages broadcast persecond, we have found that our atomic broadcastimplementation using weak ordering oracles out-performs failure detector-based implementations ofatomic broadcast using a timeout of 4 millisecondson approximately 30%. For timeout values of 10milliseconds failure detector-based implementationsoutperform weak ordering oracle based implementa-tions by less than 30%. We believe that the perfor-mances of our weak ordering oracle implementationmay further be improved.AcknowledgmentsWe thank Matthias Wiesmann for providing us withFigure 1.References[1] T. D. Chandra and S. Toueg. Unreliable failure detec-tors for reliable distributed systems. Journal of ACM,43(2):225{267, 1996.[2] F. Pedone and A. Schiper. Handling message semanticswith generic broadcast protocols. Distributed Comput-ing, 15(2):97{107, 2002.[3] F. Pedone, A. Schiper, P. Urban, and D. Cavin. Solvingagreement problems with weak ordering oracles. Tech-nical Report HPL-2002-44, Hewlett Packard Laborato-ries, 2002. Also appears as EPFL Technical ReportIC/2002/010, March 2002.B-33

Weak Ordering Oracles for Failure Detection-Free Systems

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne