Crux: Locality-Preserving Distributed Services

Distributed systems achieve scalability by distributing load across many machines, but wide-area deployments can introduce worst-case response latencies proportional to the network's diameter. Crux is a general framework to build locality-preserving distributed systems, by transforming an existing scalable distributed algorithm A into a new locality-preserving algorithm ALP, which guarantees for any two clients u and v interacting via ALP that their interactions exhibit worst-case response latencies proportional to the network latency between u and v. Crux builds on compact-routing theory, but generalizes these techniques beyond routing applications. Crux provides weak and strong consistency flavors, and shows latency improvements for localized interactions in both cases, specifically up to several orders of magnitude for weakly-consistent Crux (from roughly 900ms to 1ms). We deployed on PlanetLab locality-preserving versions of a Memcached distributed cache, a Bamboo distributed hash table, and a Redis publish/subscribe. Our results indicate that Crux is effective and applicable to a variety of existing distributed algorithms.Comment: 11 figure

WebWave: Globally Load Balanced Fully Distributed Caching of Hot Published Documents

Document publication service over such a large network as the Internet challenges us to harness available server and network resources to meet fast growing demand. In this paper, we show that large-scale dynamic caching can be employed to globally minimize server idle time, and hence maximize the aggregate server throughput of the whole service. To be efficient, scalable and robust, a successful caching mechanism must have three properties: (1) maximize the global throughput of the system, (2) find cache copies without recourse to a directory service, or to a discovery protocol, and (3) be completely distributed in the sense of operating only on the basis of local information. In this paper, we develop a precise definition, which we call tree load-balance (TLB), of what it means for a mechanism to satisfy these three goals. We present an algorithm that computes TLB off-line, and a distributed protocol that induces a load distribution that converges quickly to a TLB one. Both algorithms place cache copies of immutable documents, on the routing tree that connects the cached document's home server to its clients, thus enabling requests to stumble on cache copies en route to the home server.Harvard University; The Saudi Cultural Mission to the U.S.A

Object Distribution Networks for World-wide Document Circulation

This paper presents an Object Distribution System (ODS), a distributed system inspired by the ultra-large scale distribution models used in everyday life (e.g. food or newspapers distribution chains). Beyond traditional mechanisms of approaching information to readers (e.g. caching and mirroring), this system enables the publication, classification and subscription to volumes of objects (e.g. documents, events). Authors submit their contents to publication agents. Classification authorities provide classification schemes to classify objects. Readers subscribe to topics or authors, and retrieve contents from their local delivery agent (like a kiosk or library, with local copies of objects). Object distribution is an independent process where objects circulate asynchronously among distribution agents. ODS is designed to perform specially well in an increasingly populated, widespread and complex Internet jungle, using weak consistency replication by object distribution, asynchronous replication, and local access to objects by clients. ODS is based on two independent virtual networks, one dedicated to the distribution (replication) of objects and the other to calculate optimised distribution chains to be applied by the first network

A Scalable Implementation for Human-Friendly URIs

Abstract. In the Web, Uniform Resource Identifiers (URIs) are used to name resources. The most common form of URI, the Uniform Resource Locator (URL) has, unfortunately, some scalability problems. In this paper, we proposes the use of Human-Friendly Names (HFNs) to solve these scalability problems. HFNs are high-level names that allow (human) users to easily deal with names. We also describes a scalable HFN-to-URL resolution mechanism. This mechanism is based on the existing Domain Name System (DNS) and the Globe Location Service. To gain experience and validate our ideas, we have implemented our HFN resolution scheme

Three applications for mobile epidemic algorithms

This paper presents a framework for the pervasive sharing of data using wireless networks. 'FarCry' uses the mobility of users to carry files between separated networks. Through a mix of ad-hoc and infrastructure-based wireless networking, files are transferred between users without their direct involvement. As users move to different locations, files are then transmitted on to other users, spreading and sharing information. We examine three applications of this framework. Each of these exploits the physically proximate nature of social gatherings. As people group together in, for example, business meetings and cafés, this can be taken as an indication of similar interests, e.g. in the same presentation or in a type of music. MediaNet affords sharing of media files between strangers or friends, MeetingNet shares business documents in meetings, and NewsNet shares RSS feeds between mobile users. NewsNet also develops the use of pre-emptive caching: collecting information from others not for oneself, but for the predicted later sharing with others. We offer observations on developing this system for a mobile, multi-user, multi-device environment

Khazana An infrastructure for building distributed services

technical reportEssentially all distributed systems?? applications?? and services at some level boil down to the problem of man aging distributed shared state Unfortunately?? while the problem of managing distributed shared state is shared by many applications?? there is no common means of managing the data every application devises its own solution We have developed Khazana?? a distributed service exporting the abstraction of a distributed per sistent globally shared store that applications can use to store their shared state Khazana is responsible for performing many of the common operations needed by distributed applications?? including replication?? consis tency management?? fault recovery?? access control?? and location management Using Khazana as a form of middleware?? distributed applications can be quickly de veloped from corresponding uniprocessor applications through the insertion of Khazana data access and syn chronization operation

Replication Control in Distributed File Systems

We present a replication control protocol for distributed file systems that can guarantee strict consistency or sequential consistency while imposing no performance overhead for normal reads. The protocol uses a primary-copy scheme with server redirection when concurrent writes occur. It tolerates any number of component omission and performance failures, even when these lead to network partition. Failure detection and recovery are driven by client accesses. No heartbeat messages or expensive group communication services are required. We have implemented the protocol in NFSv4, the emerging Internet standard for distributed filing.http://deepblue.lib.umich.edu/bitstream/2027.42/107880/1/citi-tr-04-1.pd

Khazana an infrastructure for building distributed services

technical reportEssentially all distributed systems, applications and service at some level boil down to the problem of managing distributed shared state. Unfortunately, while the problem of managing distributed shared state is shared by man applications, there is no common means of managing the data - every application devises its own solution. We have developed Khazana, a distributed service exporting the abstraction of a distributed persistent globally hared store that applications can use to store their shared state. Khazana is responsible for performing many of the common operations needed by distributed applications, including replication, consistency management, fault recovery, access control, and location management. Using Khazana as a form of middleware, distributed applications can be quickly developed from corresponding uniprocessor applications through the insertion of Khazana data access and synchronization operations