SPREAD: Scalable Platform for Reliable and Efficient Automated Distribution

Abstract

We introduce SPREAD -- a new architecture for distributing and maintaining up-to-date Web content that simultaneously employs three different mechanisms: client validation, server invalidation, and replication. Proxies within SPREAD self-configure themselves to form scalable distribution hierarchies that connect the origin servers of content providers to clients. Each proxy autonomously decides on the best mechanism based on the object’s popularity and modification rates. Requests and subscriptions propagate from edge proxies to the origin server through a chain of intermediate proxies. Invalidations and replications travel in the opposite direction. SPREAD’s network of proxies automatically reconfigure when proxies go down or come up, or when new ones are added. The ability to spontaneously form hierarchies is based on a modified Transparent Proxying mechanism, called Translucent Proxying, that sanitizes Transparent Proxying. It allows proxies to be placed in an ad-hoc fashion anywhere in the network- not just at focal points within the network that are guaranteed to see all the packets of a TCP connection. In this paper we (1) describe the architecture of SPREAD, (2) discuss how proxies determine which mechanism to use based on local observations, and (3) use a trace-driven simulation to test SPREAD’s behavior in a realistic setting