2,481 research outputs found
The MDS Queue: Analysing the Latency Performance of Erasure Codes
In order to scale economically, data centers are increasingly evolving their
data storage methods from the use of simple data replication to the use of more
powerful erasure codes, which provide the same level of reliability as
replication but at a significantly lower storage cost. In particular, it is
well known that Maximum-Distance-Separable (MDS) codes, such as Reed-Solomon
codes, provide the maximum storage efficiency. While the use of codes for
providing improved reliability in archival storage systems, where the data is
less frequently accessed (or so-called "cold data"), is well understood, the
role of codes in the storage of more frequently accessed and active "hot data",
where latency is the key metric, is less clear.
In this paper, we study data storage systems based on MDS codes through the
lens of queueing theory, and term this the "MDS queue." We analytically
characterize the (average) latency performance of MDS queues, for which we
present insightful scheduling policies that form upper and lower bounds to
performance, and are observed to be quite tight. Extensive simulations are also
provided and used to validate our theoretical analysis. We also employ the
framework of the MDS queue to analyse different methods of performing so-called
degraded reads (reading of partial data) in distributed data storage
TOFEC: Achieving Optimal Throughput-Delay Trade-off of Cloud Storage Using Erasure Codes
Our paper presents solutions using erasure coding, parallel connections to
storage cloud and limited chunking (i.e., dividing the object into a few
smaller segments) together to significantly improve the delay performance of
uploading and downloading data in and out of cloud storage.
TOFEC is a strategy that helps front-end proxy adapt to level of workload by
treating scalable cloud storage (e.g. Amazon S3) as a shared resource requiring
admission control. Under light workloads, TOFEC creates more smaller chunks and
uses more parallel connections per file, minimizing service delay. Under heavy
workloads, TOFEC automatically reduces the level of chunking (fewer chunks with
increased size) and uses fewer parallel connections to reduce overhead,
resulting in higher throughput and preventing queueing delay. Our trace-driven
simulation results show that TOFEC's adaptation mechanism converges to an
appropriate code that provides the optimal delay-throughput trade-off without
reducing system capacity. Compared to a non-adaptive strategy optimized for
throughput, TOFEC delivers 2.5x lower latency under light workloads; compared
to a non-adaptive strategy optimized for latency, TOFEC can scale to support
over 3x as many requests
An Overview of a Grid Architecture for Scientific Computing
This document gives an overview of a Grid testbed architecture proposal for
the NorduGrid project. The aim of the project is to establish an inter-Nordic
testbed facility for implementation of wide area computing and data handling.
The architecture is supposed to define a Grid system suitable for solving data
intensive problems at the Large Hadron Collider at CERN. We present the various
architecture components needed for such a system. After that we go on to give a
description of the dynamics by showing the task flow
ARIS and EGIIS Installation, Con guration and Usage Manual
A scalable, production quality dynamic distributed information system for AR
Coding for Fast Content Download
We study the fundamental trade-off between storage and content download time.
We show that the download time can be significantly reduced by dividing the
content into chunks, encoding it to add redundancy and then distributing it
across multiple disks. We determine the download time for two content access
models - the fountain and fork-join models that involve simultaneous content
access, and individual access from enqueued user requests respectively. For the
fountain model we explicitly characterize the download time, while in the
fork-join model we derive the upper and lower bounds. Our results show that
coding reduces download time, through the diversity of distributing the data
across more disks, even for the total storage used.Comment: 8 pages, 6 figures, conferenc
- …