75,909 research outputs found
Explicit Construction of Optimal Exact Regenerating Codes for Distributed Storage
Erasure coding techniques are used to increase the reliability of distributed
storage systems while minimizing storage overhead. Also of interest is
minimization of the bandwidth required to repair the system following a node
failure. In a recent paper, Wu et al. characterize the tradeoff between the
repair bandwidth and the amount of data stored per node. They also prove the
existence of regenerating codes that achieve this tradeoff.
In this paper, we introduce Exact Regenerating Codes, which are regenerating
codes possessing the additional property of being able to duplicate the data
stored at a failed node. Such codes require low processing and communication
overheads, making the system practical and easy to maintain. Explicit
construction of exact regenerating codes is provided for the minimum bandwidth
point on the storage-repair bandwidth tradeoff, relevant to
distributed-mail-server applications. A subspace based approach is provided and
shown to yield necessary and sufficient conditions on a linear code to possess
the exact regeneration property as well as prove the uniqueness of our
construction.
Also included in the paper, is an explicit construction of regenerating codes
for the minimum storage point for parameters relevant to storage in
peer-to-peer systems. This construction supports a variable number of nodes and
can handle multiple, simultaneous node failures. All constructions given in the
paper are of low complexity, requiring low field size in particular.Comment: 7 pages, 2 figures, in the Proceedings of Allerton Conference on
Communication, Control and Computing, September 200
In-Network Redundancy Generation for Opportunistic Speedup of Backup
Erasure coding is a storage-efficient alternative to replication for
achieving reliable data backup in distributed storage systems. During the
storage process, traditional erasure codes require a unique source node to
create and upload all the redundant data to the different storage nodes.
However, such a source node may have limited communication and computation
capabilities, which constrain the storage process throughput. Moreover, the
source node and the different storage nodes might not be able to send and
receive data simultaneously -- e.g., nodes might be busy in a datacenter
setting, or simply be offline in a peer-to-peer setting -- which can further
threaten the efficacy of the overall storage process. In this paper we propose
an "in-network" redundancy generation process which distributes the data
insertion load among the source and storage nodes by allowing the storage nodes
to generate new redundant data by exchanging partial information among
themselves, improving the throughput of the storage process. The process is
carried out asynchronously, utilizing spare bandwidth and computing resources
from the storage nodes. The proposed approach leverages on the local
repairability property of newly proposed erasure codes tailor made for the
needs of distributed storage systems. We analytically show that the performance
of this technique relies on an efficient usage of the spare node resources, and
we derive a set of scheduling algorithms to maximize the same. We
experimentally show, using availability traces from real peer-to-peer
applications as well as Google data center availability and workload traces,
that our algorithms can, depending on the environment characteristics, increase
the throughput of the storage process significantly (up to 90% in data centers,
and 60% in peer-to-peer settings) with respect to the classical naive data
insertion approach
- …