305 research outputs found
Coding for the Clouds: Coding Techniques for Enabling Security, Locality, and Availability in Distributed Storage Systems
Cloud systems have become the backbone of many applications such as multimedia
streaming, e-commerce, and cluster computing. At the foundation of any cloud architecture
lies a large-scale, distributed, data storage system. To accommodate the massive
amount of data being stored on the cloud, these distributed storage systems (DSS) have
been scaled to contain hundreds to thousands of nodes that are connected through a networking
infrastructure. Such data-centers are usually built out of commodity components,
which make failures the norm rather than the exception.
In order to combat node failures, data is typically stored in a redundant fashion. Due to
the exponential data growth rate, many DSS are beginning to resort to error control coding
over conventional replication methods, as coding offers high storage space efficiency. This
paradigm shift from replication to coding, along with the need to guarantee reliability, efficiency,
and security in DSS, has created a new set of challenges and opportunities, opening
up a new area of research. This thesis addresses several of these challenges and opportunities
by broadly making the following contributions. (i) We design practically amenable,
low-complexity coding schemes that guarantee security of cloud systems, ensure quick
recovery from failures, and provide high availability for retrieving partial information; and
(ii) We analyze fundamental performance limits and optimal trade-offs between the key
performance metrics of these coding schemes.
More specifically, we first consider the problem of achieving information-theoretic
security in DSS against an eavesdropper that can observe a limited number of nodes. We
present a framework that enables design of secure repair-efficient codes through a joint
construction of inner and outer codes. Then, we consider a practically appealing notion
of weakly secure coding, and construct coset codes that can weakly secure a wide class of regenerating codes that reduce the amount of data downloaded during node repair.
Second, we consider the problem of meeting repair locality constraints, which specify
the number of nodes participating in the repair process. We propose a notion of unequal
locality, which enables different locality values for different nodes, ensuring quick recovery
for nodes storing important data. We establish tight upper bounds on the minimum
distance of linear codes with unequal locality, and present optimal code constructions.
Next, we extend the notion of locality from the Hamming metric to the rank and subspace
metrics, with the goal of designing codes for efficient data recovery from special types of
correlated failures in DSS.We construct a family of locally recoverable rank-metric codes
with optimal data recovery properties.
Finally, we consider the problem of providing high availability, which is ensured by
enabling node repair from multiple disjoint subsets of nodes of small size. We study
codes with availability from a queuing-theoretical perspective by analyzing the average
time necessary to download a block of data under the Poisson request arrival model when
each node takes a random amount of time to fetch its contents. We compare the delay
performance of the availability codes with several alternatives such as conventional erasure
codes and replication schemes
Zigzag Codes: MDS Array Codes with Optimal Rebuilding
MDS array codes are widely used in storage systems to protect data against
erasures. We address the \emph{rebuilding ratio} problem, namely, in the case
of erasures, what is the fraction of the remaining information that needs to be
accessed in order to rebuild \emph{exactly} the lost information? It is clear
that when the number of erasures equals the maximum number of erasures that an
MDS code can correct then the rebuilding ratio is 1 (access all the remaining
information). However, the interesting and more practical case is when the
number of erasures is smaller than the erasure correcting capability of the
code. For example, consider an MDS code that can correct two erasures: What is
the smallest amount of information that one needs to access in order to correct
a single erasure? Previous work showed that the rebuilding ratio is bounded
between 1/2 and 3/4, however, the exact value was left as an open problem. In
this paper, we solve this open problem and prove that for the case of a single
erasure with a 2-erasure correcting code, the rebuilding ratio is 1/2. In
general, we construct a new family of -erasure correcting MDS array codes
that has optimal rebuilding ratio of in the case of erasures,
. Our array codes have efficient encoding and decoding
algorithms (for the case they use a finite field of size 3) and an
optimal update property.Comment: 23 pages, 5 figures, submitted to IEEE transactions on information
theor
MDS Array Codes With (Near) Optimal Repair Bandwidth for All Admissible Repair Degrees
Abundant high-rate (n, k) minimum storage regenerating (MSR) codes have been
reported in the literature. However, most of them require contacting all the
surviving nodes during a node repair process, resulting in a repair degree of
d=n-1. In practical systems, it may not always be feasible to connect and
download data from all surviving nodes, as some nodes may be unavailable.
Therefore, there is a need for MSR code constructions with a repair degree of
d<n-1. Up to now, only a few (n, k) MSR code constructions with repair degree
d<n-1 have been reported, some have a large sub-packetization level, a large
finite field, or restrictions on the repair degree d. In this paper, we propose
a new (n, k) MSR code construction that works for any repair degree d>k, and
has a smaller sub-packetization level or finite field than some existing
constructions. Additionally, in conjunction with a previous generic
transformation to reduce the sub-packetization level, we obtain an MDS array
code with a small sub-packetization level and -optimal repair
bandwidth (i.e., times the optimal repair bandwidth) for repair
degree d=n-1. This code outperforms some existing ones in terms of either the
sub-packetization level or the field size.Comment: Submitted to the IEEE Transactions on Communication
- …