48,438 research outputs found
Joint Latency and Cost Optimization for Erasure-coded Data Center Storage
Modern distributed storage systems offer large capacity to satisfy the
exponentially increasing need of storage space. They often use erasure codes to
protect against disk and node failures to increase reliability, while trying to
meet the latency requirements of the applications and clients. This paper
provides an insightful upper bound on the average service delay of such
erasure-coded storage with arbitrary service time distribution and consisting
of multiple heterogeneous files. Not only does the result supersede known delay
bounds that only work for a single file or homogeneous files, it also enables a
novel problem of joint latency and storage cost minimization over three
dimensions: selecting the erasure code, placement of encoded chunks, and
optimizing scheduling policy. The problem is efficiently solved via the
computation of a sequence of convex approximations with provable convergence.
We further prototype our solution in an open-source, cloud storage deployment
over three geographically distributed data centers. Experimental results
validate our theoretical delay analysis and show significant latency reduction,
providing valuable insights into the proposed latency-cost tradeoff in
erasure-coded storage.Comment: 14 pages, presented in part at IFIP Performance, Oct 201
Mirrored and Hybrid Disk Arrays: Organization, Scheduling, Reliability, and Performance
Basic mirroring (BM) classified as RAID level 1 replicates data on two disks,
thus doubling disk access bandwidth for read requests. RAID1/0 is an array of
BM pairs with balanced loads due to striping. When a disk fails the read load
on its pair is doubled, which results in halving the maximum attainable
bandwidth. We review RAID1 organizations which attain a balanced load upon disk
failure, but as shown by reliability analysis tend to be less reliable than
RAID1/0. Hybrid disk arrays which store XORed instead of replicated data tend
to have a higher reliability than mirrored disks, but incur a higher overhead
in updating data. Read request response time can be improved by processing them
at a higher priority than writes, since they have a direct effect on
application response time. Shortest seek distance and affinity based routing
both shorten seek time. Anticipatory arm placement places arms optimally to
minimize the seek distance. The analysis of RAID1 in normal, degraded, and
rebuild mode is provided to quantify RAID1/0 performance. We compare the
reliability of mirrored disk organizations against each other and hybrid disks
and erasure coded disk arrays
Distributed Data Collection and Storage Algorithms for Collaborative Learning Vision Sensor Devices with Applications to Pilgrimage
This work presents novel distributed data collection systems and storage
algorithms for collaborative learning wireless sensor networks (WSNs). In a
large WSN, consider collaborative sensor devices distributed randomly to
acquire information and learn about a certain field. Such sensors have less
power, small bandwidth, and short memory, and they might disappear from the
network after certain time of operations. The goal of this work is to design
efficient strategies to learn about the field by collecting sensed data from
these sensors with less computational overhead and efficient storage
encoding operations.
In this data collection system, we propose two distributed data storage
algorithms (DSA's) to solve this problem with the means of network flooding and
connectivity among sensor devices. In the first algorithm denoted, DSA-I, it's
assumed that the total number of nodes is known for each node in the network.
We show that this algorithm is efficient in terms of the encoding/decoding
operations. Furthermore, every node uses network flooding to disseminate its
data throughout the network using mixing time approximately O(n). In the second
algorithm denoted, DSA-II, it's assumed that the total number of nodes is not
known for each learning sensor, hence dissemination of the data does not depend
on the value of . In this case we show that the encoding operations take
, where is the mean degree of the network graph and is a
system parameter. Performance of these two algorithms match the derived
theoretical results. Finally, we show how to deploy these algorithms for
monitoring and measuring certain phenomenons in American-made camp tents
located in Minna field in south-east side of Makkah.Comment: arXiv admin note: substantial text overlap with arXiv:0908.441
Decentralized Coding Algorithms for Distributed Storage in Wireless Sensor Networks
We consider large-scale wireless sensor networks with nodes, out of which
k are in possession, (e.g., have sensed or collected in some other way) k
information packets. In the scenarios in which network nodes are vulnerable
because of, for example, limited energy or a hostile environment, it is
desirable to disseminate the acquired information throughout the network so
that each of the n nodes stores one (possibly coded) packet so that the
original k source packets can be recovered, locally and in a computationally
simple way from any k(1 + \epsilon) nodes for some small \epsilon > 0. We
develop decentralized Fountain codes based algorithms to solve this problem.
Unlike all previously developed schemes, our algorithms are truly distributed,
that is, nodes do not know n, k or connectivity in the network, except in their
own neighborhoods, and they do not maintain any routing tables.Comment: Accepted for publication in IEEE JSAC, 201
Multilevel Diversity Coding Systems: Rate Regions, Codes, Computation, & Forbidden Minors
The rate regions of multilevel diversity coding systems (MDCS), a sub-class
of the broader family of multi-source multi-sink networks with special
structure, are investigated. After showing how to enumerate all non-isomorphic
MDCS instances of a given size, the Shannon outer bound and several achievable
inner bounds based on linear codes are given for the rate region of each
non-isomorphic instance. For thousands of MDCS instances, the bounds match, and
hence exact rate regions are proven. Results gained from these computations are
summarized in key statistics involving aspects such as the sufficiency of
scalar binary codes, the necessary size of vector binary codes, etc. Also, it
is shown how to generate computer aided human readable converse proofs, as well
as how to construct the codes for an achievability proof. Based on this large
repository of rate regions, a series of results about general MDCS cases that
they inspired are introduced and proved. In particular, a series of embedding
operations that preserve the property of sufficiency of scalar or vector codes
are presented. The utility of these operations is demonstrated by boiling the
thousands of MDCS instances for which binary scalar codes are insufficient down
to 12 forbidden smallest embedded MDCS instances.Comment: Submitted to IEEE Transactions on Information Theory, 52 page
A Survey of Delay Tolerant Networks Routing Protocols
Advances in Micro-Electro-Mechanical Systems (MEMS) have revolutionized the
digital age to a point where animate and inanimate objects can be used as a
communication channel. In addition, the ubiquity of mobile phones with
increasing capabilities and ample resources means people are now effectively
mobile sensors that can be used to sense the environment as well as data
carriers. These objects, along with their devices, form a new kind of networks
that are characterized by frequent disconnections, resource constraints and
unpredictable or stochastic mobility patterns. A key underpinning in these
networks is routing or data dissemination protocols that are designed
specifically to handle the aforementioned characteristics. Therefore, there is
a need to review state-of-the-art routing protocols, categorize them, and
compare and contrast their approaches in terms of delivery rate, resource
consumption and end-to-end delay. To this end, this paper reviews 63 unicast,
multicast and coding-based routing protocols that are designed specifically to
run in delay tolerant or challenged networks. We provide an extensive
qualitative comparison of all protocols, highlight their experimental setup and
outline their deficiencies in terms of design and research methodology. Apart
from that, we review research that aims to exploit studies on social networks
and epidemiology in order to improve routing protocol performance. Lastly, we
provide a list of future research directions.Comment: 56 page
A Distributed Data Collection Algorithm for Wireless Sensor Networks with Persistent Storage Nodes
A distributed data collection algorithm to accurately store and forward
information obtained by wireless sensor networks is proposed. The proposed
algorithm does not depend on the sensor network topology, routing tables, or
geographic locations of sensor nodes, but rather makes use of uniformly
distributed storage nodes. Analytical and simulation results for this algorithm
show that, with high probability, the data disseminated by the sensor nodes can
be precisely collected by querying any small set of storage nodes
Optimal Control of Storage Regeneration with Repair Codes
High availability of containerized applications requires to perform robust
storage of applications' state. Since basic replication techniques are
extremely costly at scale, storage space requirements can be reduced by means
of erasure or repairing codes. In this paper we address storage regeneration
using repair codes, a robust distributed storage technique with no need to
fully restore the whole state in case of failure. In fact, only the lost
servers' content is replaced. To do so, new cleanslate storage units are made
operational at a cost for activating new storage servers and a cost for the
transfer of repair data. Our goal is to guarantee maximal availability of
containers' state files by a given deadline. activation of servers and
communication cost. Upon a fault occurring at a subset of the storage servers,
we aim at ensuring that they are repaired by a given deadline. We introduce a
controlled fluid model and derive the optimal activation policy to replace
servers under such correlated faults. The solution concept is the optimal
control of regeneration via the Pontryagin minimum principle. We characterise
feasibility conditions and we prove that the optimal policy is of threshold
type. Numerical results describe how to apply the model for system dimensioning
and show the tradeoff betweenComment: This research was performed while the first author was visiting Nokia
Bell Lab
Application-Driven Near-Data Processing for Similarity Search
Similarity search is a key to a variety of applications including
content-based search for images and video, recommendation systems, data
deduplication, natural language processing, computer vision, databases,
computational biology, and computer graphics. At its core, similarity search
manifests as k-nearest neighbors (kNN), a computationally simple primitive
consisting of highly parallel distance calculations and a global top-k sort.
However, kNN is poorly supported by today's architectures because of its high
memory bandwidth requirements.
This paper proposes an application-driven near-data processing accelerator
for similarity search: the Similarity Search Associative Memory (SSAM). By
instantiating compute units close to memory, SSAM benefits from the higher
memory bandwidth and density exposed by emerging memory technologies. We
evaluate the SSAM design down to layout on top of the Micron hybrid memory cube
(HMC), and show that SSAM can achieve up to two orders of magnitude
area-normalized throughput and energy efficiency improvement over multicore
CPUs; we also show SSAM is faster and more energy efficient than competing GPUs
and FPGAs. Finally, we show that SSAM is also useful for other data intensive
tasks like kNN index construction, and can be generalized to semantically
function as a high capacity content addressable memory.Comment: 15 pages, 8 figures, 7 table
Repairing Multiple Failures with Coordinated and Adaptive Regenerating Codes
Erasure correcting codes are widely used to ensure data persistence in
distributed storage systems. This paper addresses the simultaneous repair of
multiple failures in such codes. We go beyond existing work (i.e., regenerating
codes by Dimakis et al.) by describing (i) coordinated regenerating codes (also
known as cooperative regenerating codes) which support the simultaneous repair
of multiple devices, and (ii) adaptive regenerating codes which allow adapting
the parameters at each repair. Similarly to regenerating codes by Dimakis et
al., these codes achieve the optimal tradeoff between storage and the repair
bandwidth. Based on these extended regenerating codes, we study the impact of
lazy repairs applied to regenerating codes and conclude that lazy repairs
cannot reduce the costs in term of network bandwidth but allow reducing the
disk-related costs (disk bandwidth and disk I/O).Comment: Update to previous version adding (i) study of lazy repairs, (ii)
adaptive codes at the MBR point, and (iii) discussion of related work.
Extended from a regular paper at NetCod 2011 available at
http://dx.doi.org/10.1109/ISNETCOD.2011.5978920 . First version: "Beyond
Regenerating Codes", September 2010 on http://hal.inria.fr/inria-00516647
- …