160,923 research outputs found
File-based storage of Digital Objects and constituent datastreams: XMLtapes and Internet Archive ARC files
This paper introduces the write-once/read-many XMLtape/ARC storage approach
for Digital Objects and their constituent datastreams. The approach combines
two interconnected file-based storage mechanisms that are made accessible in a
protocol-based manner. First, XML-based representations of multiple Digital
Objects are concatenated into a single file named an XMLtape. An XMLtape is a
valid XML file; its format definition is independent of the choice of the
XML-based complex object format by which Digital Objects are represented. The
creation of indexes for both the identifier and the creation datetime of the
XML-based representation of the Digital Objects facilitates OAI-PMH-based
access to Digital Objects stored in an XMLtape. Second, ARC files, as
introduced by the Internet Archive, are used to contain the constituent
datastreams of the Digital Objects in a concatenated manner. An index for the
identifier of the datastream facilitates OpenURL-based access to an ARC file.
The interconnection between XMLtapes and ARC files is provided by conveying the
identifiers of ARC files associated with an XMLtape as administrative
information in the XMLtape, and by including OpenURL references to constituent
datastreams of a Digital Object in the XML-based representation of that Digital
Object.Comment: 12 pages, 1 figures (camera-ready copy for ECDL 2005
Objcache: An Elastic Filesystem over External Persistent Storage for Container Clusters
Container virtualization enables emerging AI workloads such as model serving,
highly parallelized training, machine learning pipelines, and so on, to be
easily scaled on demand on the elastic cloud infrastructure. Particularly, AI
workloads require persistent storage to store data such as training inputs,
models, and checkpoints. An external storage system like cloud object storage
is a common choice because of its elasticity and scalability. To mitigate
access latency to external storage, caching at a local filesystem is an
essential technique. However, building local caches on scaling clusters must
cope with explosive disk usage, redundant networking, and unexpected failures.
We propose objcache, an elastic filesystem over external storage. Objcache
introduces an internal transaction protocol over Raft logging to enable atomic
updates of distributed persistent states with consistent hashing. The proposed
transaction protocol can also manage inode dirtiness by maintaining the
consistency between the local cache and external storage. Objcache supports
scaling down to zero by automatically evicting dirty files to external storage.
Our evaluation reports that objcache speeded up model serving startup by 98.9%
compared to direct copies via S3 interfaces. Scaling up with dirty files
completed from 2 to 14 seconds with 1024 dirty files.Comment: 13 page
Distributed Storage Allocations for Optimal Delay
We examine the problem of creating an encoded distributed storage
representation of a data object for a network of mobile storage nodes so as to
achieve the optimal recovery delay. A source node creates a single data object
and disseminates an encoded representation of it to other nodes for storage,
subject to a given total storage budget. A data collector node subsequently
attempts to recover the original data object by contacting other nodes and
accessing the data stored in them. By using an appropriate code, successful
recovery is achieved when the total amount of data accessed is at least the
size of the original data object. The goal is to find an allocation of the
given budget over the nodes that optimizes the recovery delay incurred by the
data collector; two objectives are considered: (i) maximization of the
probability of successful recovery by a given deadline, and (ii) minimization
of the expected recovery delay. We solve the problem completely for the second
objective in the case of symmetric allocations (in which all nonempty nodes
store the same amount of data), and show that the optimal symmetric allocation
for the two objectives can be quite different. A simple data dissemination and
storage protocol for a mobile delay-tolerant network is evaluated under various
scenarios via simulations. Our results show that the choice of storage
allocation can have a significant impact on the recovery delay performance, and
that coding may or may not be beneficial depending on the circumstances.Comment: Extended version of an IEEE ISIT 2011 paper. 10 pages, 4 figure
A metaobject architecture for fault-tolerant distributed systems : the FRIENDS approach
The FRIENDS system developed at LAAS-CNRS is a metalevel architecture providing libraries of metaobjects for fault
tolerance, secure communication, and group-based distributed applications. The use of metaobjects provides a nice separation of concerns between mechanisms and applications. Metaobjects can be used transparently by applications and can be composed according to the needs of a given application, a given architecture, and its underlying properties. In FRIENDS, metaobjects are used recursively to add new properties to applications. They are designed using an object oriented design method and implemented on top of basic system services. This paper describes the FRIENDS software-based architecture, the object-oriented development of metaobjects, the experiments that we have done, and summarizes the advantages and drawbacks of a metaobject approach for building fault-tolerant system
Efficient HTTP based I/O on very large datasets for high performance computing with the libdavix library
Remote data access for data analysis in high performance computing is
commonly done with specialized data access protocols and storage systems. These
protocols are highly optimized for high throughput on very large datasets,
multi-streams, high availability, low latency and efficient parallel I/O. The
purpose of this paper is to describe how we have adapted a generic protocol,
the Hyper Text Transport Protocol (HTTP) to make it a competitive alternative
for high performance I/O and data analysis applications in a global computing
grid: the Worldwide LHC Computing Grid. In this work, we first analyze the
design differences between the HTTP protocol and the most common high
performance I/O protocols, pointing out the main performance weaknesses of
HTTP. Then, we describe in detail how we solved these issues. Our solutions
have been implemented in a toolkit called davix, available through several
recent Linux distributions. Finally, we describe the results of our benchmarks
where we compare the performance of davix against a HPC specific protocol for a
data analysis use case.Comment: Presented at: Very large Data Bases (VLDB) 2014, Hangzho
- …