1 research outputs found
Enabling Lock-Free Concurrent Fine-Grain Access to Massive Distributed Data: Application to Supernovae Detection
We consider the problem of efficiently managing massive data in a large-scale
distributed environment. We consider data strings of size in the order of
Terabytes, shared and accessed by concurrent clients. On each individual
access, a segment of a string, of the order of Megabytes, is read or modified.
Our goal is to provide the clients with efficient fine-grain access the data
string as concurrently as possible, without locking the string itself. This
issue is crucial in the context of applications in the field of astronomy,
databases, data mining and multimedia. We illustrate these requiremens with the
case of an application for searching supernovae. Our solution relies on
distributed, RAM-based data storage, while leveraging a DHT-based, parallel
metadata management scheme. The proposed architecture and algorithms have been
validated through a software prototype and evaluated in a cluster environment