68,749 research outputs found
Lustre, Hadoop, Accumulo
Data processing systems impose multiple views on data as it is processed by
the system. These views include spreadsheets, databases, matrices, and graphs.
There are a wide variety of technologies that can be used to store and process
data through these different steps. The Lustre parallel file system, the Hadoop
distributed file system, and the Accumulo database are all designed to address
the largest and the most challenging data storage problems. There have been
many ad-hoc comparisons of these technologies. This paper describes the
foundational principles of each technology, provides simple models for
assessing their capabilities, and compares the various technologies on a
hypothetical common cluster. These comparisons indicate that Lustre provides 2x
more storage capacity, is less likely to loose data during 3 simultaneous drive
failures, and provides higher bandwidth on general purpose workloads. Hadoop
can provide 4x greater read bandwidth on special purpose workloads. Accumulo
provides 10,000x lower latency on random lookups than either Lustre or Hadoop
but Accumulo's bulk bandwidth is 10x less. Significant recent work has been
done to enable mix-and-match solutions that allow Lustre, Hadoop, and Accumulo
to be combined in different ways.Comment: 6 pages; accepted to IEEE High Performance Extreme Computing
conference, Waltham, MA, 201
The Impact of Data Replicatino on Job Scheduling Performance in Hierarchical data Grid
In data-intensive applications data transfer is a primary cause of job
execution delay. Data access time depends on bandwidth. The major bottleneck to
supporting fast data access in Grids is the high latencies of Wide Area
Networks and Internet. Effective scheduling can reduce the amount of data
transferred across the internet by dispatching a job to where the needed data
are present. Another solution is to use a data replication mechanism. Objective
of dynamic replica strategies is reducing file access time which leads to
reducing job runtime. In this paper we develop a job scheduling policy and a
dynamic data replication strategy, called HRS (Hierarchical Replication
Strategy), to improve the data access efficiencies. We study our approach and
evaluate it through simulation. The results show that our algorithm has
improved 12% over the current strategies.Comment: 11 pages, 7 figure
An Approach to Ad hoc Cloud Computing
We consider how underused computing resources within an enterprise may be
harnessed to improve utilization and create an elastic computing
infrastructure. Most current cloud provision involves a data center model, in
which clusters of machines are dedicated to running cloud infrastructure
software. We propose an additional model, the ad hoc cloud, in which
infrastructure software is distributed over resources harvested from machines
already in existence within an enterprise. In contrast to the data center cloud
model, resource levels are not established a priori, nor are resources
dedicated exclusively to the cloud while in use. A participating machine is not
dedicated to the cloud, but has some other primary purpose such as running
interactive processes for a particular user. We outline the major
implementation challenges and one approach to tackling them
Towards Exascale Scientific Metadata Management
Advances in technology and computing hardware are enabling scientists from
all areas of science to produce massive amounts of data using large-scale
simulations or observational facilities. In this era of data deluge, effective
coordination between the data production and the analysis phases hinges on the
availability of metadata that describe the scientific datasets. Existing
workflow engines have been capturing a limited form of metadata to provide
provenance information about the identity and lineage of the data. However,
much of the data produced by simulations, experiments, and analyses still need
to be annotated manually in an ad hoc manner by domain scientists. Systematic
and transparent acquisition of rich metadata becomes a crucial prerequisite to
sustain and accelerate the pace of scientific innovation. Yet, ubiquitous and
domain-agnostic metadata management infrastructure that can meet the demands of
extreme-scale science is notable by its absence.
To address this gap in scientific data management research and practice, we
present our vision for an integrated approach that (1) automatically captures
and manipulates information-rich metadata while the data is being produced or
analyzed and (2) stores metadata within each dataset to permeate
metadata-oblivious processes and to query metadata through established and
standardized data access interfaces. We motivate the need for the proposed
integrated approach using applications from plasma physics, climate modeling
and neuroscience, and then discuss research challenges and possible solutions
ETS (Efficient, Transparent, and Secured) Self-healing Service for Pervasive Computing Applications
To ensure smooth functioning of numerous handheld devices anywhere anytime, the importance of self-healing mechanism cannot be overlooked. Incorporation of efficient fault detection and recovery in device itself is the quest for long but there is no existing self-healing scheme for devices running in pervasive computing environments that can be claimed as the ultimate solution. Moreover, the highest degree of transparency, security and privacy attainability should also be maintained. ETS Self-healing service, an integral part of our developing middleware named MARKS (Middleware Adaptability for Resource discovery, Knowledge usability, and Self-healing), holds promise for offering all of those functionalities
A Methodology for Engineering Collaborative and ad-hoc Mobile Applications using SyD Middleware
Today’s web applications are more collaborative and utilize standard and ubiquitous Internet protocols. We have earlier developed System on Mobile Devices (SyD) middleware to rapidly develop and deploy collaborative applications over heterogeneous and possibly mobile devices hosting web objects. In this paper, we present the software engineering methodology for developing SyD-enabled web applications and illustrate it through a case study on two representative applications: (i) a calendar of meeting application, which is a collaborative application and (ii) a travel application which is an ad-hoc collaborative application. SyD-enabled web objects allow us to create a collaborative application rapidly with limited coding effort. In this case study, the modular software architecture allowed us to hide the inherent heterogeneity among devices, data stores, and networks by presenting a uniform and persistent object view of mobile objects interacting through XML/SOAP requests and responses. The performance results we obtained show that the application scales well as we increase the group size and adapts well within the constraints of mobile devices
- …