9,590 research outputs found
A Case for Cooperative and Incentive-Based Coupling of Distributed Clusters
Research interest in Grid computing has grown significantly over the past
five years. Management of distributed resources is one of the key issues in
Grid computing. Central to management of resources is the effectiveness of
resource allocation as it determines the overall utility of the system. The
current approaches to superscheduling in a grid environment are non-coordinated
since application level schedulers or brokers make scheduling decisions
independently of the others in the system. Clearly, this can exacerbate the
load sharing and utilization problems of distributed resources due to
suboptimal schedules that are likely to occur. To overcome these limitations,
we propose a mechanism for coordinated sharing of distributed clusters based on
computational economy. The resulting environment, called
\emph{Grid-Federation}, allows the transparent use of resources from the
federation when local resources are insufficient to meet its users'
requirements. The use of computational economy methodology in coordinating
resource allocation not only facilitates the QoS based scheduling, but also
enhances utility delivered by resources.Comment: 22 pages, extended version of the conference paper published at IEEE
Cluster'05, Boston, M
GridSim: A Toolkit for the Modeling and Simulation of Distributed Resource Management and Scheduling for Grid Computing
Clusters, grids, and peer-to-peer (P2P) networks have emerged as popular
paradigms for next generation parallel and distributed computing. The
management of resources and scheduling of applications in such large-scale
distributed systems is a complex undertaking. In order to prove the
effectiveness of resource brokers and associated scheduling algorithms, their
performance needs to be evaluated under different scenarios such as varying
number of resources and users with different requirements. In a grid
environment, it is hard and even impossible to perform scheduler performance
evaluation in a repeatable and controllable manner as resources and users are
distributed across multiple organizations with their own policies. To overcome
this limitation, we have developed a Java-based discrete-event grid simulation
toolkit called GridSim. The toolkit supports modeling and simulation of
heterogeneous grid resources (both time- and space-shared), users and
application models. It provides primitives for creation of application tasks,
mapping of tasks to resources, and their management. To demonstrate suitability
of the GridSim toolkit, we have simulated a Nimrod-G like grid resource broker
and evaluated the performance of deadline and budget constrained cost- and
time-minimization scheduling algorithms
A Computational Economy for Grid Computing and its Implementation in the Nimrod-G Resource Brok
Computational Grids, coupling geographically distributed resources such as
PCs, workstations, clusters, and scientific instruments, have emerged as a next
generation computing platform for solving large-scale problems in science,
engineering, and commerce. However, application development, resource
management, and scheduling in these environments continue to be a complex
undertaking. In this article, we discuss our efforts in developing a resource
management system for scheduling computations on resources distributed across
the world with varying quality of service. Our service-oriented grid computing
system called Nimrod-G manages all operations associated with remote execution
including resource discovery, trading, scheduling based on economic principles
and a user defined quality of service requirement. The Nimrod-G resource broker
is implemented by leveraging existing technologies such as Globus, and provides
new services that are essential for constructing industrial-strength Grids. We
discuss results of preliminary experiments on scheduling some parametric
computations using the Nimrod-G resource broker on a world-wide grid testbed
that spans five continents
PaPaS: A Portable, Lightweight, and Generic Framework for Parallel Parameter Studies
The current landscape of scientific research is widely based on modeling and
simulation, typically with complexity in the simulation's flow of execution and
parameterization properties. Execution flows are not necessarily
straightforward since they may need multiple processing tasks and iterations.
Furthermore, parameter and performance studies are common approaches used to
characterize a simulation, often requiring traversal of a large parameter
space. High-performance computers offer practical resources at the expense of
users handling the setup, submission, and management of jobs. This work
presents the design of PaPaS, a portable, lightweight, and generic workflow
framework for conducting parallel parameter and performance studies. Workflows
are defined using parameter files based on keyword-value pairs syntax, thus
removing from the user the overhead of creating complex scripts to manage the
workflow. A parameter set consists of any combination of environment variables,
files, partial file contents, and command line arguments. PaPaS is being
developed in Python 3 with support for distributed parallelization using SSH,
batch systems, and C++ MPI. The PaPaS framework will run as user processes, and
can be used in single/multi-node and multi-tenant computing systems. An example
simulation using the BehaviorSpace tool from NetLogo and a matrix multiply
using OpenMP are presented as parameter and performance studies, respectively.
The results demonstrate that the PaPaS framework offers a simple method for
defining and managing parameter studies, while increasing resource utilization.Comment: 8 pages, 6 figures, PEARC '18: Practice and Experience in Advanced
Research Computing, July 22--26, 2018, Pittsburgh, PA, US
Designing a Hadoop system based on computational resources and network delay for wide area networks
This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time
- …