116,545 research outputs found
Recommended from our members
A Specification-based Anycast Scheme for Scalable Resource Discovery in Distributed Systems
Anycast is a powerful paradigm for managing and locating resources in large scale distributed computing systems. This paper presents a novel specification-based anycasting scheme for resource discovery in such environments. The effectiveness of our proposal is demonstrated through simulation results, in which we observed a remarkable performance enhancement in different aspects (such as discovery latency, discovery cost, discovery load, etc.) over similar non-anycast based discovery methods
Planning And Scheduling For Large-scaledistributed Systems
Many applications require computing resources well beyond those available on any single system. Simulations of atomic and subatomic systems with application to material science, computations related to study of natural sciences, and computer-aided design are examples of applications that can benefit from the resource-rich environment provided by a large collection of autonomous systems interconnected by high-speed networks. To transform such a collection of systems into a user\u27s virtual machine, we have to develop new algorithms for coordination, planning, scheduling, resource discovery, and other functions that can be automated. Then we can develop societal services based upon these algorithms, which hide the complexity of the computing system for users. In this dissertation, we address the problem of planning and scheduling for large-scale distributed systems. We discuss a model of the system, analyze the need for planning, scheduling, and plan switching to cope with a dynamically changing environment, present algorithms for the three functions, report the simulation results to study the performance of the algorithms, and introduce an architecture for an intelligent large-scale distributed system
Towards self-resource discovery and selection models in grid computing
Global computational grids nowadays are suffered from ossification problems due to the following fundamental challenges related to different existing solutions in grid computing: scalability, adaptability, security, reliability, availability and manageability.The management difficulty is due to heterogeneity, dynamicity and locality of the resources within global grid networks.Large-scale grids make the fundamental problem of resource discovery a great challenge.This paper presents a self-resource discovery mechanism (SRDM) that achieves efficient grid resource discovery and takes advantage of the strengths of both hierarchy and decentralized approaches that were previously developed for grid based P2P resource discovery.P2P systems offer potential strengths such as self-organization, self-healing, and robustness to failure or attacks. Unfortunately, the majority of existing Distributed Hash Table (DHT) based P2P overlays are lacking of attributes range queries that are familiar in resource discovery lookups.The proposed model builds an effective distributed hierarchy that providing scalable, decentralized resource discovery and allocation as well as load balancing for distributed computing using large scale pools of heterogeneous computers. Fundamentally, SRDM employs the spatial index and partitions the overlay space to build a distributed quad tree; each computational resource in the network can calculate its Nodepower.Next, it encodes the information about each node’s available computational resources power in the structure of the links connecting the nodes in the network.This distributed encoding is self-organized, with each node managing its in-degree and local connectivity via its available Nodepower.Assignment of incoming jobs to nodes with the freest resources is also accomplished by sampling it
Towards self-resource discovery and selection models in grid computing
Global computational grids nowadays are suffered from ossification problems due to the following fundamental challenges related to different existing solutions in grid computing: scalability, adaptability, security, reliability, availability and manageability.The management difficulty is due to heterogeneity, dynamicity and locality of the resources within global grid networks.Large-scale grids make the fundamental problem of resource discovery a great challenge.This paper presents a self-resource discovery mechanism (SRDM) that achieves efficient grid resource discovery and takes advantage of the strengths of both hierarchy and decentralized approaches that were previously developed for grid based P2P resource discovery.P2P systems offer potential strengths such as self-organization, self-healing, and robustness to failure or attacks. Unfortunately, the majority of existing Distributed Hash Table (DHT) based P2P overlays are lacking of attributes range queries that are familiar in resource discovery lookups.The proposed model builds an effective distributed hierarchy that providing scalable, decentralized resource discovery and allocation as well as load balancing for distributed computing using large scale pools of heterogeneous computers. Fundamentally, SRDM employs the spatial index and partitions the overlay space to build a distributed quad tree; each computational resource in the network can calculate its Nodepower.Next, it encodes the information about each node’s available computational resources power in the structure of the links connecting the nodes in the network.This distributed encoding is self-organized, with each node managing its in-degree and local connectivity via its available Nodepower.Assignment of incoming jobs to nodes with the freest resources is also accomplished by sampling it
Recommended from our members
A Self-organizing and Self-configuration Algorithm for Resource Management in Service-oriented Systems
With the ever increasing deployment of service-oriented distributed systems in large-scale and heterogeneous computing environments, clustering and communication overlay topology design has become more and more important to address several challenging issues and conflicting requirements, such as efficient scheduling and distribution of services among computing resources, reducing communication cost between services, high performance service and resource discovery while considering both inter-service and inter-node properties and also increasing the load distribution and the load balance. In this paper, a four-stage hierarchical clustering algorithm is proposed which automates the process of the optimally composing communicating groups in a dynamic way while preserving the proximity of the nodes. The simulation results show the performance of the algorithm with respect to load balance, scalability and efficiency
The Anatomy of the Grid - Enabling Scalable Virtual Organizations
"Grid" computing has emerged as an important new field, distinguished from
conventional distributed computing by its focus on large-scale resource
sharing, innovative applications, and, in some cases, high-performance
orientation. In this article, we define this new field. First, we review the
"Grid problem," which we define as flexible, secure, coordinated resource
sharing among dynamic collections of individuals, institutions, and
resources-what we refer to as virtual organizations. In such settings, we
encounter unique authentication, authorization, resource access, resource
discovery, and other challenges. It is this class of problem that is addressed
by Grid technologies. Next, we present an extensible and open Grid
architecture, in which protocols, services, application programming interfaces,
and software development kits are categorized according to their roles in
enabling resource sharing. We describe requirements that we believe any such
mechanisms must satisfy, and we discuss the central role played by the
intergrid protocols that enable interoperability among different Grid systems.
Finally, we discuss how Grid technologies relate to other contemporary
technologies, including enterprise integration, application service provider,
storage service provider, and peer-to-peer computing. We maintain that Grid
concepts and technologies complement and have much to contribute to these other
approaches.Comment: 24 pages, 5 figure
Knowledge discovery for scheduling in computational grids
International audienceScheduling in computational grids addresses the allocation of computing jobs to globally distributed compute resources. In a frequently changing resource environment, scheduling decisions have to be made rapidly. Depending on both the job properties and the current state of the resources, those decisions are different. Thus, the performance of grid scheduling systems highly depends on their adaptivity and flexibility in changing environments. Under these conditions, methods from knowledge discovery yielded significant success to augment and substitute conventional grid scheduling techniques. This paper presents a survey on approaches to extract, represent, and utilize knowledge to improve the grid scheduling performance. It aims to give researchers insight into techniques used for knowledge-supported scheduling in large-scale distributed computing environments
An economic view of indirect reputation management for grids
Scientific collaboration are becoming interdisciplinary, and scientists are working in informal collaboration to solve complex problems that require multiple types of large resources. An option is a computational grid. A computational grid is a distributed infrastructure that appears to the end user as one large computing resource across organization boundaries. Grid technologies enable large-scale sharing of resources within formal or informal consortia of individuals and/or institutions, usually called virtual organizations. In these settings, the discovery, characterization, management, and monitoring of resources, services, and computations can be challenging due to the considerable diversity, large numbers, dynamic behavior, and geographical distribution of the entities in which a user might be interested. Trust is one of the biggest concerns in the grid resource management field. Grid systems can employ reputation mechanisms in order to provide this essential trust, but not usually without incurring in certain additional costs that negate the potential performance gains offered by grid computing technologies. Moreover, current reputation mechanisms are not appropriate for resource management in large-scale systems (generally used in P2P). In this paper, we present a new reputation model for resource management based on a economy model. Also we demonstrate how it can by employed to add trust into algorithms for grid scheduling. Finally, we simulate the proposed resource management algorithm in order to verify its effectiveness.Facultad de Informátic
Descoberta de recursos para sistemas de escala arbitrarias
Doutoramento em InformáticaTecnologias de Computação Distribuída em larga escala tais como Cloud,
Grid, Cluster e Supercomputadores HPC estão a evoluir juntamente com a
emergência revolucionária de modelos de múltiplos núcleos (por exemplo:
GPU, CPUs num único die, Supercomputadores em single die, Supercomputadores
em chip, etc) e avanços significativos em redes e soluções de
interligação. No futuro, nós de computação com milhares de núcleos podem
ser ligados entre si para formar uma única unidade de computação
transparente que esconde das aplicações a complexidade e a natureza distribuída desses sistemas com múltiplos núcleos. A fim de beneficiar de forma
eficiente de todos os potenciais recursos nesses ambientes de computação
em grande escala com múltiplos núcleos ativos, a descoberta de recursos é um elemento crucial para explorar ao máximo as capacidade de todos
os recursos heterogéneos distribuídos, através do reconhecimento preciso e
localização desses recursos no sistema. A descoberta eficiente e escalável
de recursos ´e um desafio para tais sistemas futuros, onde os recursos e as
infira-estruturas de computação e comunicação subjacentes são altamente
dinâmicas, hierarquizadas e heterogéneas. Nesta tese, investigamos o problema
da descoberta de recursos no que diz respeito aos requisitos gerais da
escalabilidade arbitrária de ambientes de computação futuros com múltiplos
núcleos ativos. A principal contribuição desta tese ´e a proposta de uma
entidade de descoberta de recursos adaptativa híbrida (Hybrid Adaptive
Resource Discovery - HARD), uma abordagem de descoberta de recursos eficiente
e altamente escalável, construída sobre uma sobreposição hierárquica
virtual baseada na auto-organizaçãoo e auto-adaptação de recursos de processamento
no sistema, onde os recursos computacionais são organizados
em hierarquias distribuídas de acordo com uma proposta de modelo de
descriçãoo de recursos multi-camadas hierárquicas. Operacionalmente, em
cada camada, que consiste numa arquitetura ponto-a-ponto de módulos que,
interagindo uns com os outros, fornecem uma visão global da disponibilidade
de recursos num ambiente distribuído grande, dinâmico e heterogéneo. O
modelo de descoberta de recursos proposto fornece a adaptabilidade e flexibilidade
para executar consultas complexas através do apoio a um conjunto
de características significativas (tais como multi-dimensional, variedade e
consulta agregada) apoiadas por uma correspondência exata e parcial, tanto
para o conteúdo de objetos estéticos e dinâmicos. Simulações mostram
que o HARD pode ser aplicado a escalas arbitrárias de dinamismo, tanto
em termos de complexidade como de escala, posicionando esta proposta
como uma arquitetura adequada para sistemas futuros de múltiplos núcleos.
Também contribuímos com a proposta de um regime de gestão eficiente
dos recursos para sistemas futuros que podem utilizar recursos distribuíos
de forma eficiente e de uma forma totalmente descentralizada. Além disso,
aproveitando componentes de descoberta (RR-RPs) permite que a nossa
plataforma de gestão de recursos encontre e aloque dinamicamente recursos
disponíeis que garantam os parâmetros de QoS pedidos.Large scale distributed computing technologies such as Cloud, Grid, Cluster
and HPC supercomputers are progressing along with the revolutionary emergence
of many-core designs (e.g. GPU, CPUs on single die, supercomputers
on chip, etc.) and significant advances in networking and interconnect solutions.
In future, computing nodes with thousands of cores may be connected
together to form a single transparent computing unit which hides from applications
the complexity and distributed nature of these many core systems. In
order to efficiently benefit from all the potential resources in such large scale
many-core-enabled computing environments, resource discovery is the vital
building block to maximally exploit the capabilities of all distributed heterogeneous
resources through precisely recognizing and locating those resources
in the system. The efficient and scalable resource discovery is challenging for
such future systems where the resources and the underlying computation and
communication infrastructures are highly-dynamic, highly-hierarchical and
highly-heterogeneous. In this thesis, we investigate the problem of resource
discovery with respect to the general requirements of arbitrary scale future
many-core-enabled computing environments. The main contribution of this
thesis is to propose Hybrid Adaptive Resource Discovery (HARD), a novel
efficient and highly scalable resource-discovery approach which is built upon
a virtual hierarchical overlay based on self-organization and self-adaptation
of processing resources in the system, where the computing resources are
organized into distributed hierarchies according to a proposed hierarchical
multi-layered resource description model. Operationally, at each layer, it
consists of a peer-to-peer architecture of modules that, by interacting with
each other, provide a global view of the resource availability in a large,
dynamic and heterogeneous distributed environment. The proposed resource
discovery model provides the adaptability and flexibility to perform complex
querying by supporting a set of significant querying features (such as
multi-dimensional, range and aggregate querying) while supporting exact
and partial matching, both for static and dynamic object contents. The
simulation shows that HARD can be applied to arbitrary scales of dynamicity,
both in terms of complexity and of scale, positioning this proposal as a
proper architecture for future many-core systems. We also contributed to
propose a novel resource management scheme for future systems which
efficiently can utilize distributed resources in a fully decentralized fashion.
Moreover, leveraging discovery components (RR-RPs) enables our resource
management platform to dynamically find and allocate available resources
that guarantee the QoS parameters on demand
Agent-based resource management for grid computing
A computational grid is a hardware and software infrastructure that provides
dependable, consistent, pervasive, and inexpensive access to high-end
computational capability. An ideal grid environment should provide access to the
available resources in a seamless manner. Resource management is an important
infrastructural component of a grid computing environment. The overall aim of
resource management is to efficiently schedule applications that need to utilise the
available resources in the grid environment. Such goals within the high
performance community will rely on accurate performance prediction capabilities.
An existing toolkit, known as PACE (Performance Analysis and Characterisation
Environment), is used to provide quantitative data concerning the performance of
sophisticated applications running on high performance resources. In this thesis an
ASCI (Accelerated Strategic Computing Initiative) kernel application, Sweep3D,
is used to illustrate the PACE performance prediction capabilities. The validation
results show that a reasonable accuracy can be obtained, cross-platform
comparisons can be easily undertaken, and the process benefits from a rapid
evaluation time. While extremely well-suited for managing a locally distributed
multi-computer, the PACE functions do not map well onto a wide-area
environment, where heterogeneity, multiple administrative domains, and communication irregularities dramatically complicate the job of resource
management. Scalability and adaptability are two key challenges that must be
addressed.
In this thesis, an A4 (Agile Architecture and Autonomous Agents) methodology is
introduced for the development of large-scale distributed software systems with
highly dynamic behaviours. An agent is considered to be both a service provider
and a service requestor. Agents are organised into a hierarchy with service
advertisement and discovery capabilities. There are four main performance
metrics for an A4 system: service discovery speed, agent system efficiency,
workload balancing, and discovery success rate.
Coupling the A4 methodology with PACE functions, results in an Agent-based
Resource Management System (ARMS), which is implemented for grid
computing. The PACE functions supply accurate performance information (e. g.
execution time) as input to a local resource scheduler on the fly. At a meta-level,
agents advertise their service information and cooperate with each other to
discover available resources for grid-enabled applications. A Performance
Monitor and Advisor (PMA) is also developed in ARMS to optimise the
performance of the agent behaviours.
The PMA is capable of performance modelling and simulation about the agents in
ARMS and can be used to improve overall system performance. The PMA can
monitor agent behaviours in ARMS and reconfigure them with optimised
strategies, which include the use of ACTs (Agent Capability Tables), limited
service lifetime, limited scope for service advertisement and discovery, agent
mobility and service distribution, etc.
The main contribution of this work is that it provides a methodology and
prototype implementation of a grid Resource Management System (RMS). The
system includes a number of original features that cannot be found in existing
research solutions
- …