335 research outputs found
Resource discovery for distributed computing systems: A comprehensive survey
Large-scale distributed computing environments provide a vast amount of heterogeneous computing resources from different sources for resource sharing and distributed computing. Discovering appropriate resources in such environments is a challenge which involves several different subjects. In this paper, we provide an investigation on the current state of resource discovery protocols, mechanisms, and platforms for large-scale distributed environments, focusing on the design aspects. We classify all related aspects, general steps, and requirements to construct a novel resource discovery solution in three categories consisting of structures, methods, and issues. Accordingly, we review the literature, analyzing various aspects for each category
Semantic-Based, Scalable, Decentralized and Dynamic Resource Discovery for Internet-Based Distributed System
Resource Discovery (RD) is a key issue in Internet-based distributed sytems such as
grid. RD is about locating an appropriate resource/service type that matches the user's
application requirements. This is very important, as resource reservation and task
scheduling are based on it. Unfortunately, RD in grid is very challenging as resources
and users are distributed, resources are heterogeneous in their platforms, status of the
resources is dynamic (resources can join or leave the system without any prior notice)
and most recently the introduction of a new type of grid called intergrid (grid of grids)
with the use of multi middlewares. Such situation requires an RD system that has rich
interoperability, scalability, decentralization and dynamism features. However,
existing grid RD systems have difficulties to attain these features. Not only that, they
lack the review and evaluation studies, which may highlight the gap in achieving the
required features. Therefore, this work discusses the problem associated with intergrid
RD from two perspectives. First, reviewing and classifying the current grid RD
systems in such a way that may be useful for discussing and comparing them. Second,
propose a novel RD framework that has the aforementioned required RD features. In
the former, we mainly focus on the studies that aim to achieve interoperability in the
first place, which are known as RD systems that use semantic information (semantic
technology). In particular, we classify such systems based on their qualitative use of
the semantic information. We evaluate the classified studies based on their degree of
accomplishment of interoperability and the other RD requirements, and draw the
future research direction of this field. Meanwhile in the latter, we name the new
framework as semantic-based scalable decentralized dynamic RD. The framework
further contains two main components which are service description, and service
registration and discovery models. The earlier consists of a set of ontologies and
services. Ontologies are used as a data model for service description, whereas the
services are to accomplish the description process. The service registration is also based on ontology, where nodes of the service (service providers) are classified to
some classes according to the ontology concepts, which means each class represents a
concept in the ontology. Each class has a head, which is elected among its own class
I
nodes/members. Head plays the role of a registry in its class and communicates with
I
the other heads of the classes in a peer to peer manner during the discovery process.
We further introduce two intelligent agents to automate the discovery process which
are Request Agent (RA) and Description Agent (DA). Eaclj. node is supposed to have
both agents. DA describes the service capabilities based on the ontology, and RA
I
carries the service requests based on the ontology as well. We design a service search
I
algorithm for the RA that starts the service look up from the class of request origin
first, then to the other classes.
We finally evaluate the performance of our framework ~ith extensive simulation
experiments, the result of which confirms the effectiveness of the proposed system in
satisfying the required RD features (interoperability, scalability, decentralization and
dynamism). In short, our main contributions are outlined new key taxonomy for the
semantic-based grid RD studies; an interoperable semantic description RD component
model for intergrid services metadata representation; a semantic distributed registry
architecture for indexing service metadata; and an agent-qased service search and
selection algorithm.
Vll
Recommended from our members
HARD: Hybrid Adaptive Resource Discovery for Jungle Computing
In recent years, Jungle Computing has emerged as a distributed computing paradigm based on simultaneous combination of various hierarchical and distributed computing environments which are composed by large number of heterogeneous resources. In such a computing environment, the resources and the underlying computation and communication infrastructures are highly-hierarchical and heterogeneous. This creates a lot of difficulty and complexity for finding the proper resources in a precise way in order to run a particular job on the system efficiently. This paper proposes Hybrid Adaptive Resource Discovery (HARD), a novel efficient and highly scalable resource-discovery approach which is built upon a virtual hierarchical overlay based on self-organization and self-adaptation of processing resources in the system, where the computing resources are organized into distributed hierarchies according to a proposed hierarchical multi-layered resource description model. The proposed approach supports distributed query processing within and across hierarchical layers by deploying various distributed resource discovery services and functionalities in the system which are implemented using different adapted algorithms and mechanisms in each level of hierarchy. The proposed approach addresses the requirements for resource discovery in Jungle Computing environments such as high-hierarchy, high-heterogeneity, high-scalability and dynamicity. Simulation results show significant scalability and efficiency of the proposed approach over highly heterogeneous, hierarchical and dynamic computing environments
Hoverlay : a peer-to-peer system for on demand sharing of capacity across network applications
EThOS - Electronic Theses Online ServiceGBUnited Kingdo
Hoverlay : a peer-to-peer system for on demand sharing of capacity across network applications
EThOS - Electronic Theses Online ServiceGBUnited Kingdo
Probabilistic best-fit multi-dimensional range query in Self-Organizing Cloud
With virtual machine (VM) technology being increasingly mature, computing resources in modern Cloud systems can be partitioned in fine granularity and allocated on demand with 'pay-as-you-go' model. In this work, we study the resource query and allocation problems in a Self- Organizing Cloud (SOC), where host machines are connected by a peer-to-peer (P2P) overlay network on the Internet. To run a user task in SOC, the requester needs to perform a multi-dimensional range search over the P2P network for locating host machines that satisfy its minimal demand on each type of resources. The multi-dimensional range search problem is known to be challenging as contentions along multiple dimensions could happen in the presence of the uncoordinated analogous queries. Moreover, low resource matching rate may happen while restricting query delay and network traffic. We design a novel resource discovery protocol, namely Proactive Index Diffusion CAN (PID-CAN), which can proactively diffuse resource indexes over the nodes and randomly route query messages among them. Such a protocol is especially suitable for the range query that needs to maximize its best-fit resource shares under possible competition along multiple resource dimensions. Via simulation, we show that PID-CAN could keep stable and optimized searching performance with low query delay and traffic overhead, for various test cases under different distributions of query ranges and competition degrees. It also performs satisfactorily in dynamic node-churning situation. © 2011 IEEE.published_or_final_versionThe 40th International Conference on Parallel Processing (ICPP-2011), Taipei City, Taiwan, 13-16 September 2011. In Proceedings of the 40th ICPP, 2011, p. 763-77
Proof-of-Concept Application - Annual Report Year 1
In this document the Cat-COVITE Application for use in the CATNETS Project is introduced and motivated. Furthermore an introduction to the catallactic middleware and Web Services Agreement (WS-Agreement) concepts is given as a basis for the future work. Requirements for the application of Cat-COVITE with in catallactic systems are analysed. Finally the integration of the Cat-COVITE application and the catallactic middleware is described. --Grid Computing
Descoberta de recursos para sistemas de escala arbitrarias
Doutoramento em InformáticaTecnologias de Computação Distribuída em larga escala tais como Cloud,
Grid, Cluster e Supercomputadores HPC estão a evoluir juntamente com a
emergência revolucionária de modelos de múltiplos núcleos (por exemplo:
GPU, CPUs num único die, Supercomputadores em single die, Supercomputadores
em chip, etc) e avanços significativos em redes e soluções de
interligação. No futuro, nós de computação com milhares de núcleos podem
ser ligados entre si para formar uma única unidade de computação
transparente que esconde das aplicações a complexidade e a natureza distribuída desses sistemas com múltiplos núcleos. A fim de beneficiar de forma
eficiente de todos os potenciais recursos nesses ambientes de computação
em grande escala com múltiplos núcleos ativos, a descoberta de recursos é um elemento crucial para explorar ao máximo as capacidade de todos
os recursos heterogéneos distribuídos, através do reconhecimento preciso e
localização desses recursos no sistema. A descoberta eficiente e escalável
de recursos ´e um desafio para tais sistemas futuros, onde os recursos e as
infira-estruturas de computação e comunicação subjacentes são altamente
dinâmicas, hierarquizadas e heterogéneas. Nesta tese, investigamos o problema
da descoberta de recursos no que diz respeito aos requisitos gerais da
escalabilidade arbitrária de ambientes de computação futuros com múltiplos
núcleos ativos. A principal contribuição desta tese ´e a proposta de uma
entidade de descoberta de recursos adaptativa híbrida (Hybrid Adaptive
Resource Discovery - HARD), uma abordagem de descoberta de recursos eficiente
e altamente escalável, construída sobre uma sobreposição hierárquica
virtual baseada na auto-organizaçãoo e auto-adaptação de recursos de processamento
no sistema, onde os recursos computacionais são organizados
em hierarquias distribuídas de acordo com uma proposta de modelo de
descriçãoo de recursos multi-camadas hierárquicas. Operacionalmente, em
cada camada, que consiste numa arquitetura ponto-a-ponto de módulos que,
interagindo uns com os outros, fornecem uma visão global da disponibilidade
de recursos num ambiente distribuído grande, dinâmico e heterogéneo. O
modelo de descoberta de recursos proposto fornece a adaptabilidade e flexibilidade
para executar consultas complexas através do apoio a um conjunto
de características significativas (tais como multi-dimensional, variedade e
consulta agregada) apoiadas por uma correspondência exata e parcial, tanto
para o conteúdo de objetos estéticos e dinâmicos. Simulações mostram
que o HARD pode ser aplicado a escalas arbitrárias de dinamismo, tanto
em termos de complexidade como de escala, posicionando esta proposta
como uma arquitetura adequada para sistemas futuros de múltiplos núcleos.
Também contribuímos com a proposta de um regime de gestão eficiente
dos recursos para sistemas futuros que podem utilizar recursos distribuíos
de forma eficiente e de uma forma totalmente descentralizada. Além disso,
aproveitando componentes de descoberta (RR-RPs) permite que a nossa
plataforma de gestão de recursos encontre e aloque dinamicamente recursos
disponíeis que garantam os parâmetros de QoS pedidos.Large scale distributed computing technologies such as Cloud, Grid, Cluster
and HPC supercomputers are progressing along with the revolutionary emergence
of many-core designs (e.g. GPU, CPUs on single die, supercomputers
on chip, etc.) and significant advances in networking and interconnect solutions.
In future, computing nodes with thousands of cores may be connected
together to form a single transparent computing unit which hides from applications
the complexity and distributed nature of these many core systems. In
order to efficiently benefit from all the potential resources in such large scale
many-core-enabled computing environments, resource discovery is the vital
building block to maximally exploit the capabilities of all distributed heterogeneous
resources through precisely recognizing and locating those resources
in the system. The efficient and scalable resource discovery is challenging for
such future systems where the resources and the underlying computation and
communication infrastructures are highly-dynamic, highly-hierarchical and
highly-heterogeneous. In this thesis, we investigate the problem of resource
discovery with respect to the general requirements of arbitrary scale future
many-core-enabled computing environments. The main contribution of this
thesis is to propose Hybrid Adaptive Resource Discovery (HARD), a novel
efficient and highly scalable resource-discovery approach which is built upon
a virtual hierarchical overlay based on self-organization and self-adaptation
of processing resources in the system, where the computing resources are
organized into distributed hierarchies according to a proposed hierarchical
multi-layered resource description model. Operationally, at each layer, it
consists of a peer-to-peer architecture of modules that, by interacting with
each other, provide a global view of the resource availability in a large,
dynamic and heterogeneous distributed environment. The proposed resource
discovery model provides the adaptability and flexibility to perform complex
querying by supporting a set of significant querying features (such as
multi-dimensional, range and aggregate querying) while supporting exact
and partial matching, both for static and dynamic object contents. The
simulation shows that HARD can be applied to arbitrary scales of dynamicity,
both in terms of complexity and of scale, positioning this proposal as a
proper architecture for future many-core systems. We also contributed to
propose a novel resource management scheme for future systems which
efficiently can utilize distributed resources in a fully decentralized fashion.
Moreover, leveraging discovery components (RR-RPs) enables our resource
management platform to dynamically find and allocate available resources
that guarantee the QoS parameters on demand
Semantic-Based, Scalable, Decentralized and Dynamic Resource Discovery for Internet-Based Distributed System
Resource Discovery (RD) is a key issue in Internet-based distributed sytems such as
grid. RD is about locating an appropriate resource/service type that matches the user's
application requirements. This is very important, as resource reservation and task
scheduling are based on it. Unfortunately, RD in grid is very challenging as resources
and users are distributed, resources are heterogeneous in their platforms, status of the
resources is dynamic (resources can join or leave the system without any prior notice)
and most recently the introduction of a new type of grid called intergrid (grid of grids)
with the use of multi middlewares. Such situation requires an RD system that has rich
interoperability, scalability, decentralization and dynamism features. However,
existing grid RD systems have difficulties to attain these features. Not only that, they
lack the review and evaluation studies, which may highlight the gap in achieving the
required features. Therefore, this work discusses the problem associated with intergrid
RD from two perspectives. First, reviewing and classifying the current grid RD
systems in such a way that may be useful for discussing and comparing them. Second,
propose a novel RD framework that has the aforementioned required RD features. In
the former, we mainly focus on the studies that aim to achieve interoperability in the
first place, which are known as RD systems that use semantic information (semantic
technology). In particular, we classify such systems based on their qualitative use of
the semantic information. We evaluate the classified studies based on their degree of
accomplishment of interoperability and the other RD requirements, and draw the
future research direction of this field. Meanwhile in the latter, we name the new
framework as semantic-based scalable decentralized dynamic RD. The framework
further contains two main components which are service description, and service
registration and discovery models. The earlier consists of a set of ontologies and
services. Ontologies are used as a data model for service description, whereas the
services are to accomplish the description process. The service registration is also based on ontology, where nodes of the service (service providers) are classified to
some classes according to the ontology concepts, which means each class represents a
concept in the ontology. Each class has a head, which is elected among its own class
I
nodes/members. Head plays the role of a registry in its class and communicates with
I
the other heads of the classes in a peer to peer manner during the discovery process.
We further introduce two intelligent agents to automate the discovery process which
are Request Agent (RA) and Description Agent (DA). Eaclj. node is supposed to have
both agents. DA describes the service capabilities based on the ontology, and RA
I
carries the service requests based on the ontology as well. We design a service search
I
algorithm for the RA that starts the service look up from the class of request origin
first, then to the other classes.
We finally evaluate the performance of our framework ~ith extensive simulation
experiments, the result of which confirms the effectiveness of the proposed system in
satisfying the required RD features (interoperability, scalability, decentralization and
dynamism). In short, our main contributions are outlined new key taxonomy for the
semantic-based grid RD studies; an interoperable semantic description RD component
model for intergrid services metadata representation; a semantic distributed registry
architecture for indexing service metadata; and an agent-qased service search and
selection algorithm.
Vll
- …