197 research outputs found
Optimising Structured P2P Networks for Complex Queries
With network enabled consumer devices becoming increasingly popular, the number of connected devices and available services is growing considerably - with the number of connected devices es- timated to surpass 15 billion devices by 2015. In this increasingly large and dynamic environment it is important that users have a comprehensive, yet efficient, mechanism to discover services.
Many existing wide-area service discovery mechanisms are centralised and do not scale to large numbers of users. Additionally, centralised services suffer from issues such as a single point of failure, high maintenance costs, and difficulty of management. As such, this Thesis seeks a Peer to Peer (P2P) approach.
Distributed Hash Tables (DHTs) are well known for their high scalability, financially low barrier of entry, and ability to self manage. They can be used to provide not just a platform on which peers can offer and consume services, but also as a means for users to discover such services.
Traditionally DHTs provide a distributed key-value store, with no search functionality. In recent years many P2P systems have been proposed providing support for a sub-set of complex query types, such as keyword search, range queries, and semantic search.
This Thesis presents a novel algorithm for performing any type of complex query, from keyword search, to complex regular expressions, to full-text search, over any structured P2P overlay. This is achieved by efficiently broadcasting the search query, allowing each peer to process the query locally, and then efficiently routing responses back to the originating peer. Through experimentation, this technique is shown to be successful when the network is stable, however performance degrades under high levels of network churn.
To address the issue of network churn, this Thesis proposes a number of enhancements which can be made to existing P2P overlays in order to improve the performance of both the existing DHT and the proposed algorithm. Through two case studies these enhancements are shown to improve not only the performance of the proposed algorithm under churn, but also the performance of traditional lookup operations in these networks
PariMulo: Kad
With the advent of broadband connections and computing power available in every kind of digital equipment there is a need to share resources, such as information, among people. To fulfill this need in these years we have seen an amazing growth of distributed systems: cloud computing services, web applications, peer-to-peer systems, etc.
In this context is born PariPari, a project which aims to build a modern peer-to-peer network of computers that runs various services, among which there is an eMule-compatible client, called PariMulo. As it is well known even to less computer-savvy people, there have been some problems with the centralized server-based structure of the original eDonkey network, and it has helped the development of a new network, Kad, based upon the Kademlia protocol.
This work focuses on the implementation of Kad in PariMulo, starting by first describing the protocol and how the network works, and then providing an in-depth vision of the implementation considering security and performance issues. Finally we make some observations about future possibilities of developmen
A Brief History of Web Crawlers
Web crawlers visit internet applications, collect data, and learn about new
web pages from visited pages. Web crawlers have a long and interesting history.
Early web crawlers collected statistics about the web. In addition to
collecting statistics about the web and indexing the applications for search
engines, modern crawlers can be used to perform accessibility and vulnerability
checks on the application. Quick expansion of the web, and the complexity added
to web applications have made the process of crawling a very challenging one.
Throughout the history of web crawling many researchers and industrial groups
addressed different issues and challenges that web crawlers face. Different
solutions have been proposed to reduce the time and cost of crawling.
Performing an exhaustive crawl is a challenging question. Additionally
capturing the model of a modern web application and extracting data from it
automatically is another open question. What follows is a brief history of
different technique and algorithms used from the early days of crawling up to
the recent days. We introduce criteria to evaluate the relative performance of
web crawlers. Based on these criteria we plot the evolution of web crawlers and
compare their performanc
Resource discovery for distributed computing systems: A comprehensive survey
Large-scale distributed computing environments provide a vast amount of heterogeneous computing resources from different sources for resource sharing and distributed computing. Discovering appropriate resources in such environments is a challenge which involves several different subjects. In this paper, we provide an investigation on the current state of resource discovery protocols, mechanisms, and platforms for large-scale distributed environments, focusing on the design aspects. We classify all related aspects, general steps, and requirements to construct a novel resource discovery solution in three categories consisting of structures, methods, and issues. Accordingly, we review the literature, analyzing various aspects for each category
CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines
Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective.
The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines.
From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research
Recommended from our members
HARD: Hybrid Adaptive Resource Discovery for Jungle Computing
In recent years, Jungle Computing has emerged as a distributed computing paradigm based on simultaneous combination of various hierarchical and distributed computing environments which are composed by large number of heterogeneous resources. In such a computing environment, the resources and the underlying computation and communication infrastructures are highly-hierarchical and heterogeneous. This creates a lot of difficulty and complexity for finding the proper resources in a precise way in order to run a particular job on the system efficiently. This paper proposes Hybrid Adaptive Resource Discovery (HARD), a novel efficient and highly scalable resource-discovery approach which is built upon a virtual hierarchical overlay based on self-organization and self-adaptation of processing resources in the system, where the computing resources are organized into distributed hierarchies according to a proposed hierarchical multi-layered resource description model. The proposed approach supports distributed query processing within and across hierarchical layers by deploying various distributed resource discovery services and functionalities in the system which are implemented using different adapted algorithms and mechanisms in each level of hierarchy. The proposed approach addresses the requirements for resource discovery in Jungle Computing environments such as high-hierarchy, high-heterogeneity, high-scalability and dynamicity. Simulation results show significant scalability and efficiency of the proposed approach over highly heterogeneous, hierarchical and dynamic computing environments
Content Distribution in P2P Systems
The report provides a literature review of the state-of-the-art for content distribution. The report's contributions are of threefold. First, it gives more insight into traditional Content Distribution Networks (CDN), their requirements and open issues. Second, it discusses Peer-to-Peer (P2P) systems as a cheap and scalable alternative for CDN and extracts their design challenges. Finally, it evaluates the existing P2P systems dedicated for content distribution according to the identied requirements and challenges
Descoberta de recursos para sistemas de escala arbitrarias
Doutoramento em InformĂĄticaTecnologias de Computação DistribuĂda em larga escala tais como Cloud,
Grid, Cluster e Supercomputadores HPC estĂŁo a evoluir juntamente com a
emergĂȘncia revolucionĂĄria de modelos de mĂșltiplos nĂșcleos (por exemplo:
GPU, CPUs num Ășnico die, Supercomputadores em single die, Supercomputadores
em chip, etc) e avanços significativos em redes e soluçÔes de
interligação. No futuro, nĂłs de computação com milhares de nĂșcleos podem
ser ligados entre si para formar uma Ășnica unidade de computação
transparente que esconde das aplicaçÔes a complexidade e a natureza distribuĂda desses sistemas com mĂșltiplos nĂșcleos. A fim de beneficiar de forma
eficiente de todos os potenciais recursos nesses ambientes de computação
em grande escala com mĂșltiplos nĂșcleos ativos, a descoberta de recursos Ă© um elemento crucial para explorar ao mĂĄximo as capacidade de todos
os recursos heterogĂ©neos distribuĂdos, atravĂ©s do reconhecimento preciso e
localização desses recursos no sistema. A descoberta eficiente e escalåvel
de recursos ÂŽe um desafio para tais sistemas futuros, onde os recursos e as
infira-estruturas de computação e comunicação subjacentes são altamente
dinùmicas, hierarquizadas e heterogéneas. Nesta tese, investigamos o problema
da descoberta de recursos no que diz respeito aos requisitos gerais da
escalabilidade arbitrĂĄria de ambientes de computação futuros com mĂșltiplos
nĂșcleos ativos. A principal contribuição desta tese ÂŽe a proposta de uma
entidade de descoberta de recursos adaptativa hĂbrida (Hybrid Adaptive
Resource Discovery - HARD), uma abordagem de descoberta de recursos eficiente
e altamente escalĂĄvel, construĂda sobre uma sobreposição hierĂĄrquica
virtual baseada na auto-organizaçãoo e auto-adaptação de recursos de processamento
no sistema, onde os recursos computacionais sĂŁo organizados
em hierarquias distribuĂdas de acordo com uma proposta de modelo de
descriçãoo de recursos multi-camadas hierårquicas. Operacionalmente, em
cada camada, que consiste numa arquitetura ponto-a-ponto de mĂłdulos que,
interagindo uns com os outros, fornecem uma visĂŁo global da disponibilidade
de recursos num ambiente distribuĂdo grande, dinĂąmico e heterogĂ©neo. O
modelo de descoberta de recursos proposto fornece a adaptabilidade e flexibilidade
para executar consultas complexas através do apoio a um conjunto
de caracterĂsticas significativas (tais como multi-dimensional, variedade e
consulta agregada) apoiadas por uma correspondĂȘncia exata e parcial, tanto
para o conteĂșdo de objetos estĂ©ticos e dinĂąmicos. SimulaçÔes mostram
que o HARD pode ser aplicado a escalas arbitrĂĄrias de dinamismo, tanto
em termos de complexidade como de escala, posicionando esta proposta
como uma arquitetura adequada para sistemas futuros de mĂșltiplos nĂșcleos.
TambĂ©m contribuĂmos com a proposta de um regime de gestĂŁo eficiente
dos recursos para sistemas futuros que podem utilizar recursos distribuĂos
de forma eficiente e de uma forma totalmente descentralizada. Além disso,
aproveitando componentes de descoberta (RR-RPs) permite que a nossa
plataforma de gestĂŁo de recursos encontre e aloque dinamicamente recursos
disponĂeis que garantam os parĂąmetros de QoS pedidos.Large scale distributed computing technologies such as Cloud, Grid, Cluster
and HPC supercomputers are progressing along with the revolutionary emergence
of many-core designs (e.g. GPU, CPUs on single die, supercomputers
on chip, etc.) and significant advances in networking and interconnect solutions.
In future, computing nodes with thousands of cores may be connected
together to form a single transparent computing unit which hides from applications
the complexity and distributed nature of these many core systems. In
order to efficiently benefit from all the potential resources in such large scale
many-core-enabled computing environments, resource discovery is the vital
building block to maximally exploit the capabilities of all distributed heterogeneous
resources through precisely recognizing and locating those resources
in the system. The efficient and scalable resource discovery is challenging for
such future systems where the resources and the underlying computation and
communication infrastructures are highly-dynamic, highly-hierarchical and
highly-heterogeneous. In this thesis, we investigate the problem of resource
discovery with respect to the general requirements of arbitrary scale future
many-core-enabled computing environments. The main contribution of this
thesis is to propose Hybrid Adaptive Resource Discovery (HARD), a novel
efficient and highly scalable resource-discovery approach which is built upon
a virtual hierarchical overlay based on self-organization and self-adaptation
of processing resources in the system, where the computing resources are
organized into distributed hierarchies according to a proposed hierarchical
multi-layered resource description model. Operationally, at each layer, it
consists of a peer-to-peer architecture of modules that, by interacting with
each other, provide a global view of the resource availability in a large,
dynamic and heterogeneous distributed environment. The proposed resource
discovery model provides the adaptability and flexibility to perform complex
querying by supporting a set of significant querying features (such as
multi-dimensional, range and aggregate querying) while supporting exact
and partial matching, both for static and dynamic object contents. The
simulation shows that HARD can be applied to arbitrary scales of dynamicity,
both in terms of complexity and of scale, positioning this proposal as a
proper architecture for future many-core systems. We also contributed to
propose a novel resource management scheme for future systems which
efficiently can utilize distributed resources in a fully decentralized fashion.
Moreover, leveraging discovery components (RR-RPs) enables our resource
management platform to dynamically find and allocate available resources
that guarantee the QoS parameters on demand
- âŠ