A Middleware framework for self-adaptive large scale distributed services

Modern service-oriented applications demand the ability to adapt to changing conditions and unexpected situations while maintaining a required QoS. Existing self-adaptation approaches seem inadequate to address this challenge because many of their assumptions are not met on the large-scale, highly dynamic infrastructures where these applications are generally deployed on. The main motivation of our research is to devise principles that guide the construction of large scale self-adaptive distributed services. We aim to provide sound modeling abstractions based on a clear conceptual background, and their realization as a middleware framework that supports the development of such services. Taking the inspiration from the concepts of decentralized markets in economics, we propose a solution based on three principles: emergent self-organization, utility driven behavior and model-less adaptation. Based on these principles, we designed Collectives, a middleware framework which provides a comprehensive solution for the diverse adaptation concerns that rise in the development of distributed systems. We tested the soundness and comprehensiveness of the Collectives framework by implementing eUDON, a middleware for self-adaptive web services, which we then evaluated extensively by means of a simulation model to analyze its adaptation capabilities in diverse settings. We found that eUDON exhibits the intended properties: it adapts to diverse conditions like peaks in the workload and massive failures, maintaining its QoS and using efficiently the available resources; it is highly scalable and robust; can be implemented on existing services in a non-intrusive way; and do not require any performance model of the services, their workload or the resources they use. We can conclude that our work proposes a solution for the requirements of self-adaptation in demanding usage scenarios without introducing additional complexity. In that sense, we believe we make a significant contribution towards the development of future generation service-oriented applications.Las Aplicaciones Orientadas a Servicios modernas demandan la capacidad de adaptarse a condiciones variables y situaciones inesperadas mientras mantienen un cierto nivel de servio esperado (QoS). Los enfoques de auto-adaptación existentes parecen no ser adacuados debido a sus supuestos no se cumplen en infrastructuras compartidas de gran escala. La principal motivación de nuestra investigación es inerir un conjunto de principios para guiar el desarrollo de servicios auto-adaptativos de gran escala. Nuesto objetivo es proveer abstraciones de modelaje apropiadas, basadas en un marco conceptual claro, y su implemetnacion en un middleware que soporte el desarrollo de estos servicios. Tomando como inspiración conceptos económicos de mercados decentralizados, hemos propuesto una solución basada en tres principios: auto-organización emergente, comportamiento guiado por la utilidad y adaptación sin modelos. Basados en estos principios diseñamos Collectives, un middleware que proveer una solución exhaustiva para los diversos aspectos de adaptación que surgen en el desarrollo de sistemas distribuidos. La adecuación y completitud de Collectives ha sido provada por medio de la implementación de eUDON, un middleware para servicios auto-adaptativos, el ha sido evaluado de manera exhaustiva por medio de un modelo de simulación, analizando sus propiedades de adaptación en diversos escenarios de uso. Hemos encontrado que eUDON exhibe las propiedades esperadas: se adapta a diversas condiciones como picos en la carga de trabajo o fallos masivos, mateniendo su calidad de servicio y haciendo un uso eficiente de los recusos disponibles. Es altamente escalable y robusto; puedeoo ser implementado en servicios existentes de manera no intrusiva; y no requiere la obtención de un modelo de desempeño para los servicios. Podemos concluir que nuestro trabajo nos ha permitido desarrollar una solucion que aborda los requerimientos de auto-adaptacion en escenarios de uso exigentes sin introducir complejidad adicional. En este sentido, consideramos que nuestra propuesta hace una contribución significativa hacia el desarrollo de la futura generación de aplicaciones orientadas a servicios.Postprint (published version

An Efficient Holistic Data Distribution and Storage Solution for Online Social Networks

In the past few years, Online Social Networks (OSNs) have dramatically spread over the world. Facebook [4], one of the largest worldwide OSNs, has 1.35 billion users, 82.2% of whom are outside the US [36]. The browsing and posting interactions (text content) between OSN users lead to user data reads (visits) and writes (updates) in OSN datacenters, and Facebook now serves a billion reads and tens of millions of writes per second [37]. Besides that, Facebook has become one of the top Internet traffic sources [36] by sharing tremendous number of large multimedia files including photos and videos. The servers in datacenters have limited resources (e.g. bandwidth) to supply latency efficient service for multimedia file sharing among the rapid growing users worldwide. Most online applications operate under soft real-time constraints (e.g., ≤ 300 ms latency) for good user experience, and its service latency is negatively proportional to its income. Thus, the service latency is a very important requirement for Quality of Service (QoS) to the OSN as a web service, since it is relevant to the OSN’s revenue and user experience. Also, to increase OSN revenue, OSN service providers need to constrain capital investment, operation costs, and the resource (bandwidth) usage costs. Therefore, it is critical for the OSN to supply a guaranteed QoS for both text and multimedia contents to users while minimizing its costs. To achieve this goal, in this dissertation, we address three problems. i) Data distribution among datacenters: how to allocate data (text contents) among data servers with low service latency and minimized inter-datacenter network load; ii) Efficient multimedia file sharing: how to facilitate the servers in datacenters to efficiently share multimedia files among users; iii) Cost minimized data allocation among cloud storages: how to save the infrastructure (datacenters) capital investment and operation costs by leveraging commercial cloud storage services. Data distribution among datacenters. To serve the text content, the new OSN model, which deploys datacenters globally, helps reduce service latency to worldwide distributed users and release the load of the existing datacenters. However, it causes higher inter-datacenter communica-tion load. In the OSN, each datacenter has a full copy of all data, and the master datacenter updates all other datacenters, generating tremendous load in this new model. The distributed data storage, which only stores a user’s data to his/her geographically closest datacenters, simply mitigates the problem. However, frequent interactions between distant users lead to frequent inter-datacenter com-munication and hence long service latencies. Therefore, the OSNs need a data allocation algorithm among datacenters with minimized network load and low service latency. Efficient multimedia file sharing. To serve multimedia file sharing with rapid growing user population, the file distribution method should be scalable and cost efficient, e.g. minimiza-tion of bandwidth usage of the centralized servers. The P2P networks have been widely used for file sharing among a large amount of users [58, 131], and meet both scalable and cost efficient re-quirements. However, without fully utilizing the altruism and trust among friends in the OSNs, current P2P assisted file sharing systems depend on strangers or anonymous users to distribute files that degrades their performance due to user selfish and malicious behaviors. Therefore, the OSNs need a cost efficient and trustworthy P2P-assisted file sharing system to serve multimedia content distribution. Cost minimized data allocation among cloud storages. The new trend of OSNs needs to build worldwide datacenters, which introduce a large amount of capital investment and maintenance costs. In order to save the capital expenditures to build and maintain the hardware infrastructures, the OSNs can leverage the storage services from multiple Cloud Service Providers (CSPs) with existing worldwide distributed datacenters [30, 125, 126]. These datacenters provide different Get/Put latencies and unit prices for resource utilization and reservation. Thus, when se-lecting different CSPs’ datacenters, an OSN as a cloud customer of a globally distributed application faces two challenges: i) how to allocate data to worldwide datacenters to satisfy application SLA (service level agreement) requirements including both data retrieval latency and availability, and ii) how to allocate data and reserve resources in datacenters belonging to different CSPs to minimize the payment cost. Therefore, the OSNs need a data allocation system distributing data among CSPs’ datacenters with cost minimization and SLA guarantee. In all, the OSN needs an efficient holistic data distribution and storage solution to minimize its network load and cost to supply a guaranteed QoS for both text and multimedia contents. In this dissertation, we propose methods to solve each of the aforementioned challenges in OSNs. Firstly, we verify the benefits of the new trend of OSNs and present OSN typical properties that lay the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters (SD3) to allocate user data among geographical distributed datacenters. In SD3,a datacenter jointly considers update rate and visit rate to select user data for replication, and further atomizes a user’s different types of data (e.g., status update, friend post) for replication, making sure that a replica always reduces inter-datacenter communication. Secondly, we analyze a BitTorrent file sharing trace, which proves the necessity of proximity-and interest-aware clustering. Based on the trace study and OSN properties, to address the second problem, we propose a SoCial Network integrated P2P file sharing system for enhanced Efficiency and Trustworthiness (SOCNET) to fully and cooperatively leverage the common-interest, geographically-close and trust properties of OSN friends. SOCNET uses a hierarchical distributed hash table (DHT) to cluster common-interest nodes, and then further clusters geographically close nodes into a subcluster, and connects the nodes in a subcluster with social links. Thus, when queries travel along trustable social links, they also gain higher probability of being successfully resolved by proximity-close nodes, simultaneously enhancing efficiency and trustworthiness. Thirdly, to handle the third problem, we model the cost minimization problem under the SLA constraints using integer programming. According to the system model, we propose an Eco-nomical and SLA-guaranteed cloud Storage Service (ES3), which finds a data allocation and resource reservation schedule with cost minimization and SLA guarantee. ES3 incorporates (1) a data al-location and reservation algorithm, which allocates each data item to a datacenter and determines the reservation amount on datacenters by leveraging all the pricing policies; (2) a genetic algorithm based data allocation adjustment approach, which makes data Get/Put rates stable in each data-center to maximize the reservation benefit; and (3) a dynamic request redirection algorithm, which dynamically redirects a data request from an over-utilized datacenter to an under-utilized datacenter with sufficient reserved resource when the request rate varies greatly to further reduce the payment. Finally, we conducted trace driven experiments on a distributed testbed, PlanetLab, and real commercial cloud storage (Amazon S3, Windows Azure Storage and Google Cloud Storage) to demonstrate the efficiency and effectiveness of our proposed systems in comparison with other systems. The results show that our systems outperform others in the network savings and data distribution efficiency

Content Sharing in Mobile Networks with Infrastructure: Planning and Management

This thesis focuses on mobile ad-hoc networks (with pedestrian or vehicular mobility) having infrastructure support. We deal with the problems of design, deployment and management of such networks. A first issue to address concerns infrastructure itself: how pervasive should it be in order for the network to operate at the same time efficiently and in a cost-effective manner? How should the units composing it (e.g., access points) be placed? There are several approaches to such questions in literature, and this thesis studies and compares them. Furthermore, in order to effectively design the infrastructure, we need to understand how and how much it will be used. As an example, what is the relationship between infrastructure-to-node and node-to-node communication? How far away, in time and space, do data travel before its destination is reached? A common assumption made when dealing with such problems is that perfect knowledge about the current and future node mobility is available. In this thesis, we also deal with the problem of assessing the impact that an imperfect, limited knowledge has on network performance. As far as the management of the network is concerned, this thesis presents a variant of the paradigm known as publish-and-subscribe. With respect to the original paradigm, our goal was to ensure a high probability of finding the requested content, even in presence of selfish, uncooperative nodes, or even nodes whose precise goal is harming the system. Each node is allowed to get from the network an amount of content which corresponds to the amount of content provided to other nodes. Nodes with caching capabilities are assisted in using their cache in order to improve the amount of offered conten

The Case for Liberal Spectrum Licenses: A Technical and Economic Perspective

The traditional system of radio spectrum allocation has inefficiently restricted wireless services. Alternatively, liberal licenses ceding de facto spectrum ownership rights yield incentives for operators to maximize airwave value. These authorizations have been widely used for mobile services in the U.S. and internationally, leading to the development of highly productive services and waves of innovation in technology, applications and business models. Serious challenges to the efficacy of such a spectrum regime have arisen, however. Seeing the widespread adoption of such devices as cordless phones and wi-fi radios using bands set aside for unlicensed use, some scholars and policy makers posit that spectrum sharing technologies have become cheap and easy to deploy, mitigating airwave scarcity and, therefore, the utility of exclusive rights. This paper evaluates such claims technically and economically. We demonstrate that spectrum scarcity is alive and well. Costly conflicts over airwave use not only continue, but have intensified with scientific advances that dramatically improve the functionality of wireless devices and so increase demand for spectrum access. Exclusive ownership rights help direct spectrum inputs to where they deliver the highest social gains, making exclusive property rules relatively more socially valuable. Liberal licenses efficiently accommodate rival business models (including those commonly associated with unlicensed spectrum allocations) while mitigating the constraints levied on spectrum use by regulators imposing restrictions in traditional licenses or via use rules and technology standards in unlicensed spectrum allocations.

Private and censorship-resistant communication over public networks

Society’s increasing reliance on digital communication networks is creating unprecedented opportunities for wholesale surveillance and censorship. This thesis investigates the use of public networks such as the Internet to build robust, private communication systems that can resist monitoring and attacks by powerful adversaries such as national governments. We sketch the design of a censorship-resistant communication system based on peer-to-peer Internet overlays in which the participants only communicate directly with people they know and trust. This ‘friend-to-friend’ approach protects the participants’ privacy, but it also presents two significant challenges. The first is that, as with any peer-to-peer overlay, the users of the system must collectively provide the resources necessary for its operation; some users might prefer to use the system without contributing resources equal to those they consume, and if many users do so, the system may not be able to survive. To address this challenge we present a new game theoretic model of the problem of encouraging cooperation between selfish actors under conditions of scarcity, and develop a strategy for the game that provides rational incentives for cooperation under a wide range of conditions. The second challenge is that the structure of a friend-to-friend overlay may reveal the users’ social relationships to an adversary monitoring the underlying network. To conceal their sensitive relationships from the adversary, the users must be able to communicate indirectly across the overlay in a way that resists monitoring and attacks by other participants. We address this second challenge by developing two new routing protocols that robustly deliver messages across networks with unknown topologies, without revealing the identities of the communication endpoints to intermediate nodes or vice versa. The protocols make use of a novel unforgeable acknowledgement mechanism that proves that a message has been delivered without identifying the source or destination of the message or the path by which it was delivered. One of the routing protocols is shown to be robust to attacks by malicious participants, while the other provides rational incentives for selfish participants to cooperate in forwarding messages

Organisation of foraging in ants

In social insects, foraging is often cooperative, and so requires considerable organisation. In most ants, organisation is a bottom-up process where decisions taken by individuals result in emergent colony level patterns. Individuals base their decisions on their internal state, their past experience, and their environment. By depositing trail pheromones, for example, ants can alter the environment, and thus affect the behaviour of their nestmates. The development of emergent patterns depends on both how individuals affect the environment, and how they react to changes in the environment. Chapters 4 – 9 investigate the role of trail pheromones and route memory in the ant Lasius niger. Route memories can form rapidly and be followed accurately, and when route memories and trail pheromones contradict each other, ants overwhelmingly follow route memories (chapter 4). Route memories and trail pheromones can also interact synergistically, allowing ants to forage faster without sacrificing accuracy (chapter 5). Home range markings also interact with other information sources to affect ant behaviour (chapter 6). Trail pheromones assist experienced ants when facing complex, difficult-to-learn routes (chapter 7). When facing complicated routes, ants deposit more pheromone to assist in navigation and learning (chapter 7). Deposition of trail pheromones is suppressed by ants leaving a marked path (chapter 5), strong pheromone trails (chapter 7) and trail crowding (chapter 8). Colony level ‘decisions’ can be driven by factors other than trail pheromones, such as overcrowding at a food source (chapter 9). Chapter 10 reviews the many roles of trail pheromones in ants. Chapters 11 – 14 focus on the organisation of cooperative food retrieval. Pheidole oxyops workers arrange themselves non-randomly around items to increase transport speeds (chapter 11). Groups of ants will rotate food items to reduce drag (chapter 12). Chapters 13 and 14 encompass the ecology of cooperative transport, and how it has shaped trail pheromone recruitment in P. oxyops and Paratrechina longicornis. Lastly, chapter 15 provide a comprehensive review of cooperative transport in ants and elsewhere

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research