Collocation Games and Their Application to Distributed Resource Management

We introduce Collocation Games as the basis of a general framework for modeling, analyzing, and facilitating the interactions between the various stakeholders in distributed systems in general, and in cloud computing environments in particular. Cloud computing enables fixed-capacity (processing, communication, and storage) resources to be offered by infrastructure providers as commodities for sale at a fixed cost in an open marketplace to independent, rational parties (players) interested in setting up their own applications over the Internet. Virtualization technologies enable the partitioning of such fixed-capacity resources so as to allow each player to dynamically acquire appropriate fractions of the resources for unencumbered use. In such a paradigm, the resource management problem reduces to that of partitioning the entire set of applications (players) into subsets, each of which is assigned to fixed-capacity cloud resources. If the infrastructure and the various applications are under a single administrative domain, this partitioning reduces to an optimization problem whose objective is to minimize the overall deployment cost. In a marketplace, in which the infrastructure provider is interested in maximizing its own profit, and in which each player is interested in minimizing its own cost, it should be evident that a global optimization is precisely the wrong framework. Rather, in this paper we use a game-theoretic framework in which the assignment of players to fixed-capacity resources is the outcome of a strategic "Collocation Game". Although we show that determining the existence of an equilibrium for collocation games in general is NP-hard, we present a number of simplified, practically-motivated variants of the collocation game for which we establish convergence to a Nash Equilibrium, and for which we derive convergence and price of anarchy bounds. In addition to these analytical results, we present an experimental evaluation of implementations of some of these variants for cloud infrastructures consisting of a collection of multidimensional resources of homogeneous or heterogeneous capacities. Experimental results using trace-driven simulations and synthetically generated datasets corroborate our analytical results and also illustrate how collocation games offer a feasible distributed resource management alternative for autonomic/self-organizing systems, in which the adoption of a global optimization approach (centralized or distributed) would be neither practical nor justifiable.NSF (CCF-0820138, CSR-0720604, EFRI-0735974, CNS-0524477, CNS-052016, CCR-0635102); Universidad Pontificia Bolivariana; COLCIENCIAS–Instituto Colombiano para el Desarrollo de la Ciencia y la Tecnología "Francisco José de Caldas

On social and technical aspects of managing mobile Ad-hoc communities

Soziale Software beschreibt eine Klasse von Anwendungen, die es Benutzern erlaubt ueber das Internet mit Freunden zu kommunizieren und Informationen auszutauschen. Mit zunehmender Leistungsfaehigkeit mobiler Prozessoren verwandeln sich Mobiltelefone in vollwertige Computer und eroeffnen neue Moeglichkeiten fuer die mobile Nutzung sozialer Software. Da Menschen Mobiltelefone haeufig bei sich fuehren, koennen vergleichbare mobile Anwendungen staerker auf ihre unmittelbare Umgebungssituation zugeschnitten werden. Moegliche Szenarien sind die Unterstuetzung realer Treffen und damit verbundenen Mitgliederinteraktionen. Client-Server-Plattformen, die dabei haeufig zum Einsatz kommen wurden allerdings nie fuer solche hochflexiblen Gruppensituationen konstruiert. Mobile Encounter Netzwerke (MENe) verprechen hier mehr Flexibilitaet. Ein MEN stellt eine mobiler Peer-to-Peer-Plattformen dar, das ueber ein kurzreichweitiges Funknetz betrieben wird. Mit diesem Netzwerk werden Beitraege ueber einen raeumlichen Diffusionsprozess von einem mobilen Endgeraet zum naechsten verbreitet. Das hat zwei entscheidende Vorteile: Zunaechst ist der direkte Nachrichtenaustausch besser geeignet zur Verbreitung von situationsspezifischer Information, da die Informationsrelevanz mit ihrer Entfehrnung abnimmt. Gleichzeitig koennen aber auch Inhalte, die fuer einen breiten Interessenkreis bestimmt sind ueber Mitglieder mit herausragenden Mobilitaetscharakteristik in weit entfernte Gebiete transportiert werden. Ein Nachteil ist jedoch der hohe Ressourcenverbrauch. Zur Loesung dieses Problems entwickeln wir ein Rahmenwerk zur Unterstuetzung mobiler ad-hoc Gruppen, das es uns erlaubt, Gruppensynergien gezielt auszunutzen. Dieses Rahmenwerk bietet Dienstleistungen zur Verwaltung der Gruppendynamik und zur Verbreitung von Inhalten an. Mittels soziale Netzwerkanalyse wird die technische Infrastruktur ohne notwendige Benutzereingriffe kontinuierlich an die reale Umgebungssituation angepasst. Dabei werden moegliche Beziehungen zwischen benachbarten Personen anhand frueher Begegnungen analysiert, spontane Gruppenbildungen mit Clusterverfahren identifiziert und jedem Gruppenmitglied eine geeignete Rolle durch eine Positionsanalyse zugewiesen. Eine Grundvorraussetzung fuer eine erfolgreiche Kooperation ist ein effizienter Wissensaustausch innerhalb einer Gemeinschaft. Wie die Small World-Theorie zeigt, koennen Menschen Wissen auch dann effizient verbreiten, wenn ihre Entscheidung nur auf lokaler Umgebungsinformation basiert. Verschiedene Forscher machten sich das zu nutze, indem sie kurze Verbreitungspfade durch eine Verkettung hochvernetzter Mitglieder innerhalb einer Gemeinschaft konstruierten. Allerdings laesst sich dieses Verfahren nicht einfach auf MENe uebertragen, da die Transferzeit im Gegensatz zu dem drahtgebundenen Internet beschraenkt ist. Unser Ansatz beruht daher, auf der von Reagan et al. vorgestellten Least Effort Transfer-Hypothese. Diese Hypothese besagt, dass Menschen Wissen nur dann weitergeben, wenn sich der Aufwand zur Informationsuebertragung innerhalb bestimmter Grenzen bewegt. Eine erfolgreiche Wissensuebertragung haengt in diesem Fall vom Hintergrundwissen aller Beteiligter ab, was wiederum von unterschiedlichen kognitiven und sozialen Faktoren abhaengt. Entsprechend leiten wir ein Diffusionsverfahren ab, dass in der Lage ist, Inhalte in verschiedene Kompexitaetstufen einzuteilen und Datenuebertragungen an die vorgefundene soziale Situation anzupassen. Mit einem Prototyp evaluieren wir die Machbarkeit der Gruppen- und Informationsmanagementkomponente unseres Rahmenwerkes. Da Laborexperimente keinen ausreichenden Aufschluss ueber Diffusionseigenschaften im groesseren Massstab geben koennen, simulieren wir die Beitragsdiffusion. Dazu dient uns eine Verkehrsimulation, bei der Agenten zusaetzlich mit aktivitaetsbezogenen, sozialen und territorialen Modellen erweitern werden. Um eine realitaetsnahe Simulation zu gewaehrleisten, werden diese Modelle in Uebereinstimmung mit verschiedenen Studien zum Stadtleben generiert. Der technische Uebertragungsprozess wird anhand der Ergebnisse einer vorangegangenen Prototypuntersuchung parametrisiert. Waehrend eines Simulationslaufes bewegen sich Agenten auf einem Stadtplan und sammeln Kontakt- und Beitragsdaten. Analysiert man anschliessend die Netzwerktopologie auf Small World-Eigenschaften, so findet man eine Netzstruktur mit einer ausgepraegten Neigung zum Clustering (Freundschaftsnetzwerke) und einer ueberdurschnittlichen kurzen Weglaenge. Offensichtlich reicht die Alltagsmobilitaet aus, um ausreichend viele Verknuepfungen zwischen Gemeinschaftmitgliedern zu bilden. Die nachfolgende Diffusionsanalyse zeigt, dass vergleichbare Reichweiten wie bei einem flutungsbasierten Ansatz erzielt werden, allerdings mit anfaenglichen Verzoegerungen. Da unser Verfahren bei einem Ortswechsel die Anzahl der Informationsuebermittler auf zentrale Gruppenmitglieder begrenzt, steht mehr Bandbreite fuer den Datenaustausch zur Verfuegung. Herkoemliche Mitglieder (ohne Leitungsaufgaben) tauschen Inhalte vornehmlich in zeitunkritschen Situationen aus. Das hat den positiven Nebeneffekt, dass im Cache erheblich weniger Kopien aussortiert werden muessen. Wechselt man waehrend der Simulation die Beitragskategorie so erkennt man, dass zeitabhaengige Inhalte besser ueber regelmaessige Kontakte und zeitunabhaengig Inhalte durch zufaellige Kontakte verbreitet werden. Eine abschliessende Precision-Recall Analyse zeigt, dass herkoemmliche Gruppenmitglieder eine bessere Genauigkeit (Precision), und zentrale Mitglieder eine bessere Trefferquote (Recall) im Vergleich zu traditionellen Ansaetzen besitzen. Eine Erklaerung dafuer ist, dass der von uns gewaehlte gruppenbasierte Cacheansatz zu weniger Saeuberungszyklen aller Gruppenmitglieder fuehrt und somit nachhaltiger ausgerichtet ist.Social software encompasses a range of software systems that allow users to interact and share data. This computer-mediated communication has become very popular with social networking sites like Facebook and Twitter. The evolvement of smart phones toward mobile computers opens new possibilities to use social software also in mobile usage scenarios. Since mobile phones are permanently carried by their owners, the support focus is, however, much stronger set on promoting and augmenting real group gatherings. Traditional client-server platforms are not flexible enough to support complex and dynamic human encounter behavior. Mobile encounter networks (MENs) which represent a mobile peer-to-peer platform on top of a short range wireless network promise better flexibility. MENs diffuse content from neighbor-to-neighbor in a spatial diffusion process. For physical group gatherings this is advantageous for two reasons. Direct device-to-device interactions encourage sharing of situation-dependent content. Moreover, content is not necessarily locked within friend groups and may trigger networking effects by reaching larger audiences through user mobility. One disadvantage is, however, the high resource usage. We develop a social software framework for mobile ad-hoc groups, which partly solves this problem. This framework supports services for the management of group dynamics and content diffusion within and between groups. Social network analysis as an inherent part of the framework is used to adapt internal community states continuously with real world encounter situations. We hereby qualify interpersonal relationships based on encounter and communication statistics, identify social groups through incremental clustering and assign diffusion roles through position analysis. To achieve efficient content dissemination we make use of social diffusion phenomena. Other researchers have experimented extensively with the small world model as it proofs that people transfer knowledge based on local knowledge but are still capable of diffusing it efficiently on a global scale. Their approach is often based on identifying short paths through member connectivity. However, this scenario is not applicable in MENs as transfer time is limited in contrast to the wired Internet. Our approach is therefore based on the least effort transfer theory. Following Reagan et al., who first postulated this hypothesis, people transfer knowledge only if the transfer effort is within specific limits, which depends on different social and cognitive factors. We derive routing mechanisms, which are capable of distinguishing between different content complexities and apply information about peer's expertise and social network to identify advantageous paths and content transfers options. We evaluate the feasibility of the group management and content transfer component with prototypes. Since labor settings do not allow to obtain information about large scale diffusion experiences, we also conduct a multi-agent simulation to evaluate the diffusion capabilities of the system. Experiences from an earlier prototype implementation have been used to quantify the technical routing process. To emulate realistic community life, we assigned to each agent an individual daily agenda, social contacts and territory preferences specified according to outcomes from different urban city life surveys. During the simulation agents move on a city map according to these models and collect contact and content specific data. Analyzing the network topology according to small world characteristics shows a structure with a high tendency for clustering (friend networks) and a short average path length. Daily urban mobility creates enough opportunities to form shortcuts through the community. Content diffusion analysis shows that our approach reaches a similar amount of peers as network flooding but with delays in the beginning. Since our approach artificially limits the number of intermediates to central community peers more bandwidth is available during traveling and more content can be transferred as in the case of the flooding approach. Ordinary peers seem to have significantly fewer content replications if an unlimited cache is assumed proofing that our mechanism is more efficient. By varying the content type used during the simulation we recognize that time dependent content is better disseminated through frequent contacts and time independent content through random contacts. Performing a precision-recall analysis on peers caches shows that ordinary peers gain an overall better context precision, and central peers a better community recall. One explanation is that the shared cache approach leads to fewer content replacements in the cache as for instance the least recently used cache strategy

A systematic survey of online data mining technology intended for law enforcement

As an increasing amount of crime takes on a digital aspect, law enforcement bodies must tackle an online environment generating huge volumes of data. With manual inspections becoming increasingly infeasible, law enforcement bodies are optimising online investigations through data-mining technologies. Such technologies must be well designed and rigorously grounded, yet no survey of the online data-mining literature exists which examines their techniques, applications and rigour. This article remedies this gap through a systematic mapping study describing online data-mining literature which visibly targets law enforcement applications, using evidence-based practices in survey making to produce a replicable analysis which can be methodologically examined for deficiencies

Partitioning and Offloading for IoT and Video Streaming Applications that Utilize Computing Resources at the Network Edge

The Internet of Things (IoT) is a concept in which physical objects embedded with sensors, actuators, and network connectivity can communicate and react to their surroundings. IoT applications connect physical objects for the purpose of decision making by sensing and analysing generated data from the embedded sensors in physical objects. IoT applications are growing rapidly as sensors become less expensive. Sensors generate large amounts of data that may meaningless unless the data is used to derive knowledge with in a certain period of time. Stream processing paradigm is used by IoT to provide requirements of IoT applications. In a stream processing paradigm, unlike traditional data bases, data is not stored but rather processed as it is generated. To transfer generated data from distributed data sources to a processing center such as cloud may not allow for real-time processing due to the network delay. Another high-demand application is live streaming of video. The performance of live video stream systems is inferior when there is a sudden large demand in the number of users. This thesis addresses some of the limitations of current architectures for video streaming systems and IoT applications based on the use of nearby computing resources (e.g., cloudlet, fog). First, we addressed the degrading performance in video stream systems when a flash crowd occurs. The performance of video streaming systems is affected by flash crowd and degrade the quality of service for subscribers to the content delivery system. A flash crowd happens when there is a sudden large increase in the number of users. Therefore, flash crowds increase network traffic for any particular server. The main challenge is to make sure that the video streaming system has sufficient capacity to handle the occurrence of flash crowds. Second, we address the limitation of current architectures for running mobile applications by introducing a dynamic partitioning and offloading of a mobile application. Mobile devices have limited resources including short battery life, storage capacity and processor performance. This limits the applications that can run on it. Mobile applications can be partitioned so that some of the application runs on a cloud. This works well for applications with relatively little data to be transferred and that do not have a high level of interaction with the user. Challenges with applications that have large amounts of data to be transferred and have a high level interactiveness is the high latency incurred by the network and packet loss of the wireless network. A mobile application can be partitioned so that part of it runs on a nearby computing resource e.g., fog node or cloudlet. This thesis presents a framework that introduces fine-grained offloading approach and support for runtime and dynamic partitioning of an application. Third, we present a solution for placement of stream operators over distributed fog nodes for live processing of data streams from geographically distributed data sources. This placement of stream operators takes place in such a way that it supports applications with a high volume of data that require real-time (or near real-time) analysis To this end, this thesis proposed a set of algorithms for placement of stream operators among fog nodes

Deep Learning for Network Traffic Monitoring and Analysis (NTMA): A Survey

Modern communication systems and networks, e.g., Internet of Things (IoT) and cellular networks, generate a massive and heterogeneous amount of traffic data. In such networks, the traditional network management techniques for monitoring and data analytics face some challenges and issues, e.g., accuracy, and effective processing of big data in a real-time fashion. Moreover, the pattern of network traffic, especially in cellular networks, shows very complex behavior because of various factors, such as device mobility and network heterogeneity. Deep learning has been efficiently employed to facilitate analytics and knowledge discovery in big data systems to recognize hidden and complex patterns. Motivated by these successes, researchers in the field of networking apply deep learning models for Network Traffic Monitoring and Analysis (NTMA) applications, e.g., traffic classification and prediction. This paper provides a comprehensive review on applications of deep learning in NTMA. We first provide fundamental background relevant to our review. Then, we give an insight into the confluence of deep learning and NTMA, and review deep learning techniques proposed for NTMA applications. Finally, we discuss key challenges, open issues, and future research directions for using deep learning in NTMA applications.publishedVersio

Macro- and microscopic analysis of the internet economy from network measurements

Tesi per compendi de publicacions.The growth of the Internet impacts multiple areas of the world economy, and it has become a permanent part of the economic landscape both at the macro- and at microeconomic level. On-line traffic and information are currently assets with large business value. Even though commercial Internet has been a part of our lives for more than two decades, its impact on global, and everyday, economy still holds many unknowns. In this work we analyse important macro- and microeconomic aspects of the Internet. First we investigate the characteristics of the interdomain traffic, which is an important part of the macroscopic economy of the Internet. Finally, we investigate the microeconomic phenomena of price discrimination in the Internet. At the macroscopic level, we describe quantitatively the interdomain traffic matrix (ITM), as seen from the perspective of a large research network. The ITM describes the traffic flowing between autonomous systems (AS) in the Internet. It depicts the traffic between the largest Internet business entities, therefore it has an important impact on the Internet economy. In particular, we analyse the sparsity and statistical distribution of the traffic, and observe that the shape of the statistical distribution of the traffic sourced from an AS might be related to congestion within the network. We also investigate the correlations between rows in the ITM. Finally, we propose a novel method to model the interdomain traffic, that stems from first-principles and recognizes the fact that the traffic is a mixture of different Internet applications, and can have regional artifacts. We present and evaluate a tool to generate such matrices from open and available data. Our results show that our first-principles approach is a promising alternative to the existing solutions in this area, which enables the investigation of what-if scenarios and their impact on the Internet economy. At the microscopic level, we investigate the rising phenomena of price discrimination (PD). We find empirical evidences that Internet users can be subject to price and search discrimination. In particular, we present examples of PD on several ecommerce websites and uncover the information vectors facilitating PD. Later we show that crowd-sourcing is a feasible method to help users to infer if they are subject to PD. We also build and evaluate a system that allows any Internet user to examine if she is subject to PD. The system has been deployed and used by multiple users worldwide, and uncovered more examples of PD. The methods presented in the following papers are backed with thorough data analysis and experiments.Internet es hoy en día un elemento crucial en la economía mundial, su constante crecimiento afecta directamente múltiples aspectos tanto a nivel macro- como a nivel microeconómico. Entre otros aspectos, el tráfico de red y la información que transporta se han convertido en un producto de gran valor comercial para cualquier empresa. Sin embargo, más de dos decadas después de su introducción en nuestras vidas y siendo un elemento de vital importancia, el impacto de Internet en la economía global y diaria es un tema que alberga todavía muchas incógnitas que resolver. En esta disertación analizamos importantes aspectos micro y macroeconómicos de Internet. Primero, investigamos las características del tráfico entre Sistemas Autónomos (AS), que es un parte decisiva de la macroeconomía de Internet. A continuacin, estudiamos el controvertido fenómeno microeconómico de la discriminación de precios en Internet. A nivel macroeconómico, mostramos cuantitatívamente la matriz del tráfico entre AS ("Interdomain Traffic Matrix - ITM"), visto desde la perspectiva de una gran red científica. La ITM obtenida empíricamente muestra la cantidad de tráfico compartido entre diferentes AS, las entidades más grandes en Internet, siendo esto uno de los principales aspectos a evaluar en la economiá de Internet. Esto nos permite por ejemplo, analizar diferentes propiedades estadísticas del tráfico para descubrir si la distribución del tráfico producido por un AS está directamente relacionado con la congestión dentro de la red. Además, este estudio también nos permite investigar las correlaciones entre filas de la ITM, es decir, entre diferentes AS. Por último, basándonos en el estudio empírico, proponemos una innovadora solución para modelar el tráfico en una ITM, teniendo en cuenta que el tráfico modelado es dependiente de las particularidades de cada escenario (e.g., distribución de apliaciones, artefactos). Para obtener resultados representativos, la herramienta propuesta para crear estas matrices es evaluada a partir de conjuntos de datos abiertos, disponibles para toda la comunidad científica. Los resultados obtenidos muestran que el método propuesto es una prometedora alternativa a las soluciones de la literatura. Permitiendo así, la nueva investigación de escenarios desconocidos y su impacto en la economía de Internet. A nivel microeconómico, en esta tesis investigamos el fenómeno de la discriminación de precios en Internet ("price discrimination" - PD). Nuestros estudios permiten mostrar pruebas empíricas de que los usuarios de Internet están expuestos a discriminación de precios y resultados de búsquedas. En particular, presentamos ejemplos de PD en varias páginas de comercio electrónico y descubrimos que informacin usan para llevarlo a cabo. Posteriormente, mostramos como una herramienta crowdsourcing puede ayudar a la comunidad de usuarios a inferir que páginas aplican prácticas de PD. Con el objetivo de mitigar esta cada vez más común práctica, publicamos y evaluamos una herramienta que permite al usuario deducir si está siendo víctima de PD. Esta herramienta, con gran repercusión mediática, ha sido usada por multitud de usuarios alrededor del mundo, descubriendo así más ejemplos de discriminación. Por último remarcar que todos los metodos presentados en esta disertación están respaldados por rigurosos análisis y experimentos.Postprint (published version

Introducing the new paradigm of Social Dispersed Computing: Applications, Technologies and Challenges

[EN] If last decade viewed computational services as a utility then surely this decade has transformed computation into a commodity. Computation is now progressively integrated into the physical networks in a seamless way that enables cyber-physical systems (CPS) and the Internet of Things (IoT) meet their latency requirements. Similar to the concept of ¿platform as a service¿ or ¿software as a service¿, both cloudlets and fog computing have found their own use cases. Edge devices (that we call end or user devices for disambiguation) play the role of personal computers, dedicated to a user and to a set of correlated applications. In this new scenario, the boundaries between the network node, the sensor, and the actuator are blurring, driven primarily by the computation power of IoT nodes like single board computers and the smartphones. The bigger data generated in this type of networks needs clever, scalable, and possibly decentralized computing solutions that can scale independently as required. Any node can be seen as part of a graph, with the capacity to serve as a computing or network router node, or both. Complex applications can possibly be distributed over this graph or network of nodes to improve the overall performance like the amount of data processed over time. In this paper, we identify this new computing paradigm that we call Social Dispersed Computing, analyzing key themes in it that includes a new outlook on its relation to agent based applications. We architect this new paradigm by providing supportive application examples that include next generation electrical energy distribution networks, next generation mobility services for transportation, and applications for distributed analysis and identification of non-recurring traffic congestion in cities. The paper analyzes the existing computing paradigms (e.g., cloud, fog, edge, mobile edge, social, etc.), solving the ambiguity of their definitions; and analyzes and discusses the relevant foundational software technologies, the remaining challenges, and research opportunities.Garcia Valls, MS.; Dubey, A.; Botti, V. (2018). Introducing the new paradigm of Social Dispersed Computing: Applications, Technologies and Challenges. Journal of Systems Architecture. 91:83-102. https://doi.org/10.1016/j.sysarc.2018.05.007S831029