Improvements for Store-Collect and Atomic Snapshot Objects under Continuous Churn

The field of distributed computing has given rise to many algorithms to share data among nodes in a network. This work focuses on the store-collect and the atomic snapshot objects in an asynchronous, crash-prone message-passing dynamic system with nodes continuously entering and leaving the system. We assume that the maximum number of nodes that enter, leave or crash during some time interval is proportional to the size of the system. A store-collect object is a distributed object that allows nodes to store data in the system in a variable that can be read by all nodes, but only modified by the node that stored it. This is achieved through two basic operations: the store operation, which stores information into the network, and collect, which collects a copy of all the information stored by every node in the network at the beginning of the time interval in which the operation is active. The atomic snapshot object is quite similar. It provides two operations, scan and update, that behave in a very similar fashion to the collect and store operations given by the store-collect object; however the atomic snapshot object must satisfy the linearizability condition, which means that it is always possible to arrange all the operations performed into an ordered sequence even if there are operations that occur simultaneously. This work improves upon the store-collect and atomic snapshot implementations given in Attiya et al [SSS, 2020]. We developed a method for quantifying the churn of a network subject to certain assumptions. This new method allows us to prove the correctness of the store-collect algorithm under less restrictive conditions than those found in the original proof of Attiya et al. Additionally, we developed an improved implementation of the atomic snapshot object based on a store-collect object that requires fewer messages to complete a scan or an update operation

Static Analysis in Practice

Static analysis tools search software looking for defects that may cause an application to deviate from its intended behavior. These include defects that compute incorrect values, cause runtime exceptions or crashes, expose applications to security vulnerabilities, or lead to performance degradation. In an ideal world, the analysis would precisely identify all possible defects. In reality, it is not always possible to infer the intent of a software component or code fragment, and static analysis tools sometimes output spurious warnings or miss important bugs. As a result, tool makers and researchers focus on developing heuristics and techniques to improve speed and accuracy. But, in practice, speed and accuracy are not sufficient to maximize the value received by software makers using static analysis. Software engineering teams need to make static analysis an effective part of their regular process. In this dissertation, I examine the ways static analysis is used in practice by commercial and open source users. I observe that effectiveness is hampered, not only by false warnings, but also by true defects that do not affect software behavior in practice. Indeed, mature production systems are often littered with true defects that do not prevent them from functioning, mostly correctly. To understand why this occurs, observe that developers inadvertently create both important and unimportant defects when they write software, but most quality assurance activities are directed at finding the important ones. By the time the system is mature, there may still be a few consequential defects that can be found by static analysis, but they are drowned out by the many true but low impact defects that were never fixed. An exception to this rule is certain classes of subtle security, performance, or concurrency defects that are hard to detect without static analysis. Software teams can use static analysis to find defects very early in the process, when they are cheapest to fix, and in so doing increase the effectiveness of later quality assurance activities. But this effort comes with costs that must be managed to ensure static analysis is worthwhile. The cost effectiveness of static analysis also depends on the nature of the defect being sought, the nature of the application, the infrastructure supporting tools, and the policies governing its use. Through this research, I interact with real users through surveys, interviews, lab studies, and community-wide reviews, to discover their perspectives and experiences, and to understand the costs and challenges incurred when adopting static analysis tools. I also analyze the defects found in real systems and make observations about which ones are fixed, why some seemingly serious defects persist, and what considerations static analysis tools and software teams should make to increase effectiveness. Ultimately, my interaction with real users confirms that static analysis is well received and useful in practice, but the right environment is needed to maximize its return on investment

Cloud-edge hybrid applications

Many modern applications are designed to provide interactions among users, including multi- user games, social networks and collaborative tools. Users expect application response time to be in the order of milliseconds, to foster interaction and interactivity. The design of these applications typically adopts a client-server model, where all interac- tions are mediated by a centralized component. This approach introduces availability and fault- tolerance issues, which can be mitigated by replicating the server component, and even relying on geo-replicated solutions in cloud computing infrastructures. Even in this case, the client-server communication model leads to unnecessary latency penalties for geographically close clients and high operational costs for the application provider. This dissertation proposes a cloud-edge hybrid model with secure and ecient propagation and consistency mechanisms. This model combines client-side replication and client-to-client propagation for providing low latency and minimizing the dependency on the server infras- tructure, fostering availability and fault tolerance. To realize this model, this works makes the following key contributions. First, the cloud-edge hybrid model is materialized by a system design where clients maintain replicas of the data and synchronize in a peer-to-peer fashion, and servers are used to assist clients’ operation. We study how to bring most of the application logic to the client-side, us- ing the centralized service primarily for durability, access control, discovery, and overcoming internetwork limitations. Second, we dene protocols for weakly consistent data replication, including a novel CRDT model (∆-CRDTs). We provide a study on partial replication, exploring the challenges and fundamental limitations in providing causal consistency, and the diculty in supporting client- side replicas due to their ephemeral nature. Third, we study how client misbehaviour can impact the guarantees of causal consistency. We propose new secure weak consistency models for insecure settings, and algorithms to enforce such consistency models. The experimental evaluation of our contributions have shown their specic benets and limitations compared with the state-of-the-art. In general, the cloud-edge hybrid model leads to faster application response times, lower client-to-client latency, higher system scalability as fewer clients need to connect to servers at the same time, the possibility to work oine or disconnected from the server, and reduced server bandwidth usage. In summary, we propose a hybrid of cloud-and-edge which provides lower user-to-user la- tency, availability under server disconnections, and improved server scalability – while being ecient, reliable, and secure.Muitas aplicações modernas são criadas para fornecer interações entre utilizadores, incluindo jogos multiutilizador, redes sociais e ferramentas colaborativas. Os utilizadores esperam que o tempo de resposta nas aplicações seja da ordem de milissegundos, promovendo a interação e interatividade. A arquitetura dessas aplicações normalmente adota um modelo cliente-servidor, onde todas as interações são mediadas por um componente centralizado. Essa abordagem apresenta problemas de disponibilidade e tolerância a falhas, que podem ser mitigadas com replicação no componente do servidor, até com a utilização de soluções replicadas geogracamente em infraestruturas de computação na nuvem. Mesmo neste caso, o modelo de comunicação cliente-servidor leva a penalidades de latência desnecessárias para clientes geogracamente próximos e altos custos operacionais para o provedor das aplicações. Esta dissertação propõe um modelo híbrido cloud-edge com mecanismos seguros e ecientes de propagação e consistência. Esse modelo combina replicação do lado do cliente e propagação de cliente para cliente para fornecer baixa latência e minimizar a dependência na infraestrutura do servidor, promovendo a disponibilidade e tolerância a falhas. Para realizar este modelo, este trabalho faz as seguintes contribuições principais. Primeiro, o modelo híbrido cloud-edge é materializado por uma arquitetura do sistema em que os clientes mantêm réplicas dos dados e sincronizam de maneira ponto a ponto e onde os servidores são usados para auxiliar na operação dos clientes. Estudamos como trazer a maior parte da lógica das aplicações para o lado do cliente, usando o serviço centralizado principalmente para durabilidade, controlo de acesso, descoberta e superação das limitações inter-rede. Em segundo lugar, denimos protocolos para replicação de dados fracamente consistentes, incluindo um novo modelo de CRDTs (∆-CRDTs). Fornecemos um estudo sobre replicação parcial, explorando os desaos e limitações fundamentais em fornecer consistência causal e a diculdade em suportar réplicas do lado do cliente devido à sua natureza efémera. Terceiro, estudamos como o mau comportamento da parte do cliente pode afetar as garantias da consistência causal. Propomos novos modelos seguros de consistência fraca para congurações inseguras e algoritmos para impor tais modelos de consistência. A avaliação experimental das nossas contribuições mostrou os benefícios e limitações em comparação com o estado da arte. Em geral, o modelo híbrido cloud-edge leva a tempos de resposta nas aplicações mais rápidos, a uma menor latência de cliente para cliente e à possibilidade de trabalhar oine ou desconectado do servidor. Adicionalmente, obtemos uma maior escalabilidade do sistema, visto que menos clientes precisam de estar conectados aos servidores ao mesmo tempo e devido à redução na utilização da largura de banda no servidor. Em resumo, propomos um modelo híbrido entre a orla (edge) e a nuvem (cloud) que fornece menor latência entre utilizadores, disponibilidade durante desconexões do servidor e uma melhor escalabilidade do servidor – ao mesmo tempo que é eciente, conável e seguro

Designing peer-to-peer overlays:a small-world perspective

The Small-World phenomenon, well known under the phrase "six degrees of separation", has been for a long time under the spotlight of investigation. The fact that our social network is closely-knitted and that any two people are linked by a short chain of acquaintances was confirmed by the experimental psychologist Stanley Milgram in the sixties. However, it was only after the seminal work of Jon Kleinberg in 2000 that it was understood not only why such networks exist, but also why it is possible to efficiently navigate in these networks. This proved to be a highly relevant discovery for peer-to-peer systems, since they share many fundamental similarities with the social networks; in particular the fact that the peer-to-peer routing solely relies on local decisions, without the possibility to invoke global knowledge. In this thesis we show how peer-to-peer system designs that are inspired by Small-World principles can address and solve many important problems, such as balancing the peer load, reducing high maintenance cost, or efficiently disseminating data in large-scale systems. We present three peer-to-peer approaches, namely Oscar, Gravity, and Fuzzynet, whose concepts stem from the design of navigable Small-World networks. Firstly, we introduce a novel theoretical model for building peer-to-peer systems which supports skewed node distributions and still preserves all desired properties of Kleinberg's Small-World networks. With such a model we set a reference base for the design of data-oriented peer-to-peer systems which are characterized by non-uniform distribution of keys as well as skewed query or access patterns. Based on this theoretical model we introduce Oscar, an overlay which uses a novel scalable network sampling technique for network construction, for which we provide a rigorous theoretical analysis. The simulations of our system validate the developed theory and evaluate Oscar's performance under typical conditions encountered in real-life large-scale networked systems, including participant heterogeneity, faults, as well as skewed and dynamic load-distributions. Furthermore, we show how by utilizing Small-World properties it is possible to reduce the maintenance cost of most structured overlays by discarding a core network connectivity element – the ring invariant. We argue that reliance on the ring structure is a serious impediment for real life deployment and scalability of structured overlays. We propose an overlay called Fuzzynet, which does not rely on the ring invariant, yet has all the functionalities of structured overlays. Fuzzynet takes the idea of lazy overlay maintenance further by eliminating the need for any explicit connectivity and data maintenance operations, relying merely on the actions performed when new Fuzzynet peers join the network. We show that with a sufficient amount of neighbors, even under high churn, data can be retrieved in Fuzzynet with high probability. Finally, we show how peer-to-peer systems based on the Small-World design and with the capability of supporting non-uniform key distributions can be successfully employed for large-scale data dissemination tasks. We introduce Gravity, a publish/subscribe system capable of building efficient dissemination structures, inducing only minimal dissemination relay overhead. This is achieved through Gravity's property to permit non-uniform peer key distributions which allows the subscribers to be clustered close to each other in the key space where data dissemination is cheap. An extensive experimental study confirms the effectiveness of our system under realistic subscription patterns and shows that Gravity surpasses existing approaches in efficiency by a large margin. With the peer-to-peer systems presented in this thesis we fill an important gap in the family of structured overlays, bringing into life practical systems, which can play a crucial role in enabling data-oriented applications distributed over wide-area networks

Resilience-Building Technologies: State of Knowledge -- ReSIST NoE Deliverable D12

This document is the first product of work package WP2, "Resilience-building and -scaling technologies", in the programme of jointly executed research (JER) of the ReSIST Network of Excellenc

The Vibrancy and Resilience of British High Streets

British high streets have endured significant economic and cultural challenges both in the leadup to and as a consequence of the COVID-19 pandemic. The volatile and challenging socio-economic environment has been brought about by the lingering effects of the 2008 recession, high business rates, competition from online retailers, and the impact and implications of the global pandemic. The changes to the high street retail landscape have been recorded using new sources of data that can supplement traditional data sources such as local government retail surveys. New sources of data such as consumer data, property portal data and mobility data are more spatially and temporally granular. As a result, local governments, the retail sector and stakeholders can use these emerging forms of data to create more easily updateable measures of high street composition and performance. This thesis utilises the Local Data Company’s Britain-wide database on retail location, type and vacancy. The data ranges between the start of 2017 and June of 2021, containing around 800,000 records of occupiers. The analysis within this thesis starts by describing the composition and vibrancy of British high streets in the lead-up to the pandemic. Next, the thesis provides an evaluation of the impact of the COVID-19 pandemic and the subsequent shift towards remote working on the viable resilience of commuter towns. This section is followed by an exploration of the short-term impacts of the COVID-19 lockdown restrictions on the resilience of Britain’s high streets. Finally, the application of new forms of data in informing local government high street regeneration policy is studied as part of a knowledge exchange with the London Borough of Camden. This thesis contributes to our understanding of how the circumstances of different British high streets can be monitored and mapped, with the goal of improving understanding of vibrancy, resilience, and potential for regeneration

Statistical physics approaches to large-scale socio-economic networks

Die statistische Physik erforschte im letzten Jahrzehnt eine Fülle von wissenschaftlichen Gebieten, was zu einem besseren quantitativen Verständnis von verschiedenen, aus vielen Elementen bestehenden Systemen, z.B. von sozialen Systemen, geführt hat. Eine empirische Quantifizierung von menschlichem Verhalten auf gesellschaftlichem Niveau hat sich allerdings bisher als sehr schwierig erwiesen, wegen Problemen bei der Gewinnung und Qualität von Daten. In dieser Doktorarbeit erstellen wir zum ersten mal einen umfangreichen über fünf Jahre gesammelten Datensatz, der praktisch alle Aktionen und Eigenschaften der 350.000 Teilnehmer einer gesamten menschlichen Gesellschaft aus einem selbstentwickelten Massive Multiplayer Online Game enthält. Wir beschreiben dieses aus stark wechselwirkenden Spielern bestehende soziale System in drei Ebenen. In einem ersten Schritt analysieren wir die Individuen und deren Verhalten im Verlauf der Zeit. Eine Skalen- und Fluktuationsanalyse von Aktions-Reaktions-Zeitreihen enthüllt Persistenz der möglichen Aktionen und qualitative Unterschiede zwischen "guten" und "schlechten" Spielern. Wir untersuchen danach den Diffusionsprozess der im Spieluniversum stattfindenden Bewegungen der Individuen. Wir finden Subdiffusivität und eine durch ein Potenzgesetz verteilte Präferenz zu kürzlich besuchten Orten zurückzukehren. Zweitens, auf der nächsthöheren Ebene, verwenden wir Netzwerktheorie um die topologische Struktur der Interaktionen zwischen Individuen zu quantifizieren. Wir konzentrieren uns auf sechs durch direkte Interaktionen definierte Netzwerke, drei davon positiv (Handel, Freundschaft, Kommunikation), drei negativ (Feindschaft, Attacke, Bestrafung). Diese Netzwerke weisen nichttriviale statistische Eigenschaften auf, z.B. skaleninvariante Topologie, und entwickeln sich in der Zeit, was uns erlaubt eine Reihe von Hypothesen über sozialdynamische Phänomene zu testen. Wir finden qualitative Unterschiede zwischen positiven und negativen Netzwerken in Evolution und Struktur. Schließlich untersuchen wir das Multiplex-Netzwerk der Spielergesellschaft, das sich aus den einzelnen Netzwerk-Schichten zusammensetzt. Wir quantifizieren Interaktionen zwischen verschiedenen Netzwerken und zeigen die nichttrivialen Organisationsprinzipien auf die auch in echten menschlichen Gesellschaften beobachtet wurden. Unsere Erkenntnisse liefern Belege für die Hypothese der strukturellen Balance, die eine Vermeidung von gewissen frustrierten Zuständen auf mikroskopischem Niveau postuliert. Mit diesem Aufbau demonstrieren wir die Möglichkeit der Gewinnung neuartiger wissenschaftlicher Erkenntnisse über die Natur von kollektivem menschlichen Verhalten in großangelegten sozialen Systemen.In the past decade a variety of fields has been explored by statistical physicists, leading to an increase of our quantitative understanding of various systems composed of many interacting elements, such as social systems. However, an empirical quantification of human behavior on a societal level has so far proved to be tremendously difficult due to problems in data availability, quality and ways of acquisition. In this doctoral thesis we compile for the first time a large-scale data set consisting of practically all actions and properties of 350,000 odd participants of an entire human society interacting in a self-developed Massive Multiplayer Online Game, over a period of five years. We describe this social system composed of strongly interacting players in the game in three consecutive levels. In a first step, we examine the individuals and their behavioral properties over time. A scaling and fluctuation analysis of action-reaction time-series reveals persistence of the possible actions and qualitative differences between "good" and "bad" players. We then study and model the diffusion process of human mobility occurring within the "game universe". We find subdiffusion and a power-law distributed preference to return to more recently visited locations. Second, on a higher level, we use network theory to quantify the topological structure of interactions between the individuals. We focus on six network types defined by direct interactions, three of them with a positive connotation (trade, friendship, communication), three with a negative one (enmity, attack, punishment). These networks exhibit non-trivial statistical properties, e.g. scale-free topology, and evolve over time, allowing to test a series of long-standing social-dynamics hypotheses. We find qualitative differences in evolution and topological structure between positive and negative tie networks. Finally, on a yet higher level, we consider the multiplex network of the player society, constituted by the coupling of the single network layers. We quantify interactions between different networks and detect the non-trivial organizational principles which lead to the observed structure of the system and which have been observed in real human societies as well. Our findings with the multiplex framework provide evidence for the half-century old hypothesis of structural balance, where certain frustrated states on a microscopic level tend to be avoided. Within this setup we demonstrate the feasibility for generating novel scientific insights on the nature of collective human behavior in large-scale social systems