The Weakest Failure Detector for Solving Wait-Free, Eventually Bounded-Fair Dining Philosophers

This dissertation explores the necessary and sufficient conditions to solve a variant of the dining philosophers problem. This dining variant is defined by three properties: wait-freedom, eventual weak exclusion, and eventual bounded fairness. Wait-freedom guarantees that every correct hungry process eventually enters its critical section, regardless of process crashes. Eventual weak exclusion guarantees that every execution has an infinite suffix during which no two live neighbors execute overlapping critical sections. Eventual bounded fairness guarantees that there exists a fairness bound k such that every execution has an infinite suffix during which no correct hungry process is overtaken more than k times by any neighbor. This dining variant (WF-EBF dining for short) is important for synchronization tasks where eventual safety (i.e., eventual weak exclusion) is sufficient for correctness (e.g., duty-cycle scheduling, self-stabilizing daemons, and contention managers). Unfortunately, it is known that wait-free dining is unsolvable in asynchronous message-passing systems subject to crash faults. To circumvent this impossibility result, it is necessary to assume the existence of bounds on timing properties, such as relative process speeds and message delivery time. As such, it is of interest to characterize the necessary and sufficient timing assumptions to solve WF-EBF dining. We focus on implicit timing assumptions, which can be encapsulated by failure detectors. Failure detectors can be viewed as distributed oracles that can be queried for potentially unreliable information about crash faults. The weakest detector D for WF-EBF dining means that D is both necessary and sufficient. Necessity means that every failure detector that solves WF-EBF dining is at least as strong as D. Sufficiency means that there exists at least one algorithm that solves WF-EBF dining using D. As such, our research goal is to characterize the weakest failure detector to solve WF-EBF dining. We prove that the eventually perfect failure detector 3P is the weakest failure detector for solving WF-EBF dining. 3P eventually suspects crashed processes permanently, but may make mistakes by wrongfully suspecting correct processes finitely many times during any execution. As such, 3P eventually stops suspecting correct processes

Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture

We present the architecture behind Twitter's real-time related query suggestion and spelling correction service. Although these tasks have received much attention in the web search literature, the Twitter context introduces a real-time "twist": after significant breaking news events, we aim to provide relevant results within minutes. This paper provides a case study illustrating the challenges of real-time data processing in the era of "big data". We tell the story of how our system was built twice: our first implementation was built on a typical Hadoop-based analytics stack, but was later replaced because it did not meet the latency requirements necessary to generate meaningful real-time results. The second implementation, which is the system deployed in production, is a custom in-memory processing engine specifically designed for the task. This experience taught us that the current typical usage of Hadoop as a "big data" platform, while great for experimentation, is not well suited to low-latency processing, and points the way to future work on data analytics platforms that can handle "big" as well as "fast" data

A simple approach to distributed objects in prolog

We present the design of a distributed object system for Prolog, based on adding remote execution and distribution capabilities to a previously existing object system. Remote execution brings RPC into a Prolog system, and its semantics is easy to express in terms of well-known Prolog builtins. The final distributed object design features state mobility and user-transparent network behavior. We sketch an implementation which provides distributed garbage collection and some degree of tolerance to network failures. We provide a preliminary study of the overhead of the communication mechanism for some test cases

The weakest failure detector for wait-free dining under eventual weak exclusion

Dining philosophers is a classic scheduling problem for local mutual exclusion on arbitrary conflict graphs. We establish necessary conditions to solve wait-free dining under eventual weak exclusion in message-passing systems with crash faults. Wait-free dining ensures that every correct hungry process eventually eats. Eventual weak exclusion permits finitely many scheduling mistakes, but eventually no live neighbors eat simultaneously; this exclusion criterion models scenarios where scheduling mistakes are recoverable or only affect per-formance. Previous work showed that the eventually perfect failure detector (3P) is sufficient to solve wait-free dining under eventual weak exclusion; we prove that 3P is also necessary, and thus 3P is the weakest oracle to solve this problem. Our reduction also establishes that any such din-ing solution can be made eventually fair. Finally, the reduc-tion itself may be of more general interest; when applied to wait-free perpetual weak exclusion, our reduction produces an alternative proof that the more powerful trusting oracle (T) is necessary (but not sufficient) to solve the problem o

Self-stabilizing protocol for anonymous oriented bi-directional rings under unfair distributed schedulers with a leader

We propose a self-stabilizing protocol for anonymous oriented bi-directional rings of any size under unfair distributed schedulers with a leader. The protocol is a randomized self-stabilizing, meaning that starting from an arbitrary configuration it converges (with probability 1) in finite time to a legitimate configuration (i.e. global system state) without the need for explicit exception handler of backward recovery. A fault may throw the system into an illegitimate configuration, but the system will autonomously resume a legitimate configuration, by regarding the current illegitimate configuration as an initial configuration, if the fault is transient. A self-stabilizing system thus tolerates any kind and any finite number of transient faults. The protocol can be used to implement an unfair distributed mutual exclusion in any ring topology network; Keywords: self-stabilizing protocol, anonymous oriented bi-directional ring, unfair distributed schedulers. Ring topology network, non-uniform and anonymous network, self-stabilization, fault tolerance, legitimate configuration

Control of transport dynamics in overlay networks

Transport control is an important factor in the performance of Internet protocols, particularly in the next generation network applications involving computational steering, interactive visualization, instrument control, and transfer of large data sets. The widely deployed Transport Control Protocol is inadequate for these tasks due to its performance drawbacks. The purpose of this dissertation is to conduct a rigorous analytical study on the design and performance of transport protocols, and systematically develop a new class of protocols to overcome the limitations of current methods. Various sources of randomness exist in network performance measurements due to the stochastic nature of network traffic. We propose a new class of transport protocols that explicitly accounts for the randomness based on dynamic stochastic approximation methods. These protocols use congestion window and idle time to dynamically control the source rate to achieve transport objectives. We conduct statistical analyses to determine the main effects of these two control parameters and their interaction effects. The application of stochastic approximation methods enables us to show the analytical stability of the transport protocols and avoid pre-selecting the flow and congestion control parameters. These new protocols are successfully applied to transport control for both goodput stabilization and maximization. The experimental results show the superior performance compared to current methods particularly for Internet applications. To effectively deploy these protocols over the Internet, we develop an overlay network, which resides at the application level to provide data transmission service using User Datagram Protocol. The overlay network, together with the new protocols based on User Datagram Protocol, provides an effective environment for implementing transport control using application-level modules. We also study problems in overlay networks such as path bandwidth estimation and multiple quickest path computation. In wireless networks, most packet losses are caused by physical signal losses and do not necessarily indicate network congestion. Furthermore, the physical link connectivity in ad-hoc networks deployed in unstructured areas is unpredictable. We develop the Connectivity-Through-Time protocols that exploit the node movements to deliver data under dynamic connectivity. We integrate this protocol into overlay networks and present experimental results using network to support a team of mobile robots

Advanced synchronization techniques for task-based runtime systems

Task-based programming models like OmpSs-2 and OpenMP provide a flexible data-flow execution model to exploit dynamic, irregular and nested parallelism. Providing an efficient implementation that scales well with small granularity tasks remains a challenge, and bottlenecks can manifest in several runtime components. In this paper, we analyze the limiting factors in the scalability of a task-based runtime system and propose individual solutions for each of the challenges, including a wait-free dependency system and a novel scalable scheduler design based on delegation. We evaluate how the optimizations impact the overall performance of the runtime, both individually and in combination. We also compare the resulting runtime against state of the art OpenMP implementations, showing equivalent or better performance, especially for fine-grained tasks.This project is supported by the European Union’s Horizon 2020 Research and Innovation programme under grant agreement No.s 754304 (DEEP-EST), by the Spanish Ministry of Science and Innovation (contract PID2019-107255GB and TIN2015-65316P) and by the Generalitat de Catalunya (2017-SGR-1414).Peer ReviewedPostprint (author's final draft

Internet Daemons: Digital Communications Possessed

We’re used to talking about how tech giants like Google, Facebook, and Amazon rule the internet, but what about daemons? Ubiquitous programs that have colonized the Net’s infrastructure—as well as the devices we use to access it—daemons are little known. Fenwick McKelvey weaves together history, theory, and policy to give a full account of where daemons come from and how they influence our lives—including their role in hot-button issues like network neutrality. Going back to Victorian times and the popular thought experiment Maxwell’s Demon, McKelvey charts how daemons evolved from concept to reality, eventually blossoming into the pandaemonium of code-based creatures that today orchestrates our internet. Digging into real-life examples like sluggish connection speeds, Comcast’s efforts to control peer-to-peer networking, and Pirate Bay’s attempts to elude daemonic control (and skirt copyright), McKelvey shows how daemons have been central to the internet, greatly influencing everyday users. Internet Daemons asks important questions about how much control is being handed over to these automated, autonomous programs, and the consequences for transparency and oversight. Table of Contents Abbreviations and Technical Terms Introduction 1. The Devil We Know: Maxwell’s Demon, Cyborg Sciences, and Flow Control 2. Possessing Infrastructure: Nonsynchronous Communication, IMPs, and Optimization 3. IMPs, OLIVERs, and Gateways: Internetworking before the Internet 4. Pandaemonium: The Internet as Daemons 5. Suffering from Buffering? Affects of Flow Control 6. The Disoptimized: The Ambiguous Tactics of the Pirate Bay 7. A Crescendo of Online Interactive Debugging? Gamers, Publics and Daemons Conclusion Acknowledgments Appendix: Internet Measurement and Mediators Notes Bibliography Index Reviews Beneath social media, beneath search, Internet Daemons reveals another layer of algorithms: deeper, burrowed into information networks. Fenwick McKelvey is the best kind of intellectual spelunker, taking us deep into the infrastructure and shining his light on these obscure but vital mechanisms. What he has delivered is a precise and provocative rethinking of how to conceive of power in and among networks. —Tarleton Gillespie, author of Custodians of the Internet Internet Daemons is an original and important contribution to the field of digital media studies. Fenwick McKelvey extensively maps and analyzes how daemons influence data exchanges across Internet infrastructures. This study insightfully demonstrates how daemons are transformative entities that enable particular ways of transferring information and connecting up communication, with significant social and political consequences. —Jennifer Gabrys, author of Program Eart