Automatic detection and resolution of deadlocks in Go programs

The Go programming language is acquiring momentum in the development of concurrent software. Even though Go supports the shared-memory model, the message-passing alternative is the favoured idiomatic approach. Naturally, this practice is not exempt of the usual difficulties: programs may deadlock and the language run-time has only very basic support for deadlock detection. Previous research on deadlock detection mainly focused on shared-memory concurrency models. For mainstream languages, tools and approaches specific to the message-passing paradigm are scarce and incipient. There is however a large body of work on models of concurrency that only recently started to be applied to languages like Go. Since the Go run-time lets many deadlocks pass unnoticed, and the existing solutions provided by third party tools detect many deadlocks but only try to fix a limited set of specific patterns, imposing severe conditions to do so, there is a clear need for more general deadlock resolution strategies, going beyond prevention and avoidance. To gain insight on real-world deadlock bugs, we first built and categorized a collection of bugs sourced from high-profile open-source Go projects. Next, we extended and implemented an algorithm that takes an abstraction of the communication behaviour of the program and, when all the communication operations on channels necessary for progress are present, but a deadlock is possible, presents the problem and offers a possible resolution for the error. The extensions allows our approach to analyse a much wider range of real world programs. We conclude with an evaluation, comparing with two other state-of-the-art solutions.A linguagem de programação Go tem ganhado tração no desenvolvimento de software concorrente. Apesar de o Go suportar o modelo de partilha de memória, o modelo alternativo de partilha de mensagens é a abordagem idiomática. Naturalmente, esta prática não está isenta das dificuldades usuais: os programas podem bloquear e o runtime da linguagem só possui um suporte muito básico para a deteção destes bloqueios. Investigação anterior na deteção de bloqueios focou principalmente no modelo de partilha de memória. Para linguagens convencionais, ferramentas e abordagens dedicadas ao paradigma de passagem de mensagens são escassas e incipientes. No entanto, existe um grande conjunto de trabalhos sobre modelos de concorrência que só recentemente começou a ser aplicado em linguagens como o Go. Visto que o run-time do Go deixa muitos bloqueios passar despercebidos e as soluções existentes detetam muitos bloqueios, mas só tentam resolver um conjunto muito pequeno de padrões. De modo a ganhar conhecimento sobre erros de bloqueio reais, nós começámos por construir e categorizar uma coleção de erros obtidos a partir de projetos Go open-source de alto perfil. De seguida, nós estendemos e implementámos um algoritmo que recebe uma abstração do comportamento da comunicação de um programa e quando todas as operações de comunicação nos canais necessários para o progresso estão presentes, mas um bloqueio é possível, apresenta o erro e oferece uma possível resolução do erro. A nossa extensão permite analisar um conjunto muito maior de programas reais. Concluímos com uma avaliação, comparando com duas outras soluções do estado da arte

Algorithms and Implementation

In the past few years Tabling has emerged as a powerful logic programming model. The integration of concurrent features into the implementation of Tabling systems is demanded by need to use recently developed tabling applications within distributed systems, where a process has to respond concurrently to several requests. The support for sharing of tables among the concurrent threads of a Tabling process is a desirable feature, to allow one of Tabling’s virtues, the re-use of computations by other threads and to allow efficient usage of available memory. However, the incremental completion of tables which are evaluated concurrently is not a trivial problem. In this dissertation we describe the integration of concurrency mechanisms, by the way of multi-threading, in a state of the art Tabling and Prolog system, XSB. We begin by reviewing the main concepts for a formal description of tabled computations, called SLG resolution and for the implementation of Tabling under the SLG-WAM, the abstract machine supported by XSB. We describe the different scheduling strategies provided by XSB and introduce some new properties of local scheduling, a scheduling strategy for SLG resolution. We proceed to describe our implementation work by describing the process of integrating multi-threading in a Prolog system supporting Tabling, without addressing the problem of shared tables. We describe the trade-offs and implementation decisions involved. We then describe an optimistic algorithm for the concurrent sharing of completed tables, Shared Completed Tables, which allows the sharing of tables without incurring in deadlocks, under local scheduling. This method relies on the execution properties of local scheduling and includes full support for negation. We provide a theoretical framework and discuss the implementation’s correctness and complexity. After that, we describe amethod for the sharing of tables among threads that allows parallelism in the computation of inter-dependent subgoals, which we name Concurrent Completion. We informally argue for the correctness of Concurrent Completion. We give detailed performance measurements of the multi-threaded XSB systems over a variety of machines and operating systems, for both the Shared Completed Tables and the Concurrent Completion implementations. We focus our measurements inthe overhead over the sequential engine and the scalability of the system. We finish with a comparison of XSB with other multi-threaded Prolog systems and we compare our approach to concurrent tabling with parallel and distributed methods for the evaluation of tabling. Finally, we identify future research directions

Drinking from Both Glasses: Combining Pessimistic and Optimistic Tracking of Cross-Thread Dependences *

Abstract It is notoriously challenging to develop parallel software systems that are both scalable and correct. Runtime support for parallelism-such as multithreaded record & replay, data race detectors, transactional memory, and enforcement of stronger memory models-helps achieve these goals, but existing commodity solutions slow programs substantially in order to track (i.e., detect or control) an execution's cross-thread dependences accurately. Prior work tracks cross-thread dependences either "pessimistically," slowing every program access, or "optimistically," allowing for lightweight instrumentation of most accesses but dramatically slowing accesses involved in cross-thread dependences. This paper seeks to hybridize pessimistic and optimistic tracking, which is challenging because there exists a fundamental mismatch between pessimistic and optimistic tracking. We address this challenge based on insights about how dependence tracking and program synchronization interact, and introduce a novel approach called hybrid tracking. Hybrid tracking is suitable for building efficient runtime support, which we demonstrate by building hybridtracking-based versions of a dependence recorder and a region serializability enforcer. An adaptive, profile-based policy makes runtime decisions about switching between pessimistic and optimistic tracking. Our evaluation shows that hybrid tracking enables runtime support to overcome the performance limitations of both pessimistic and optimistic tracking alone

Challenges and applications of assembly level software model checking

This thesis addresses the application of a formal method called Model Checking to the domain of software verification. Here, exploration algorithms are used to search for errors in a program. In contrast to the majority of other approaches, we claim that the search should be applied to the actual source code of the program, rather than to some formal model. There are several challenges that need to be overcome to build such a model checker. First, the tool must be capable to handle the full semantics of the underlying programming language. This implies a considerable amount of additional work unless the interpretation of the program is done by some existing infrastructure. The second challenge lies in the increased memory requirements needed to memorize entire program configurations. This additionally aggravates the problem of large state spaces that every model checker faces anyway. As a remedy to the first problem, the thesis proposes to use an existing virtual machine to interpret the program. This takes the burden off the developer, who can fully concentrate on the model checking algorithms. To address the problem of large program states, we call attention to the fact that most transitions in a program only change small fractions of the entire program state. Based on this observation, we devise an incremental storing of states which considerably lowers the memory requirements of program exploration. To further alleviate the per-state memory requirement, we apply state reconstruction, where states are no longer memorized explicitly but through their generating path. Another problem that results from the large state description of a program lies in the computational effort of hashing, which is exceptionally high for the used approach. Based on the same observation as used for the incremental storing of states, we devise an incremental hash function which only needs to process the changed parts of the program’s state. Due to the dynamic nature of computer programs, this is not a trivial task and constitutes a considerable part of the overall thesis. Moreover, the thesis addresses a more general problem of model checking - the state explosion, which says that the number of reachable states grows exponentially in the number of state components. To minimize the number of states to be memorized, the thesis concentrates on the use of heuristic search. It turns out that only a fraction of all reachable states needs to be visited to find a specific error in the program. Heuristics can greatly help to direct the search forwards the error state. As another effective way to reduce the number of memorized states, the thesis proposes a technique that skips intermediate states that do not affect shared resources of the program. By merging several consecutive state transitions to a single transition, the technique may considerably truncate the search tree. The proposed approach is realized in StEAM, a model checker for concurrent C++ programs, which was developed in the course of the thesis. Building on an existing virtual machine, the tool provides a set of blind and directed search algorithms for the detection of errors in the actual C++ implementation of a program. StEAM implements all of the aforesaid techniques, whose effectiveness is experimentally evaluated at the end of the thesis. Moreover, we exploit the relation between model checking and planning. The claim is, that the two fields of research have great similarities and that technical advances in one fields can easily carry over to the other. The claim is supported by a case study where StEAM is used as a planner for concurrent multi-agent systems. The thesis also contains a user manual for StEAM and technical details that facilitate understanding the engineering process of the tool

Autonomous Database Management at Scale: Automated Tuning, Performance Diagnosis, and Resource Decentralization

Database administration has always been a challenging task, and is becoming even more difficult with the rise of public and private clouds. Today, many enterprises outsource their database operation to cloud service providers (CSPs) in order to reduce operating costs. CSPs, now tasked with managing an extremely large number of database instances, cannot simply rely on database administrators. In fact, humans have become a bottleneck in the scalability and profitability of cloud offerings. This has created a massive demand for building autonomous databases—systems that operate with little or zero human supervision. While autonomous databases have gained much attention in recent years in both academia and industry, many of the existing techniques remain limited to automating parameter tuning, backup/recovery, and monitoring. Consequently, there is much to be done before realizing a fully autonomous database. This dissertation examines and offers new automation techniques for three specific areas of modern database management. 1. Automated Tuning – We propose a new generation of physical database designers that are robust against uncertainty in future workloads. Given the rising popularity of approximate databases, we also develop an optimal, hybrid sampling strategy that enables efficient join processing on offline samples, a long-standing open problem in approximate query processing. 2. Performance Diagnosis – We design practical tools and algorithms for assisting database administrators in quickly and reliably diagnosing performance problems in their transactional databases. 3. Resource Decentralization – To achieve autonomy among database components in a shared environment, we propose a highly efficient, starvation-free, and fully decentralized distributed lock manager for distributed database clusters.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/153349/1/dyoon_1.pd