Enabling modern concurrency through program transformations

Concurrency is becoming the norm in modern software because multicores are now everywhere. Developers use concurrency constructs (i) to make their software responsive and scalable via asynchrony and (ii) to increase the throughput of their software via parallelism. Yet, we know little about how developers use these constructs in practice. Without such knowledge, (i) other developers cannot educate themselves about the state of the practice, (ii) language and library designers are unaware of any misuse, (iii) researchers make wrong assumptions, and (iv) tool providers do not provide the tools that developers really need. We claim that concurrent programming deserves first-class citizenship in empirical research and tool support. In this dissertation, we study the use, misuse, and underuse of concurrency constructs for C#. Based on our large-scale empirical studies, we found two main problems. First, developers still use complex, slow, low-level, and old constructs. Second, when developers use modern concurrency constructs, they highly misuse them, causing severe consequences such as subtle data-races and swallowed exceptions. Some significantly degrade the performance of async programs or can even deadlock the program. Other misuses make developers erroneously think that their code runs sequentially, when in reality the code runs concurrently; this causes unintended behavior and wrong results. This dissertation presents four tools to enable C# developers (i) to migrate their software to modern concurrency constructs and (ii) to fix misused modern constructs. First, we present an automated migration tool, Taskifier, that transforms old style Thread and ThreadPool constructs to higher-level Task constructs. Second, we present a refactoring tool, Simplifier, that extracts and converts Task-related code snippets into higher-level parallel patterns. Third, we provide Asyncifier to developers who want to refactor their callback-based asynchronous constructs to new async/await keywords. Fourth, we developed AsyncFixer, a static analysis tool that detects and corrects 14 common kinds of async/await misuses that cannot be detected by any existing tools. We released all our tools as plugins for the widely used Visual Studio IDE which has millions of users. We conducted both quantitative and qualitative evaluation of our tools. We applied the tools thousands of times over the real-world software and we show that they (i) are highly applicable, (ii) are much safer than manual transformations, and (iii) fast enough to be used interactively in Visual Studio. Our tools are useful in practice for both industry and open-source communities. Developers of the open-source software accepted 408 patches which are generated by the tools. We also received positive feedback from the early adopters. In industry, our tools are embraced by two companies. They started to actively use AsyncFixer in their automated build process. Our empirical findings have (i) educated thousands of developers through our educational web portals and (ii) influenced the design of modern constructs across programming languages. We hope that our dissertation serves as a call to action for the testing, program verification, and program analysis communities to study new constructs from programming languages and standard libraries. Researchers can empirically study these constructs and provide necessary tools to help developers adopt them. Hence, we also present a set of guidelines for researchers that aim to make a practical impact on the usage of new programming constructs

Methods to Improve Applicability and Efficiency of Distributed Data-Centric Compute Frameworks

The success of modern applications depends on the insights they collect from their data repositories. Data repositories for such applications currently exceed exabytes and are rapidly increasing in size, as they collect data from varied sources - web applications, mobile phones, sensors and other connected devices. Distributed storage and data-centric compute frameworks have been invented to store and analyze these large datasets. This dissertation focuses on extending the applicability and improving the efficiency of distributed data-centric compute frameworks