The Truth, the Whole Truth, and Nothing but the Truth: A Pragmatic Guide to Assessing Empirical Evaluations

An unsound claim can misdirect a field, encouraging the pursuit of unworthy ideas and the abandonment of promising ideas. An inadequate description of a claim can make it difficult to reason about the claim, for example to determine whether the claim is sound. Many practitioners will acknowledge the threat of un- sound claims or inadequate descriptions of claims to their field. We believe that this situation is exacerbated and even encouraged by the lack of a systematic approach to exploring, exposing, and addressing the source of unsound claims and poor exposition. This paper proposes a framework that identifies three sins of reasoning that lead to unsound claims and two sins of exposition that lead to poorly described claims. Sins of exposition obfuscate the objective of determining whether or not a claim is sound, while sins of reasoning lead directly to unsound claims. Our framework provides practitioners with a principled way of critiquing the integrity of their own work and the work of others. We hope that this will help individuals conduct better science and encourage a cultural shift in our research community to identify and promulgate sound claims

Actionable Program Analyses for Improving Software Performance

Nowadays, we have greater expectations of software than ever before. This is followed by constant pressure to run the same program on smaller and cheaper machines. To meet this demand, the application’s performance has become the essential concern in software development. Unfortunately, many applications still suffer from performance issues: coding or design errors that lead to performance degradation. However, finding performance issues is a challenging task: there is limited knowledge on how performance issues are discovered and fixed in practice, and current performance profilers report only where resources are spent, but not where resources are wasted. The goal of this dissertation is to investigate actionable performance analyses that help developers optimize their software by applying relatively simple code changes. To understand causes and fixes of performance issues in real-world software, we first present an empirical study of 98 issues in popular JavaScript projects. The study illustrates the prevalence of simple and recurring optimization patterns that lead to significant performance improvements. Then, to help developers optimize their code, we propose two actionable performance analyses that suggest optimizations based on reordering opportunities and method inlining. In this work, we focus on optimizations with four key properties. First, the optimizations are effective, that is, the changes suggested by the analysis lead to statistically significant performance improvements. Second, the optimizations are exploitable, that is, they are easy to understand and apply. Third, the optimizations are recurring, that is, they are applicable across multiple projects. Fourth, the optimizations are out-of-reach for compilers, that is, compilers can not guarantee that a code transformation preserves the original semantics. To reliably detect optimization opportunities and measure their performance benefits, the code must be executed with sufficient test inputs. The last contribution complements state-of-the-art test generation techniques by proposing a novel automated approach for generating effective tests for higher-order functions. We implement our techniques in practical tools and evaluate their effectiveness on a set of popular software systems. The empirical evaluation demonstrates the potential of actionable analyses in improving software performance through relatively simple optimization opportunities

Effective Performance Analysis and Debugging

Performance is once again a first-class concern. Developers can no longer wait for the next generation of processors to automatically optimize their software. Unfortunately, existing techniques for performance analysis and debugging cannot cope with complex modern hardware, concurrent software, or latency-sensitive software services. While processor speeds have remained constant, increasing transistor counts have allowed architects to increase processor complexity. This complexity often improves performance, but the benefits can be brittle; small changes to a program’s code, inputs, or execution environment can dramatically change performance, resulting in unpredictable performance in deployed software and complicating performance evaluation and debugging. Developers seeking to improve performance must resort to manual performance tuning for large performance gains. Software profilers are meant to guide developers to important code, but conventional profilers do not produce actionable information for concurrent applications. These profilers report where a program spends its time, not where optimizations will yield performance improvements. Furthermore, latency is a critical measure of performance for software services and interactive applications, but conventional profilers measure only throughput. Many performance issues appear only when a system is under high load, but generating this load in development is often impossible. Developers need to identify and mitigate scalability issues before deploying software, but existing tools offer developers little or no assistance. In this dissertation, I introduce an empirically-driven approach to performance analysis and debugging. I present three systems for performance analysis and debugging. Stabilizer mitigates the performance variability that is inherent in modern processors, enabling both predictable performance in deployment and statistically sound performance evaluation. Coz conducts performance experiments using virtual speedups to create the effect of an optimization in a running application. This approach accurately predicts the effect of hypothetical optimizations, guiding developers to code where optimizations will have the largest effect. Amp allows developers to evaluate system scalability using load amplification to create the effect of high load in a testing environment. In combination, Amp and Coz allow developers to pinpoint code where manual optimizations will improve the scalability of their software

Ernst Denert Award for Software Engineering 2019

This open access book provides an overview of the dissertations of the five nominees for the Ernst Denert Award for Software Engineering in 2019. The prize, kindly sponsored by the Gerlind & Ernst Denert Stiftung, is awarded for excellent work within the discipline of Software Engineering, which includes methods, tools and procedures for better and efficient development of high quality software. An essential requirement for the nominated work is its applicability and usability in industrial practice. The book contains five papers describing the works by Sebastian Baltes (U Trier) on Software Developers’Work Habits and Expertise, Timo Greifenberg’s thesis on Artefaktbasierte Analyse modellgetriebener Softwareentwicklungsprojekte, Marco Konersmann’s (U Duisburg-Essen) work on Explicitly Integrated Architecture, Marija Selakovic’s (TU Darmstadt) research about Actionable Program Analyses for Improving Software Performance, and Johannes Späth’s (Paderborn U) thesis on Synchronized Pushdown Systems for Pointer and Data-Flow Analysis – which actually won the award. The chapters describe key findings of the respective works, show their relevance and applicability to practice and industrial software engineering projects, and provide additional information and findings that have only been discovered afterwards, e.g. when applying the results in industry. This way, the book is not only interesting to other researchers, but also to industrial software professionals who would like to learn about the application of state-of-the-art methods in their daily work

Supporting Evolution and Maintenance of android Apps

Mobile developers and testers face a number of emerging challenges. These include rapid platform evolution and API instability; issues in bug reporting and reproduction involving complex multitouch gestures; platform fragmentation; the impact of reviews and ratings on the success of their apps; management of crowd-sourced requirements; continuous pressure from the market for frequent releases; lack of effective and usable testing tools; and limited computational resources for handheld devices. Traditional and contemporary methods in software evolution and maintenance were not designed for these types of challenges; therefore, a set of studies and a new toolbox of techniques for mobile development are required to analyze current challenges and propose new solutions. This dissertation presents a set of empirical studies, as well as solutions for some of the key challenges when evolving and maintaining android apps. In particular, we analyzed key challenges experienced by practitioners and open issues in the mobile development community such as (i) android API instability, (ii) performance optimizations, (iii) automatic GUI testing, and (iv) energy consumption. When carrying out the studies, we relied on qualitative and quantitative analyses to understand the phenomena on a large scale by considering evidence extracted from software repositories and the opinions of open-source mobile developers. From the empirical studies, we identified that dynamic analysis is a relevant method for several evolution and maintenance tasks, in particular, because of the need of practitioners to execute/validate the apps on a diverse set of platforms (i.e., device and OS) and under pressure for continuous delivery. Therefore, we designed and implemented an extensible infrastructure that enables large-scale automatic execution of android apps to support different evolution and maintenance tasks (e.g., testing and energy optimization). In addition to the infrastructure we present a taxonomy of issues, single solutions to the issues, and guidelines to enable large execution of android apps. Finally, we devised novel approaches aimed at supporting testing and energy optimization of mobile apps (two key challenges in evolution and maintenance of android apps). First, we propose a novel hybrid approach for automatic GUI-based testing of apps that is able to generate (un)natural test sequences by mining real applications usages and learning statistical models that represent the GUI interactions. In addition, we propose a multi-objective approach for optimizing the energy consumption of GUIs in android apps that is able to generate visually appealing color compositions, while reducing the energy consumption and keeping a design concept close to the original

Rohelisema tarkvaratehnoloogia poole tarkvaraanalüüsi abil

Mobiilirakendused, mis ei tühjenda akut, saavad tavaliselt head kasutajahinnangud. Mobiilirakenduste energiatõhusaks muutmiseks on avaldatud mitmeid refaktoreerimis- suuniseid ja tööriistu, mis aitavad rakenduse koodi optimeerida. Neid suuniseid ei saa aga seoses energiatõhususega üldistada, sest kõigi kontekstide kohta ei ole piisavalt energiaga seotud andmeid. Olemasolevad energiatõhususe parandamise tööriistad/profiilid on enamasti prototüübid, mis kohalduvad ainult väikese alamhulga energiaga seotud probleemide suhtes. Lisaks käsitlevad olemasolevad suunised ja tööriistad energiaprobleeme peamiselt a posteriori ehk tagantjärele, kui need on juba lähtekoodi sees. Android rakenduse koodi saab põhijoontes jagada kaheks osaks: kohandatud kood ja korduvkasutatav kood. Kohandatud kood on igal rakendusel ainulaadne. Korduvkasutatav kood hõlmab kolmandate poolte teeke, mis on rakendustesse lisatud arendusprotessi kiirendamiseks. Alustuseks hindame mitmete lähtekoodi halbade lõhnade refaktoreerimiste energiatarbimist Androidi rakendustes. Seejärel teeme empiirilise uuringu Androidi rakendustes kasutatavate kolmandate osapoolte võrguteekide energiamõju kohta. Pakume üldisi kontekstilisi suuniseid, mida võiks rakenduste arendamisel kasutada. Lisaks teeme süstemaatilise kirjanduse ülevaate, et teha kindlaks ja uurida nüüdisaegseid tugitööriistu, mis on rohelise Androidi arendamiseks saadaval. Selle uuringu ja varem läbi viidud katsete põhjal toome esile riistvarapõhiste energiamõõtmiste jäädvustamise ja taasesitamise probleemid. Arendame tugitööriista ARENA, mis võib aidata koguda energiaandmeid ja analüüsida Androidi rakenduste energiatarbimist. Viimasena töötame välja tugitööriista REHAB, et soovitada arendajatele energiatõhusaid kolmanda osapoole võrguteekeMobile apps that do not drain the battery usually get good user ratings. To make mobile apps energy efficient many refactoring guidelines and tools are published that help optimize the app code. However, these guidelines cannot be generalized w.r.t energy efficiency, as there is not enough energy-related data for every context. Existing energy enhancement tools/profilers are mostly prototypes applicable to only a small subset of energy-related problems. In addition, the existing guidelines and tools mostly address the energy issues a posteriori, i.e., once they have already been introduced into the code. Android app code can be roughly divided into two parts: the custom code and the reusable code. Custom code is unique to each app. Reusable code includes third-party libraries that are included in apps to speed up the development process. We start by evaluating the energy consumption of various code smell refactorings in native Android apps. Then we conduct an empirical study on the energy impact of third-party network libraries used in Android apps. We provide generalized contextual guidelines that could be used during app development Further, we conduct a systematic literature review to identify and study the current state of the art support tools available to aid green Android development. Based on this study and the experiments we conducted before, we highlight the problems in capturing and reproducing hardware-based energy measurements. We develop the support tool ‘ARENA’ that could help gather energy data and analyze the energy consumption of Android apps. Last, we develop the support tool ‘REHAB’ to recommend energy efficient third-party network libraries to developers.https://www.ester.ee/record=b547174

Educational Theories and Learning Analytics : From Data to Knowledge

Under embargo until 17.01.21.acceptedVersio

Infrastructure for Performance Monitoring and Analysis of Systems and Applications

The growth of High Performance Computer (HPC) systems increases the complexity with respect to understanding resource utilization, system management, and performance issues. HPC performance monitoring tools need to collect information at both the application and system levels to yield a complete performance picture. Existing approaches limit the abilities of the users to do meaningful analysis on actionable timescale. Efficient infrastructures are required to support largescale systems performance data analysis for both run-time troubleshooting and post-run processing modes. In this dissertation, we present methods to fill these gaps in the infrastructure for HPC performance monitoring and analysis. First, we enhance the architecture of a monitoring system to integrate streaming analysis capabilities at arbitrary locations within its data collection, transport, and aggregation facilities. Next, we present an approach to streaming collection of application performance data. We integrate these methods with a monitoring system used on large-scale computational platforms. Finally, we present a new approach for constructing durable transactional linked data structures that takes advantage of byte-addressable non-volatile memory technologies. Transactional data structures are building blocks of in-memory databases that are used by HPC monitoring systems to store and retrieve data efficiently. We evaluate the presented approaches on a series of case studies. The experiment results demonstrate the impact of our tools, while keeping the overhead in an acceptable margin