Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

This report documents the program and the outcomes of GI-Dagstuhl Seminar 16394 "Software Performance Engineering in the DevOps World". The seminar addressed the problem of performance-aware DevOps. Both, DevOps and performance engineering have been growing trends over the past one to two years, in no small part due to the rise in importance of identifying performance anomalies in the operations (Ops) of cloud and big data systems and feeding these back to the development (Dev). However, so far, the research community has treated software engineering, performance engineering, and cloud computing mostly as individual research areas. We aimed to identify cross-community collaboration, and to set the path for long-lasting collaborations towards performance-aware DevOps. The main goal of the seminar was to bring together young researchers (PhD students in a later stage of their PhD, as well as PostDocs or Junior Professors) in the areas of (i) software engineering, (ii) performance engineering, and (iii) cloud computing and big data to present their current research projects, to exchange experience and expertise, to discuss research challenges, and to develop ideas for future collaborations

Automated root cause isolation in performance regression testing

Testing of software is an important aspect of software development. There exist multiple kinds of tests, like unit tests and integration tests. The tests this thesis will focus on will be load tests, which are used to observe a system’s behavior under load. The presented approach will use these load tests in order to observe and analyze the performance of a system, like e.g. the response times of methods. Next these observations are compared with those made on other versions of the system, in order to detect performance regressions, deteriorations in performance, between versions. Another goal of the approach will be to identify the root cause of the regressions, which is the source code change responsible for introducing them. By doing this, the task of fixing this problem will be made easier for the software engineer, since he has an entry point for the problem.Das Testen von Software ist ein wichtiger Bestandteil der Software-Entwicklung. Es existieren viele Arten von Tests, wie Unit-Tests und Integrationstests. Die Tests, auf welche sich diese Thesis fokussiert, sind Lasttests. Diese werden genutzt um zu beobachten, wie ein System sich unter Belastung verhält. Der vorgestellte Ansatz wird diese Lasttests nutzen, um das Betriebsverhalten eines Systems zu erfassen und analysieren, wie z.B. das Antwortzeitverhalten von einzelnen Methoden. Als Nächstes werden diese Beobachtungen mit denen verglichen, die auf anderen Versionen des Systems gemacht wurden, um Regressionen im Betriebsverhalten, wie Verschlechterungen des Antwortzeitverhaltens, zwischen den Versionen zu finden. Ein weiteres Ziel des Ansatzes wird es sein, die Hauptursache einer Regression zu identifizieren, welches die Quellcodeänderung ist, die für die Einführung der Regression verantwortlich ist. Dies wird es dem Software-Entwickler, der beauftragt wurde die Regression zu verbessern, einfacher machen dies zu tun, da er bereits einen festen Ansatzpunkt geliefert bekommen hat

Improving software engineering processes using machine learning and data mining techniques

The availability of large amounts of data from software development has created an area of research called mining software repositories. Researchers mine data from software repositories both to improve understanding of software development and evolution, and to empirically validate novel ideas and techniques. The large amount of data collected from software processes can then be leveraged for machine learning applications. Indeed, machine learning can have a large impact in software engineering, just like it has had in other fields, supporting developers, and other actors involved in the software development process, in automating or improving parts of their work. The automation can not only make some phases of the development process less tedious or cheaper, but also more efficient and less prone to errors. Moreover, employing machine learning can reduce the complexity of difficult problems, enabling engineers to focus on more interesting problems rather than the basics of development. The aim of this dissertation is to show how the development and the use of machine learning and data mining techniques can support several software engineering phases, ranging from crash handling, to code review, to patch uplifting, to software ecosystem management. To validate our thesis we conducted several studies tackling different problems in an industrial open-source context, focusing on the case of Mozilla

Mining Performance Regression Inducing Code Changes in Evolving Software

During software evolution, the source code of a system frequently changes due to bug fixes or new feature requests. Some of these changes may accidentally degrade performance of a newly released software version. A notable problem of regression testing is how to find problematic changes (out of a large number of committed changes) that may be responsible for performance regressions under certain test inputs. We propose a novel recommendation system, coined as PerfImpact, for automatically identifying code changes that may potentially be responsible for performance regressions using a combination of search-based input profiling and change impact analysis techniques. PerfImpact independently sends the same input values to two releases of the application under test, and uses a genetic algorithm to mine execution traces and explore a large space of input value combinations to find specific inputs that take longer time to execute in a new release. Since these input values are likely to expose performance regressions, PerfImpact automatically mines the corresponding execution traces to evaluate the impact of each code change on the performance and ranks the changes based on their estimated contribution to performance regressions. We implemented PerfImpact and evaluated it on different releases of two open-source web applications. The results demonstrate that PerfImpact effectively detects input value combinations to expose performance regressions and mines the code changes are likely to be responsible for these performance regressions

Performance Problem Diagnostics by Systematic Experimentation

Diagnostics of performance problems requires deep expertise in performance engineering and entails a high manual effort. As a consequence, performance evaluations are postponed to the last minute of the development process. In this thesis, we introduce an automatic, experiment-based approach for performance problem diagnostics in enterprise software systems. With this approach, performance engineers can concentrate on their core competences instead of conducting repeating tasks