Computing change of invariants to support software evolution

Software is always evolving. In the recent years, the development community has shifted towards Agile development paradigm resulting in faster release cycle. This emphasis on speed is, generally, accompanied by an increase in the number of bugs and reduced focus on updating non-functional software artifacts like specification document. Recent studies have found that developers find it difficult to determine whether a change might break code elsewhere in the program, resulting in 25% of bugs fixes to be incorrect or buggy. A method to capture the semantic changes between different versions of a program is vital in understanding the impact of the change and in preventing bugs. An invariant is a condition that is always true at a given program point. Invariants are used to specify the requirements and desired behavior of a program at any program point. The difference in invariants between different program versions can be used to capture the changes made to the program. In this thesis, we use the change of invariants as a way to capture the semantic changes over different program versions. We designed a static demand-driven algorithm for automatically computing the change of invariants between different versions of a program. To evaluate the algorithm and its ability to capture semantic changes over different program versions, we built a prototype framework called Hydrogen. Our experimental results show that Hydrogen is able to compute the change of invariants between different versions of the programs, and the computed change of invariants can be used for understanding changes and generating assertions to prevent similar bugs in future

COST-EFFECTIVE TECHNIQUES FOR CONTINUOUS INTEGRATION TESTING

Continuous integration (CI) development environments allow software engineers to frequently integrate and test their code. While CI environments provide advantages, they also utilize non-trivial amounts of time and resources. To address this issue, researchers have adapted techniques for test case prioritization (TCP) and regression test selection (RTS) to CI environments. To date, current TCP techniques under CI environments have operated on test suites, and have not achieved substantial improvements. In this thesis, we use a lightweight approach based on test suite failure and execution history, and “continuously” prioritizes commits that are waiting for execution in response to the arrival of each new commit and the completion of each previously commit scheduled for testing. We conduct an empirical study on three datasets, and the result shows that, after prioritization, our technique can effectively detect failing commits earlier. To date, current RTS techniques under CI environment is based on two windows in terms of time. But this technique fails to consider the arrival rate of test suites and only takes the results of test suites execution history into account. In this thesis, we present a Count-Based RTS technique, which is based on the test suite failures and execution history by utilizing two window sizes in terms of number of test suites, and a Transition-Based RTS technique, which adds the test suites’ “pass to malfunction” transitions for selection prediction in addition to the two window sizes. We again conduct an empirical study on three datasets, and the results show that, after selection, Transition-Based technique detects more malfunctions and more “pass to malfunction” transitions than the existing techniques. Adviser: Gregg Rothermel, Sebastian Elbau

Continuous Integration Build Failures in Practice

Automated software testing is a popular method of quality control that aims to detect bugs before software is released to the end user. Unfortunately, writing, maintaining, and executing automated test suites is expensive and not necessarily cost effective. To gain a better understanding of the return on investment and cost of automated testing, we studied the continuous integration build results of 61 open source projects. We found that 19.4% of build failures are resolved by a change to only test code. These failures do not find bugs in the code under test, but rather bugs in the test suites themselves and represent a maintenance cost. We also found that 12.8% of the build failures we studied are due to non-deterministic tests that can be difficult to debug. To gain a better understanding of what leads to test maintenance we manually inspected both valuable and costly tests and found a variety of reasons for test maintenance. Our empirical analysis of real projects quantifies the cost of maintaining test suites and shows that reducing maintenance costs could make automated testing much more affordable

Enhancing the test and evaluation process: implementing agile development, test automation, and model-based systems engineering concepts

2020 Fall.Includes bibliographical references.With the growing complexity of modern systems, traditional testing methods are falling short. Test documentation suites used to verify the software for these types of large, complex systems can become bloated and unclear, leading to extremely long execution times and confusing, unmanageable test procedures. Additionally, the complexity of these systems can prevent the rapid understanding of complicated system concepts and behaviors, which is a necessary part of keeping up with the demands of modern testing efforts. Opportunities for optimization and innovation exist within the Test and Evaluation (T&E) domain, evidenced by the emergence of automated testing frameworks and iterative testing methodologies. Further opportunities lie with the directed expansion and application of related concepts such as Model-Based Systems Engineering (MBSE). This dissertation documents the development and implementation of three methods of enhancing the T&E field when applied to a real-world project. First, the development methodology of the system was transitioned from Waterfall to Agile, providing a more responsive approach when creating new features. Second, the Test Automation Framework (TAF) was developed, enabling the automatic execution of test procedures. Third, a method of test documentation using the Systems Modeling Language (SysML) was created, adopting concepts from MBSE to standardize the planning and analysis of test procedures. This dissertation provides the results of applying the three concepts to the development process of an airborne Electronic Warfare Management System (EWMS), which interfaces with onboard and offboard aircraft systems to receive and process the threat environment, providing the pilot or crew with a response solution for the protection of the aircraft. This system is representative of a traditional, long-term aerospace project that has been constantly upgraded over its lifetime. Over a two-year period, this new process produced a number of qualitative and quantitative results, including improving the quality and organization of the test documentation suite, reducing the minimum time to execute the test procedures, enabling the earlier identification of defects, and increasing the overall quality of the system under test. The application of these concepts generated many lessons learned, which are also provided. Transitioning a project's development methodology, modernizing the test approach, and introducing a new system of test documentation may provide significant benefits to the development of a system, but these types of process changes must be weighed against the needs of the project. This dissertation provides details of the effort to improve the effectiveness of the T&E process on an example project, as a framework for possible implementation on similar systems

Data Science for Software Maintenance

Maintaining and evolving modern software systems is a difficult task: their scope and complexity mean that seemingly inconsequential changes can have far-reaching consequences. Most software development companies attempt to reduce the number of faults introduced by adopting maintenance processes. These processes can be developed in various ways. In this thesis, we argue that data science techniques can be used to support process development. Specifically, we claim that robust development processes are necessary to minimize the number of faults introduced when evolving complex software systems. These processes should be based on empirical research findings. Data science techniques allow software engineering researchers to develop research insights that may be difficult or impossible to obtain with other research methodologies. These research insights support the creation of development processes. Thus, data science techniques support the creation of empirically-based development processes. We support this argument with three examples. First, we present insights into automated malicious Android application (app) detection. Many of the prior studies done on this topic used small corpora that may provide insufficient variety to create a robust app classifier. Currently, no empirically established guidelines for corpus size exist, meaning that previous studies have used anywhere from tens of apps to hundreds of thousands of apps to draw their conclusions. This variability makes it difficult to judge if the findings of any one study generalize. We attempted to establish such guidelines and found that 1,000 apps may be sufficient for studies that are concerned with what the majority of apps do, while more than a million apps may be required in studies that want to identify outliers. Moreover, many prior studies of malicious app detection used outdated malware corpora in their experiments that, combined with the rapid evolution of the Android API, may have influenced the accuracy of the studies. We investigated this problem by studying 1.3 million apps and showed that the evolution of the API does affect classifier accuracy, but not in the way we originally predicted. We also used our API usage data to identify the most infrequently used API methods. The use of data science techniques allowed us to study an order of magnitude more apps than previous work in the area; additionally, our insights into infrequently used methods illustrate how data science can be used to guide API deprecation. Second, we present insights into the costs and benefits of regression testing. Regression test suites grow over time, and while a comprehensive suite can detect faults that are introduced into the system, such a suite can be expensive to write, maintain, and execute. These costs may or may not be justified, depending on the number and severity of faults the suite can detect. By studying 61 projects that use Travis CI, a continuous integration system, we were able to characterize the cost/benefit tradeoff of their test suites. For example, we found that only 74% of non-flaky test failures are caused by defects in the system under test; the other 26% were caused by incorrect or obsolete tests and thus represent a maintenance cost rather than a benefit of the suite. Data about the costs and benefits of testing can help system maintainers understand whether their test suite is a good investment, shaping their subsequent maintenance decisions. The use of data science techniques allowed us to study a large number of projects, increasing the external generalizability of the study and making the insights gained more useful. Third, we present insights into the use of mutants to replace real faulty programs in testing research. Mutants are programs that contain deliberately injected faults, where the faults are generated by applying mutation operators. Applying an operator means making a small change to the program source code, such as replacing a constant with another constant. The use of mutants is appealing because large numbers of mutants can be automatically generated and used when known faults are unavailable or insufficient in number. However, prior to this work, there was little experimental evidence to support the use of mutants as a replacement for real faults. We studied this problem and found that, in general, mutants are an adequate substitute for faults when conducting testing research. That is, a test suite’s ability to detect mutants is correlated with its ability to detect real faults that developers have fixed, for both developer-written and automatically-generated test suites. However, we also found that additional mutation operators should be developed and some classes of faults cannot be generated via mutation. The use of data science techniques was an essential part of generating the set of real faults used in the study. Taken together, the results of these three studies provide evidence that data science techniques allow software engineering researchers to develop insights that are difficult or impossible to obtain using other research methodologie

On the Cost and Value of Build Automation

RÉSUMÉ : L’Intégration Continue (CI) est un processus de développement par lequel des changements de code d’un projet sont régulièrement testés et validés, souvent de manière quotidienne, afin de pouvoir mettre à jour plus régulièrement un produit. Ce système est de plus en plus utilisé notamment dans de grosses compagnies qui doivent gérer la participation de nombreux collaborateurs à un projet/produit commun. La validation du maintien de la qualité logicielle d’un produit est primordiale pour que le produit puisse être employé en continu par ses utilisateurs. Afin de faciliter cette intégration continue, les gestionnaires CI font souvent appel à des systèmes de builds automatisés: un build est une vérification de code via une compilation et un/des test(s) dans des conditions variées (différentes plateformes, versions, etc.). La gestion automatisée de ces builds permet aux systèmes de tester la validation de code à chaque arrivée de changements de code.----------ABSTRACT : Continuous Integration (CI) has become the core infrastructure of modern software organizations, enabling them to update projects more regularly (i.e., daily) while keeping the quality of the code valid and relevant. However, with more than 100,000 automated build jobs triggered each day by, for example, the Mozilla CI, build automation comes with a financial and time cost that should not be neglected. Through this thesis, we aim at validating the hypothesis that the cost of build automation is not negligible and that the build activity could be reduced. This thesis presents the results of a quantitative study of the CI over the course of 20 months, analyzing over 54 million Mozilla build jobs to identify the cost of build automation triggered for each code change as well as a build reduction proposition, focusing on the case of brown builds at Ubisoft

Towards a Model-Centric Software Testing Life Cycle for Early and Consistent Testing Activities

The constant improvement of the available computing power nowadays enables the accomplishment of more and more complex tasks. The resulting implicit increase in the complexity of hardware and software solutions for realizing the desired functionality requires a constant improvement of the development methods used. On the one hand over the last decades the percentage of agile development practices, as well as testdriven development increases. On the other hand, this trend results in the need to reduce the complexity with suitable methods. At this point, the concept of abstraction comes into play, which manifests itself in model-based approaches such as MDSD or MBT. The thesis is motivated by the fact that the earliest possible detection and elimination of faults has a significant influence on product costs. Therefore, a holistic approach is developed in the context of model-driven development, which allows applying testing already in early phases and especially on the model artifacts, i.e. it provides a shift left of the testing activities. To comprehensively address the complexity problem, a modelcentric software testing life cycle is developed that maps the process steps and artifacts of classical testing to the model-level. Therefore, the conceptual basis is first created by putting the available model artifacts of all domains into context. In particular, structural mappings are specified across the included domain-specific model artifacts to establish a sufficient basis for all the process steps of the life cycle. Besides, a flexible metamodel including operational semantics is developed, which enables experts to carry out an abstract test execution on the modellevel. Based on this, approaches for test case management, automated test case generation, evaluation of test cases, and quality verification of test cases are developed. In the context of test case management, a mechanism is realized that enables the selection, prioritization, and reduction of Test Model artifacts usable for test case generation. I.e. a targeted set of test cases is generated satisfying quality criteria like coverage at the model-level. These quality requirements are accomplished by using a mutation-based analysis of the identified test cases, which builds on the model basis. As the last step of the model-centered software testing life cycle two approaches are presented, allowing an abstract execution of the test cases in the model context through structural analysis and a form of model interpretation concerning data flow information. All the approaches for accomplishing the problem are placed in the context of related work, as well as examined for their feasibility by of a prototypical implementation within the Architecture And Analysis Framework. Subsequently, the described approaches and their concepts are evaluated by qualitative as well as quantitative evaluation. Moreover, case studies show the practical applicability of the approach

Mining Software Repositories for Release Engineers - Empirical Studies on Integration and Infrastructures-as-Code

RÉSUMÉ Release engineering (Releng) est le processus de la mise en production logicielle des contributions des développeurs en un produit intégré livré aux utilisateurs. Ce processus consiste des phases d’intégration, de construction et des tests, de déploiement et de livraison, pour finalement entrer au marché. Alors que, traditionnellement, la mise en production prend plusieurs mois pour livrer un produit logiciel complet aux clients, la mise en production moderne vise à apporter de la valeur au client plus rapidement, dans l’ordre des jours ou des semaines, pour recevoir de rétroaction utile plus rapidement et pour minimiser le temps perdu sur des fonctionnalités échouées. De nos jours, une panoplie d’outils et de techniques a émergé pour soutenir la mise en production. Ils visent essentiellement à automatiser les phases dans le pipeline de la mise en production, ce qui réduit le travail manuel et rend le processus reproductible et rapide. Par exemple, Puppet est l’un des outils les plus populaires pour Infrastructure-as-Code (IaC), ce qui automatise le processus de mettre en place une nouvelle infrastructure (par exemple, une machine virtuelle ou un conteneur dans lequel une application peut être compilée, testée et déployée) selon des spécifications textuelles. IaC a évolué rapidement en raison de la croissance de l’infonuagique. Cependant, de nombreux problèmes existent encore pour la mise en production. Par exemple, alors que de nombreux outils de la mise en production gagnent en popularité, le choix de la technique la plus appropriée exige des praticiens d’évaluer empiriquement la performance des techniques dans un contexte réaliste, avec des données représentatives. Pire encore, à un niveau plus haut, les ingénieurs de la mise en production doivent analyser le progrès de l’organisation dans chaque phase de la mise en production, afin de savoir si la prochaine date de sortie peut être respectée ou si des obstacles se produisent. De nouveau, il n’y a pas de méthode cohérente et établie pour ce faire. Pour aider les praticiens à mieux analyser leur processus de la mise en production, nous explorons la façon selon laquelle la fouille de référentiels logiciels (Mining Software Repositories; MSR) est capable d’analyser le progrès d’une phase de la mise en production ou d’évaluer la performance d’un outil de mise en production. MSR agit sur les données stockées dans des référentiels de logiciels tels que les systèmes de gestion de versions, les référentiels de bogues ou des environnements de révision technique. Au lieu que les développeurs, les testeurs et les examinateurs utilisent ces référentiels juste pour enregistrer des données de développement (telles que les changements de code, rapports de bogues ou des révisions techniques), MSR rend ces données actionnables en les analysant. Par exemple, en faisant l’extraction de l’information des changements de code source et de rapports de bogues, on peut recréer l’ensemble du processus de développement d’un projet ou de la mise en production. De nos jours, de nombreux référentiels logiciels à source libre sont disponibles au public, offrant des possibilités pour l’analyse empirique de la mise en production en utilisant des technologies MSR. Dans cette thèse, on a fait des analyses MSR pour deux phases critiques de la mise en production, c.-à-d. l’intégration et le provisionnement de l’environnement d’un logiciel (avec IaC), sur plusieurs projets larges à source libre. Cette série d’études empiriques ciblait de comprendre le progrès du processus de la mise en production et d’évaluer la performance des outils de point. Nous nous sommes concentrés principalement sur ces deux phases parce qu’elles sont essentielles dans la mise en production, et un grand nombre de données est disponible pour elles. D’abord, nous avons constaté que la révision technique et l’intégration de changements de code sont impactées par de différents facteurs. Nos résultats suggèrent que les développeurs réussissent à faire passer leurs contributions à travers la révision technique plus rapidement en changeant moins de sous-systèmes à la fois et de diviser une grande contribution en plusieurs contributions plus petites. En outre, les développeurs peuvent faire accepter leurs contributions plus facilement et plus rapidement en participant davantage dans la communauté de développeurs à source libre et d’acquérir plus d’expérience dans des sous-systèmes similaires. Dans cette étude sur le noyau Linux, nous avons trouvé que l’un des défis majeurs de MSR dans le contexte de la mise en production logicielle est de relier les différents référentiels nécessaires. Par exemple, le noyau Linux ou le projet Apache HTTPD utilisent tous les deux des listes de diffusion pour effectuer le processus de révision technique. Les experts examinent des contributions par courriel, et un fil de courriels est utilisé pour ramasser toutes les différentes versions d’une contribution. Cependant, souvent un nouveau fil de discussion, avec sujet différent, est utilisé pour soumettre une nouvelle révision, ce qui signifie qu’aucun lien physique étroit n’existe entre toutes les révisions d’une contribution. En plus, les versions révisées d’une contribution non plus n’ont de lien physique avec la version acceptée dans le référentiel de gestion de versions, à moins qu’un identifiant de validation soit affiché dans un courriel. Surtout quand une contribution a été révisée plusieurs fois et a beaucoup évolué en comparaison avec la version initiale, le suivi à partir de sa toute première version est difficile à faire. Nous avons proposé trois approches de différente granularité et de rigueur différente, dans le but de récupérer les liens physiques entre les révisions de contributions dans le même fil de courriels. Dans notre étude, nous avons constaté que la technique au niveau des lignes individuelles fonctionne le mieux pour lier des contributions entre différents fils de courriels, tandis qu’une combinaison de cette approche avec celle à base de sommes de contrôle réalise la meilleure performance pour relier les contributions dans un fil de courriels avec la version finale dans le référentiel de gestion de versions. Être capable de reconstituer l’historique complet de contributions nous a permis d’analyser le progrès de la phase de révision du noyau Linux. Nous avons constaté que 25% des contributions acceptées prennent plus de quatre semaines pour leur révision technique. Deuxièmement, pour évaluer la capacité de MSR pour analyser la performance des outils de mise en production, nous avons évalué dans un projet commercial une approche d’intégration hybride qui combine les techniques de branchement et de “feature toggles”. Des branches permettent aux développeurs de travailler sur différentes fonctionnalités d’un système en parallèle, en isolation (sans impacter d’autres équipes), tandis qu’un feature toggle permet aux développeurs de travailler dans une branche sur différentes tâches en cachant des fonctionnalités sous développement avec des conditions “if” dans le code source. Au lieu de réviser leur processus d’intégration entièrement pour abandonner les branches et de passer aux feature toggles, l’approche hybride est un compromis qui tente de minimiser les risques des branches tout en profitant des avantages des feature toggles. Nous avons comparé la performance avant et après l’adoption de l’approche hybride, et avons constaté que cette structure hybride peut réduire l’effort d’intégration et améliorer la productivité. Par conséquent, l’approche hybride semble une pratique valable. Dans la phase de provisionnement, nous nous sommes concentrés sur l’évaluation de l’utilisation et de l’effort requis pour des outils populaires de “Infrastructure-as-Code” (IaC), qui permettent de spécifier les requis d’environnement dans un format textuel. Nous avons étudié empiriquement les outils IaC dans OpenStack et MediaWiki, deux projets énormément larges qui ont adopté deux des langues IaC actuellement les plus populaires: Puppet et Chef. Tout d’abord, nous avons comparé l’effort de maintenance lié à IaC avec celui du codage et des tests. Nous avons constaté que le code IaC prend une partie importante du système dans les deux projets et change fréquemment, avec de grands changements de code. Les changements de code IaC sont étroitement couplés avec les changements de code source, ce qui implique que les changements de code source ou des tests nécessitent des changements complémentaires au code source IaC, et pourrait causer un effort plus large de maintenance et de gestion de complexité. Cependant, nous avons également observé un couplage léger avec des cas de test IaC et les données de provisionnement, qui sont de nouveaux types d’artéfacts dans le domaine de IaC. Par conséquent, IaC peut nécessiter plus d’effort que les ingénieurs expectent. D’autres études empiriques devraient être envisagées. L’ingénierie de la mise en production moderne a développé rapidement, tandis que de nombreux nouvelles techniques et outils ont émergé pour le soutenir de différentes perspectives. Cependant, le manque de techniques pour comprendre le progrès des phases de la mise en production ou d’évaluer la performance d’outils de la mise en production rend le travail difficile pour les praticiens qui ont à maintenir la qualité de leur processus de mise en production. Dans cette thèse, nous avons mené une série d’études empiriques en utilisant des techniques de fouille des référentiels logiciels sur des données de larges projets à source libre, qui montrent que, malgré des défis, la technologie MSR peut aider les ingénieurs de la mise en production à mieux comprendre leur progrès et à évaluer le coût des outils et des activités de la mise en production. Nous sommes heureux de voir que notre travail a inspiré d’autres chercheurs pour analyser davantage le processus d’intégration, ainsi que la qualité du code IaC.---------- ABSTRACT Release engineering (Releng) is the process of delivering integrated work from developers as a complete product to end users. This process comprises the phases of Integration, Building and Testing, Deployment and Release to finally reach the market. While traditional software engineering takes several months to deliver a complete software product to customers, modern Release engineering aims to bring value to customer more quickly, receive useful feedback faster, and reduce time wasted on unsuccessful features in development process. A wealth of tools/techniques emerged to support Release engineering. They basically aim to automate phases in the Release engineering pipeline, reducing the manual labor, and making the procedure repeatable and fast. For example, Puppet is one of the most popular Infrastructure-as-Code (IaC) tools, which automates the process of setting up a new infrastructure (e.g., a virtual machine or a container in which an application can be compiled, tested and deployed) according to specifications. Infrastructure-as-Code has evolved rapidly due to the growth of cloud computing. However, many problems also come along. For example, while many Release engineering tools gain popularity, choosing the most suitable technique requires practitioners to empirically evaluate the performance of the technique in a realistic setting, with data mimicking their own setup. Even worse, at a higher level, release engineers need to understand the progress of each release engineering phase, in order to know whether the next release deadline can be met or where bottlenecks occur. Again, they have no clear methodology to do this. To help practitioners analyze their Release engineering process better, we explore the way of mining software repositories (MSR) on two critical phases of Releng of large open-source projects. Software repositories like version control systems, bug repositories or code reviewing environments, are used on a daily basis by developers, testers and reviewers to record information about the development process, such as code changes, bug reports or code reviews. By analyzing the data, one can recreate the process of how software is built and analyze how each phase of Releng applies in this project. Many repositories of open-source software projects are available publicly, which offers opportunities for empirical research of Release engineering. Therefore, we conduct a series of empirical studies of mining software repositories of popular open-source software projects, to understand the progress of Release engineering and evaluate the performance of state-of-the-art tools. We mainly focus on two phases: Integration and Provisioning (Infrastructure-as-Code), because these two phases are most critical in Release engineering and ample quantity data is available. In our empirical study of the Integration process, we evaluate how well MSR techniques based on version control and review data explain the major factors impacting the probability and time taken for a patch to be successfully integrated into an upcoming release. We selected the Linux kernel, one of the most popular OSS projects having a long history and a strict integration hierarchy, as our case study. We collected data from reviewing and integration tools of the Linux kernel (mailing lists and Git respectively), and extracted characteristics covering six dimensions. Then, we built models with acceptance/time as output and analyzed which characteristics have impact on the reviewing and integration processes. We found that reviewing and integration are impacted by different factors. Our findings suggest that developers manage to get their patch go through review phase faster by changing less subsystems at a time and splitting a large patch into multiple smaller patches. Also, developers can make patches accepted more easily and sooner by participating more in the community and gaining more experience in similar patches. In this study on the Linux kernel, we found that one major challenge of MSR is to link different repositories. For example, the Linux kernel and Apache project both use mailing lists to perform the reviewing process. Contributors submit and maintainers review patches all by emails, where usually an email thread is used to collect all different versions of a patch. However, often a new email thread, with different subject, is being used to submit a new patch revision, which means that no strong physical links between all patch revisions exist. On top of that, the accepted patch also does not have a physical link to the resulting commit in the version control system, unless a commit identifier is posted in an email. Especially when a patch has been revised multiple times and evolved a lot from the original version, tracking its very first version is difficult. We proposed three approaches of different granularity and strictness, aiming to recover the physical links between emails in the same thread. In the study, we found that a line-based technique works best to link emails between threads while the combination of line-based and checksum-based technique achieves the best performance for linking emails in a thread with the final, accepted commit. Being able to reconstruct the full history of a patch allowed us to analyze the performance of the reviewing phase. We found that 25% of commits have a reviewing history longer than four weeks. To evaluate the ability of MSR to analyze the performance of Releng tools, we evaluated a hybrid integration approach, which combines branching and toggling techniques together, in a commercial project. Branching allows developers to work on different branches in parallel, while toggling enables developers on the same branch on different tasks. Instead of revising their whole integration process to drop branching and move to toggling, hybrid toggling is a compromise that tries to minimize the risks of branching while enjoy the benefits of toggling. We compared the performance before and after adopting hybrid toggling, and found that this hybrid structure can reduce integration effort and improve productivity. Hence, hybrid toggling seems a worthwhile practice. In the Provisioning phase, we focus on evaluating the usage and effort of the popular tools used in modern Release engineering: Infrastructure-as-Code (IaC). We empirically studied IaC tools in OpenStack and MediaWiki, which have a huge code base and adopt two currently most popular IaC languages: Puppet and Chef. First, we study maintenance effort related to the regular development and testing process of OpenStack, then compare this to IaC-related effort in both case studies. We found that IaC code takes a large proportion in both projects and it changes frequently, with large churn size. Meanwhile, IaC code changes are tightly coupled with source code changes, which implies that changes to source or test code require accompanying changes to IaC, which might lead to increased complexity and maintenance effort. Furthermore, the most common reason for such coupling is “Integration of new modules or service”. However, we also observed IaC code has light coupling with IaC test cases and test data, which are new kinds of artifacts in IaC domain. Hence, IaC may take more effort than engineers expect and further empirical studies should be considered. Modern Release engineering has developed rapidly, while many new techniques/tools emerge to support it from different perspectives. However, lack of knowledge of the current Release engineering progress and performance of these techniques makes it difficult for practitioners to sustain high quality Releng approach in practice. In this thesis, we conducted a series of empirical studies of mining software repositories of large open-source projects, that show that, despite some challenges, MSR technology can help release engineers understand better the progress of and evaluate the cost of Release engineering tools and activities. We are glad to see our work has inspired other researchers to further analyze the integration process as well as the quality of IaC code

Black-Box Testfall-Selektion und -Priorisierung für Software-Varianten und -Versionen

Software testing is a fundamental task in software quality assurance. Especially when dealing with several product variants or software versions under test, testing everything for each variant and version is infeasible due to limited testing resources. To cope with increasing complexity both in time (i.e., versions) and space (i.e., variants), new techniques have to be developed to focus on the most important parts for testing. In the past, regression testing techniques such as test case selection and prioritization have emerged to tackle these issues for single-software systems. However, testing of variants and versions is still a challenging task, especially when no source code is available. Most existing regression testing techniques analyze source code to identify important changes to be retested, i.e., they are likely to reveal a failure. To this end, this thesis contributes different techniques for both, variants and versions, to allow more efficient and effective testing in difficult black-box scenarios by identifying important test cases to be re-executed. Four major contributions in software testing are made. (1) We propose a test case prioritization framework for software product lines based on delta-oriented test models to reduce the redundancy in testing between different product variants.(2) We introduce a risk-based testing technique for software product lines. Our semi-automatic test case prioritization approach is able to compute risk values for test model elements and scales with large numbers of product variants. (3) For black-box software versions, we provide a test case selection technique based on genetic algorithms. In particular, seven different black-box selection objectives are defined, thus, we perform a multi-objective test case selection finding Pareto optimal test sets to reduce the testing effort. (4) We propose a novel test case prioritization technique based on supervised machine learning. It is able to imitate decisions made by experts based on different features, such as natural language test case descriptions and black-box meta-data. All of these techniques have been evaluated using the Body Comfort System case study. For testing of software versions, we also assesses our testing techniques using an industrial system. Our evaluation results indicate that our black-box testing approaches for software variants and versions are able to successfully reduce testing effort compared to existing techniques.Testen ist eine fundamentale Aufgabe zur Qualitätssicherung von modernen Softwaresystemen. Mangels limitierter Ressourcen ist das Testen von vielen Produktvarianten oder Versionen sehr herausfordernd und das wiederholte Ausführen aller Testfälle nicht wirtschaftlich. Um mit der Raum- (Varianten) und Zeitdimension (Versionen) in der Entwicklung umzugehen, wurden in der Vergangenheit verschiedene Testansätze entwickelt. Es existieren jedoch nach wie vor große Herausforderungen, welche es zu lösen gilt. Dies ist vor allem der Fall, wenn der Quellcode der getesteten Softwaresysteme unbekannt ist. Das Testen von Black-Box-Systemen erschwert die Identifikation von zu testenden Unterschieden zu vorher getesteten Varianten oder Versionen. In der Literatur finden sich wenige Ansätze, welche versuchen diese Herausforderungen zu lösen. Daher werden in dieser Dissertation neue Ansätze entwickelt und vorgestellt, welche beim Black-Box Testen von Software-Varianten und -Versionen helfen, wichtige Testfälle zur erneuten Ausführung zu identifizieren. Dies erspart die Ausführung von Testfällen, welche weder neues Verhalten testen noch mit hoher Wahrscheinlichkeit neue Fehler zu finden. Insgesamt leistet diese Dissertation die folgenden vier wissenschaftlichen Beiträge: (1) Ein modell-basiertes Framework zur Definition von Testfallpriorisierungsfunktionen für variantenreiche Systeme. Das Framework ermöglicht eine flexible Priorisierung von Testfällen für individuelle Produktvarianten. (2) Einen risiko-basierten Testfallpriorisierungsansatz für variantenreiche Systeme. Das Verfahren ermöglicht eine semi-automatisierte Berechnung von Risikowerten für Elemente von Produktvarianten und skaliert mit großen Produktzahlen. (3) Ein multi-kriterielles Testfallselektionsverfahren für den Regressionstest von Black-Box Software-Versionen. Es werden Black-Box Testkriterien aufgestellt und mittels eines genetischen Algorithmus optimiert um Pareto-optimale Testsets zu berechnen. (4) Ein Testfallpriorisierungsverfahren für Black-Box Regressionstests mit Hilfe von Machine Learning. Der verwendete Algorithmus imitiert Entscheidungen von Testexperten um wichtige Testfälle zu identifizieren. Diese Ansätze wurden alle mit Hilfe von Fallstudien evaluiert. Die resultierenden Ergebnisse zeigen, dass die Ansätze die gewünschten Ziele erreichen und helfen, wichtige Testfälle effektiv zu identifizieren. Insgesamt wird der Testaufwand im Vergleich zu existierenden Techniken verringert