Internal quality evolution of a large test system–an industrial study

This paper presents our empirical observations related to the evolution of a large automated test system. The system observed is used in the industry as a test tool for complex telecommunication systems, itself consisting of more than one million lines of source code. This study evaluates how different changes during the development have changed the number of observed Code Smells in the test system. We have monitored the development of the test scripts and measured the code quality characteristics over a five years period

Test software quality issues and connections to international standards

This paper examines how ISO/IEC 9126-1 and ISO/IEC 25010 quality models can be applied to software testing products in industrial environment. We present a set of code smells for test systems written in TTCN-3 and their categorization according to quality model standards. We demonstrate our measurements on industrial and ETSI projects, and provide a method for estimating their effects on product risks in current projects

Refactorisation methods for TTCN-3

In this paper we introduce automatic methods for restructuring source codes written in test description languages. We modify the structure of these sources without making any changes to their behavior. This technique is called refactorisation. There are many approaches to refactorisation. The goal of our refactorisation methods is to increase the maintainability of source codes. We focus on TTCN-3 (Testing and Test Control Notation), which is a rapidly spreading test description language nowadays. A TTCN-3 source consists of a data description (static) part and a test execution (dynamic) part. We have developed models and refactorisation methods based on these models, separately for the two parts. The static part is mapped into a layered graph structure, while the dynamic part is mapped to a CEFSM (Communicating Extended Finite State Machine) – based model.

On the Distribution of Test Smells in Open Source Android Applications: An Exploratory Study

The impact of bad programming practices, such as code smells, in production code has been the focus of numerous studies in software engineering. Like production code, unit tests are also affected by bad programming practices which can have a negative impact on the quality and maintenance of a software system. While several studies addressed code and test smells in desktop applications, there is little knowledge of test smells in the context of mobile applications. In this study, we extend the existing catalog of test smells by identifying and defining new smells and survey over 40 developers who confirm that our proposed smells are bad programming practices in test suites. Additionally, we perform an empirical study on the occurrences and distribution of the proposed smells on 656 open-source Android apps. Our findings show a widespread occurrence of test smells in apps. We also show that apps tend to exhibit test smells early in their lifetime with different degrees of co-occurrences on different smell types. This empirical study demonstrates that test smells can be used as an indicator for necessary preventive software maintenance for test suites

What the Smell? An Empirical Investigation on the Distribution and Severity of Test Smells in Open Source Android Applications

The widespread adoption of mobile devices, coupled with the ease of developing mobile-based applications (apps) has created a lucrative and competitive environment for app developers. Solely focusing on app functionality and time-to-market is not enough for developers to ensure the success of their app. Quality attributes exhibited by the app must also be a key focus point; not just at the onset of app development, but throughout its lifetime. The impact analysis of bad programming practices, or code smells, in production code has been the focus of numerous studies in software maintenance. Similar to production code, unit tests are also susceptible to bad programming practices which can have a negative impact not only on the quality of the software system but also on maintenance activities. With the present corpus of studies on test smells primarily on traditional applications, there is a need to fill the void in understanding the deviation of testing guidelines in the mobile environment. Furthermore, there is a need to understand the degree to which test smells are prevalent in mobile apps and the impact of such smells on app maintenance. Hence, the purpose of this research is to: (1) extend the existing set of bad test-code practices by introducing new test smells, (2) provide the software engineering community with an open-source test smell detection tool, and (3) perform a large-scale empirical study on test smell occurrence, distribution, and impact on the maintenance of open-source Android apps. Through multiple experiments, our findings indicate that most Android apps lack an automated verification of their testing mechanisms. As for the apps with existing test suites, they exhibit test smells early on in their lifetime with varying degrees of co-occurrences with different smell types. Our exploration of the relationship between test smells and technical debt proves that test smells are a strong measurement of technical debt. Furthermore, we observed positive correlations between specific smell types and highly changed/buggy test files. Hence, this research demonstrates that test smells can be used as indicators for necessary preventive software maintenance for test suites

Generic Quality-Aware Refactoring and Co-Refactoring in Heterogeneous Model Environments

Software has been subject to change, at all times, in order to make parts of it, for instance, more reusable, better to understand by humans, or to increase efficiency under a certain point of view. Restructurings of existing software can be complex. To prevent developers from doing this manually, they got tools at hand being able to apply such restructurings automatically. These automatic changes of existing software to improve quality while preserving its behaviour is called refactoring. Refactoring is well investigated for programming languages and mature tools exist for executing refactorings in integrated development environments (IDEs). In recent years, the development paradigm of Model-Driven Software Development (MDSD) became more and more popular and we experience a shift in the sense that development artefacts are considered as models which conform metamodels. This can be understood as abstraction, which resulted in the trend that a plethora of new so-called model-based Domain-Specific Languages (DSLs) arose. DSLs have become an integral part in the MDSD and it is obvious that models are subject to change, as well. Thus, refactoring support is required for DSLs in order to prevent users from doing it manually. The problem is that the amount of DSLs is huge and refactorings should not be implemented for new for each of them, since they are quite similar from an abstract viewing. Existing approaches abstract from the target language, which is not flexible enough because some assumptions about the languages have to be made and arbitrary DSLs are not supported. Furthermore, the relation between a strategy which finds model deficiencies that should be improved, a resolving refactoring, and the improved quality is only implicit. Focussing on a particular quality and only detecting those deficiencies deteriorating this quality is difficult, and elements of detected deficient structures cannot be referred to in the resolving refactoring. In addition, heterogeneous models in an IDE might be connected physically or logically, thus, they are dependent. Finding such connections is difficult and can hardly be achieved manually. Applying a restructuring in a model implied by a refactoring in a dependent model must also be a refactoring, in order to preserve the meaning. Thus, this kind of dependent refactorings require an appropriate abstraction mechanism, since they must be specified for dependent models of different DSLs. The first contribution, Role-Based Generic Model Refactoring, uses role models to abstract from refactorings instead of the target languages. Thus, participating structures in a refactoring can be specified generically by means of role models. As a consequence, arbitrary model-based DSLs are supported, since this approach does not make any assumptions regarding the target languages. Our second contribution, Role-Based Quality Smells, is a conceptual framework and correlates deficiencies, their deteriorated qualities, and resolving refactorings. Roles are used to abstract from the causing structures of a deficiency, which then are subject to resolving refactorings. The third contribution, Role-Based Co-Refactoring, employs the graph-logic isomorphism to detect dependencies between models. Dependent refactorings, which we call co-refactorings, are specified on the basis of roles for being independent from particular target DSLs. All introduced concepts are implemented in our tool Refactory. An evaluation in different scenarios complements the thesis. It shows that role models emerged as very powerful regarding the reuse of generic refactorings in arbitrary languages. Role models are suited as an interface for certain structures which are to be refactored, scanned for deficiencies, or co-refactored. All of the presented approaches benefit from it.:List of Figures xv List of Tables xvii List of Listings xix 1. Introduction 1 1.1. Language-Tool Generation Without Consideration Of Time And Space . . . . . 4 1.2. Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3. Generic Quality-Aware Refactoring and Co-Refactoring in Heterogeneous Model Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2. Foundations 15 2.1. Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2. Model-Driven Software Development . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2.1. Levels of Abstraction and Metamodelling . . . . . . . . . . . . . . . . . 17 2.2.2. Model Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.3. Role-Based Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3. Related Work 23 3.1. Model Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.1.1. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.1.2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.1.3. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.2. Determination of Quality-Related De ciencies . . . . . . . . . . . . . . . . . . . 32 3.2.1. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2.3. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.3. Co-Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.3.1. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.3.2. Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3.3. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4. Role-Based Generic Model Refactoring 51 4.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2. Specifying Generic Refactorings with Role Models . . . . . . . . . . . . . . . . . 53 4.2.1. Specifying Structural Constraints using Role Models . . . . . . . . . . . 55 4.2.2. Mapping Roles to Language Concepts Using Role Mappings . . . . . . . 57 4.2.3. Specifying Language-Independent Transformations using Refactoring Speci cations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.2.4. Composition of Refactorings . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3. Preserving Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5. Suggesting Role Mappings as Concrete Refactorings 73 5.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.2. Automatic Derivation of Suggestions for Role Mappings with Graph Querying . 74 5.3. Reduction of the Number of Valid Matches . . . . . . . . . . . . . . . . . . . . . 76 5.4. Comparison to Model Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6. Role-Based Quality Smells as Refactoring Indicator 79 6.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.2. Correlating Model De ciencies, Qualities and Refactorings . . . . . . . . . . . . 80 6.2.1. Quality Smell Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.2.2. Quality Smell Calculation Repository . . . . . . . . . . . . . . . . . . . . 85 6.3. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 7. A Quality Smell Catalogue for Android Applications 89 7.1. Quality Smell Catalogue Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.2. Acquiring Quality Smells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7.3. Structure-Based Quality Smells—A Detailed Example . . . . . . . . . . . . . . . 92 7.3.1. The Pattern Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 7.3.2. Quality Smell: Interruption from Background . . . . . . . . . . . . . . . 93 7.4. Quality Smells for Android Applications . . . . . . . . . . . . . . . . . . . . . . 96 7.4.1. Quality Smell: Data Transmission Without Compression . . . . . . . . . 96 7.4.2. Quality Smell: Dropped Data . . . . . . . . . . . . . . . . . . . . . . . . 98 7.4.3. Quality Smell: Durable WakeLock . . . . . . . . . . . . . . . . . . . . . 98 7.4.4. Quality Smell: Internal Use of Getters/Setters . . . . . . . . . . . . . . . 99 7.4.5. Quality Smell: No Low Memory Resolver . . . . . . . . . . . . . . . . . 101 7.4.6. Quality Smell: Rigid AlarmManager . . . . . . . . . . . . . . . . . . . . 101 7.4.7. Quality Smell: Unclosed Closeable . . . . . . . . . . . . . . . . . . . . . 102 7.4.8. Quality Smell: Untouchable . . . . . . . . . . . . . . . . . . . . . . . . . 103 7.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 8. Role-Based Co-Refactoring in Multi-Language Development Environments 105 8.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.2. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 8.3. Dependency Knowledge Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 8.3.1. Categories of Model Dependencies . . . . . . . . . . . . . . . . . . . . . 108 8.3.2. When to Determine Model Dependencies . . . . . . . . . . . . . . . . . 110 8.3.3. How to Determine Model Dependencies . . . . . . . . . . . . . . . . . . 111 8.4. Co-Refactoring Knowledge Base . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.4.1. Specifying Coupled Refactorings with Co-Refactoring Speci cations . . 114 8.4.2. Specifying Bindings for Co-Refactorings . . . . . . . . . . . . . . . . . . 116 8.4.3. Determination of Co-Refactoring Speci cations . . . . . . . . . . . . . . 118 8.5. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 8.6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 9. Refactory: An Eclipse Tool For Quality-Aware Refactoring and Co-Refactoring 121 9.1. Refactoring Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 9.1.1. Role Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 9.1.2. Refactoring Speci cation . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 9.1.3. Role Model Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 9.1.4. Refactoring Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 9.1.5. Custom Refactoring Extensions . . . . . . . . . . . . . . . . . . . . . . . 129 9.1.6. Pre- and Post-conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 9.1.7. Integration Into the Eclipse Refactoring Framework . . . . . . . . . . . . 130 9.2. Quality Smell Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 9.3. Co-Refactoring Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 9.3.1. Concrete Syntax of a CoRefSpec . . . . . . . . . . . . . . . . . . . . . . . 138 9.3.2. Expression Evaluation by Using an Expression Language . . . . . . . . . 138 9.3.3. UI and Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 10. Evaluation 143 10.1. Case Study: Reuse of Generic Refactorings in many DSLs . . . . . . . . . . . . . 143 10.1.1. Threats to validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 10.1.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 10.1.3. Experience Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 10.2. Case Study: Suggestion of Valid Role Mappings . . . . . . . . . . . . . . . . . . 147 10.2.1. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 10.2.2. Evaluation and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 151 10.3. Proof of Concept: Co-Refactoring OWL and Ecore Models . . . . . . . . . . . . 155 10.3.1. Coupled OWL-Ecore Refactorings . . . . . . . . . . . . . . . . . . . . . 156 10.3.2. Realisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 10.3.3. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 11. Summary, Conclusion and Outlook 161 11.1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 11.2. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 11.3. Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Appendix 169 A. List of Role Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 B. Comparison to Role Feature Model . . . . . . . . . . . . . . . . . . . . . . . . . 171 C. Complete List of Role Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 D. List of all IncPL Patterns for Detecting Quality Smells . . . . . . . . . . . . . . . 176 E. Post-Processor of the Extract CompositeState refactoring for UML State Machines 183 F. Speci cation of the Conference Language . . . . . . . . . . . . . . . . . . . . . . 185 List of Abbreviations 187 Bibliography 19

When testing meets code review: why and how developers review tests

Automated testing is considered an essential process for ensuring software quality. However, writing and maintaining high-quality test code is challenging and frequently considered of secondary importance. For production code, many open source and industrial software projects employ code review, a well-established software quality practice, but the question remains whether and how code review is also used for ensuring the quality of test code. The aim of this research is to answer this question and to increase our understanding of what developers think and do when it comes to reviewing test code. We conducted both quantitative and qualitative methods to analyze more than 300,000 code reviews, and interviewed 12 developers about how they review test files. This work resulted in an overview of current code reviewing practices, a set of identified obstacles limiting the review of test code, and a set of issues that developers would like to see improved in code review tools. The study reveals that reviewing test files is very different from reviewing production files, and that the navigation within the review itself is one of the main issues developers currently face. Based on our findings, we propose a series of recommendations and suggestions for the design of tools and future research

Quality Assessment and Quality Improvement for UML Models

UML-Modelle sind heutzutage Teil der Dokumentation, der Spezifikation und manchmal sogar der Implementierung von Softwaresystemen. Allerdings kann UML sehr unterschiedlich benutzt werden. Die Gründe hierfür sind vielfältig. So fehlen zum Beispiel allgemein akzeptierte Normen und Richtlinien für die Verwendung von UML. Des Weiteren ist die Sprache UML sehr komplex und Teile der Sprache besitzen nur eine semi-formale Semantik. All diese Faktoren führen zu Qualitätsproblemen bei UML-Modellen, die untersucht und bearbeitet werden müssen.In der vorliegenden Arbeit wird ein Verfahren für eine integrierte und kontinuierliche Qualitätsbewertung und -verbesserung von UML-Modellen vorgestellt. Das Verfahren basiert auf einem neuen Qualitätsmodell für UML-Modelle, dessen exemplarische Instanziierung in der Arbeit beschrieben wird. Es wurde ein prototypisches Werkzeug entwickelt, mit dessen Hilfe die Qualitätsbeurteilung und die -verbesserung von UML-Modellen automatisiert durchgeführt werden kann.Zum Nachweis der Anwendbarkeit des vorgestellten Verfahrens wurde eine Fallstudie im Rahmen eines UML-Praktikums durchgeführt. Die Qualität der während des Praktikums entwickelten UML-Modelle wurde kontinuierlich bewertet und identifizierte Qualitätsprobleme mussten von den Teilnehmern des Praktikums fortlaufend beseitigt werden. Die Ergebnisse der Fallstudie unterstreichen die praktische Anwendbarkeit und das hohe Potential des vorgestellten Verfahrens zur automatisierten Qualitätsbewertung und -verbesserung von UML-Modellen