Mutation Testing as a Safety Net for Test Code Refactoring

Refactoring is an activity that improves the internal structure of the code without altering its external behavior. When performed on the production code, the tests can be used to verify that the external behavior of the production code is preserved. However, when the refactoring is performed on test code, there is no safety net that assures that the external behavior of the test code is preserved. In this paper, we propose to adopt mutation testing as a means to verify if the behavior of the test code is preserved after refactoring. Moreover, we also show how this approach can be used to identify the part of the test code which is improperly refactored

Search based software engineering: Trends, techniques and applications

© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version is available from the link below.In the past five years there has been a dramatic increase in work on Search-Based Software Engineering (SBSE), an approach to Software Engineering (SE) in which Search-Based Optimization (SBO) algorithms are used to address problems in SE. SBSE has been applied to problems throughout the SE lifecycle, from requirements and project planning to maintenance and reengineering. The approach is attractive because it offers a suite of adaptive automated and semiautomated solutions in situations typified by large complex problem spaces with multiple competing and conflicting objectives. This article provides a review and classification of literature on SBSE. The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.EPSRC and E

A heuristic-based approach to code-smell detection

Encapsulation and data hiding are central tenets of the object oriented paradigm. Deciding what data and behaviour to form into a class and where to draw the line between its public and private details can make the difference between a class that is an understandable, flexible and reusable abstraction and one which is not. This decision is a difficult one and may easily result in poor encapsulation which can then have serious implications for a number of system qualities. It is often hard to identify such encapsulation problems within large software systems until they cause a maintenance problem (which is usually too late) and attempting to perform such analysis manually can also be tedious and error prone. Two of the common encapsulation problems that can arise as a consequence of this decomposition process are data classes and god classes. Typically, these two problems occur together – data classes are lacking in functionality that has typically been sucked into an over-complicated and domineering god class. This paper describes the architecture of a tool which automatically detects data and god classes that has been developed as a plug-in for the Eclipse IDE. The technique has been evaluated in a controlled study on two large open source systems which compare the tool results to similar work by Marinescu, who employs a metrics-based approach to detecting such features. The study provides some valuable insights into the strengths and weaknesses of the two approache

An Annotation-based Approach for Finding Bugs in Neural Network Programs

As neural networks are increasingly included as core components of safety-critical systems, developing effective testing techniques specialized for them becomes crucial. The bulk of the research has focused on testing neural-network models; but these models are defined by writing programs, and there is growing evidence that these neural-network programs often have bugs too. This paper presents aNNoTest: an approach to generating test inputs for neural-network programs. A fundamental challenge is that the dynamically-typed languages (e.g., Python) commonly used to program neural networks cannot express detailed constraints about valid function inputs (e.g., matrices with certain dimensions). Without knowing these constraints, automated test-case generation is prone to producing invalid inputs, which trigger spurious failures and are useless for identifying real bugs. To address this problem, we introduce a simple annotation language tailored for concisely expressing valid function inputs in neural-network programs. aNNoTest takes as input an annotated program, and uses property-based testing to generate random inputs that satisfy the validity constraints. In the paper, we also outline guidelines that simplify writing aNNoTest annotations. We evaluated aNNoTest on 19 neural-network programs from Islam et al's survey., which we manually annotated following our guidelines -- producing 6 annotations per tested function on average. aNNoTest automatically generated test inputs that revealed 94 bugs, including 63 bugs that the survey reported for these projects. These results suggest that aNNoTest can be a valuable approach to finding widespread bugs in real-world neural-network programs.Comment: New content: (1) discussing test oracles in the related work section, (2) adding future work, and (3) discussing the possibility of using other back-end

Behave Nicely! Automatic Generation of Code for Behaviour Driven Development Test Suites

Behaviour driven development (BDD) has gained widespread use in the software industry. System specifications can be expressed as test scenarios, describing the circumstances, actions and expected outcomes. These scenarios are written in a structured natural language (Gherkin), with each step in the scenario associated with a corresponding step implementation function in the underlying programming language. A challenge recognised by industry is ensuring that the natural language scenarios, step implementation functions and underlying system implementation remain consistent with one another, requiring on-going maintenance effort as changes are made to a system. To address this, we have developed behave_nicely, a tool, for automatically generating step implementation functions from structured natural language steps, with the intention of eliminating the need for maintaining step implementation functions. We evaluated our approach on a sample of 20 white box and 50 black box projects using behaviour driven development, drawn from GitHub. Our results show that behave_nicely can generate step implementation functions for 80% of the white box and 17% of black box projects. We conclude that (a) there is significant potential for automating the process of code generation for BDD tests and (b) that the development of guidelines for writing tests in Gherkin would significantly improve the results

Evaluating Random Mutant Selection at Class-Level in Projects with Non-Adequate Test Suites

Mutation testing is a standard technique to evaluate the quality of a test suite. Due to its computationally intensive nature, many approaches have been proposed to make this technique feasible in real case scenarios. Among these approaches, uniform random mutant selection has been demonstrated to be simple and promising. However, works on this area analyze mutant samples at project level mainly on projects with adequate test suites. In this paper, we fill this lack of empirical validation by analyzing random mutant selection at class level on projects with non-adequate test suites. First, we show that uniform random mutant selection underachieves the expected results. Then, we propose a new approach named weighted random mutant selection which generates more representative mutant samples. Finally, we show that representative mutant samples are larger for projects with high test adequacy.Comment: EASE 2016, Article 11 , 10 page

A Model to Estimate First-Order Mutation Coverage from Higher-Order Mutation Coverage

The test suite is essential for fault detection during software development. First-order mutation coverage is an accurate metric to quantify the quality of the test suite. However, it is computationally expensive. Hence, the adoption of this metric is limited. In this study, we address this issue by proposing a realistic model able to estimate first-order mutation coverage using only higher-order mutation coverage. Our study shows how the estimation evolves along with the order of mutation. We validate the model with an empirical study based on 17 open-source projects.Comment: 2016 IEEE International Conference on Software Quality, Reliability, and Security. 9 page

Reasoning and Improving on Software Resilience against Unanticipated Exceptions

In software, there are the errors anticipated at specification and design time, those encountered at development and testing time, and those that happen in production mode yet never anticipated. In this paper, we aim at reasoning on the ability of software to correctly handle unanticipated exceptions. We propose an algorithm, called short-circuit testing, which injects exceptions during test suite execution so as to simulate unanticipated errors. This algorithm collects data that is used as input for verifying two formal exception contracts that capture two resilience properties. Our evaluation on 9 test suites, with 78% line coverage in average, analyzes 241 executed catch blocks, shows that 101 of them expose resilience properties and that 84 can be transformed to be more resilient

Do Null-Type Mutation Operators Help Prevent Null-Type Faults?

The null-type is a major source of faults in Java programs, and its overuse has a severe impact on software maintenance. Unfortunately traditional mutation testing operators do not cover null-type faults by default, hence cannot be used as a preventive measure. We address this problem by designing four new mutation operators which model null-type faults explicitly. We show how these mutation operators are capable of revealing the missing tests, and we demonstrate that these mutation operators are useful in practice. For the latter, we analyze the test suites of 15 open-source projects to describe the trade-offs related to the adoption of these operators to strengthen the test suite

Enhancing unit testing to improve maintainability of the software

Many companies are using software that have been developed many years ago. These legacy programs are important to their users and often their development is still ongoing. New features are developed and old ones adapted to suit the current needs of the users. Legacy programs often have quality problems, which makes it increasingly difficult to maintain them. But because of their size, numerous features, and business rules that are documented only in the code, it is difficult to replace them. Therefore, it is imperative to improve their quality safely without introducing defects to the current features. Quality can be enhanced by unit testing and refactoring, but in legacy projects writing unit tests and refactoring is usually error prone and difficult due to the characteristics of legacy software. Taking quality into account when developing new and legacy software is important to ensure the software can be developed further and used in the future. High quality code is maintainable even when it has been developed a long time. First in this thesis, legacy software problems in quality, refactoring and unit testing were researched. Then, solutions to avoid the problems and increase the quality of software and especially legacy software mainly by safe refactoring and unit testing were searched from literature and research publications. Quality control and metrics were also included in the thesis to supplement the quality enhancement process. In this thesis, it was found that safe refactoring, unit test quality, code and test quality control, and developer training need to be considered to improve software quality. Refactoring should be done in small steps while writing tests to ensure the current functionality does not change. Unit test quality, especially, test isolation, readability, and test focus need to be considered while writing tests. The quality of the tests need to be monitored to maintain a proper quality level. It is also important to support the developers for them to improve and maintain their unit testing and refactoring skills.Monessa yrityksessä on käytössä ohjelmistoja, joiden kehitys on aloitettu useita vuosia sitten. Nämä legacy-ohjelmistot ovat käyttäjilleen tärkeitä ja usein niiden kehitys jatkuu koko ajan. Uusia ominaisuuksia lisätään ja vanhoja muokataan täyttämään käyttäjien tarpeita. Legacy-ohjelmistot sisältävät usein laatuongelmia, jonka takia niiden ylläpitäminen ja kehittäminen vaikeutuu koko ajan. Niitä ei voida kuitenkaan helposti korvata kokonsa, lukuisten ominaisuuksiensa ja vain koodiin dokumentoidun tietonsa takia. Tämän takia niiden laatua pitäisi saada parannettua turvallisesti ilman, että virheitä ilmestyy toimiviin ominaisuuksiin. Yksikkötestauksen ja refaktoroinnilla voidaan parantaa ohjelmiston laatua, mutta niiden käyttönottaminen ja käyttäminen legacy-ohjelmistossa voi olla riskialtista ja vaikeaa legacy-ohjelmiston ominaisuuksien takia. Laadun huomioonottaminen uuden ja legacy-ohjelmistoa kehittäessä on tärkeää, jotta ohjelmistoa pystytään kehittämään edelleen ja käyttämään myös tulevaisuudessa. Korkealaatuinen koodi on melko ylläpidettävää myös, kun sitä on kehitetty pidemmän aikaa. Aluksi tässä työssä selvitettiin, mitä ongelmia kuuluu legacy-ohjelmiston yksikkötestaukseen ja refaktorointiin. Sen jälkeen etsittiin ratkaisuja, kuinka välttää näitä ongelmia ja kuinka parantaa ohjelmien erityisesti legacy-ohjelmistojen laatua turvallisella refaktoroinnilla ja yksikkötestauksella. Tapoja etsittiin kirjallisuudesta ja tutkimusjulkaisuista. Laadun valvonta ja metriikat otettiin täydentämään laadun parannusprosessia Työn aikana saatiin selville, että ohjelmiston laadun parantamisessa pitää ensisijaisesti huomioda testien laatu, turvalliset refaktorointitavat, koodin ja testien laadunvalvonta ja kehittäjien osaamisen kehittäminen. Refaktorointi tulee tehdä pienissä askeleissa kurinalaisesti ja testien avustamana. Yksikkötestien laadussa tulee erityisesti ottaa huomioon eristyneisyys, luettavuus ja kohdennus. Koodin ja testien laatua tulee valvoa, että se pysyy yllä. Kehittäjiä täytyy myös tukea, jotta heidän yksikkötestaus- ja refaktorointitaitonsa kasvaisivat ja pysyisivät yllä