A survey on test suite reduction frameworks and tools

Software testing is a widely accepted practice that ensures the quality of a System under Test (SUT). However, the gradual increase of the test suite size demands high portion of testing budget and time. Test Suite Reduction (TSR) is considered a potential approach to deal with the test suite size problem. Moreover, a complete automation support is highly recommended for software testing to adequately meet the challenges of a resource constrained testing environment. Several TSR frameworks and tools have been proposed to efficiently address the test-suite size problem. The main objective of the paper is to comprehensively review the state-of-the-art TSR frameworks to highlights their strengths and weaknesses. Furthermore, the paper focuses on devising a detailed thematic taxonomy to classify existing literature that helps in understanding the underlying issues and proof of concept. Moreover, the paper investigates critical aspects and related features of TSR frameworks and tools based on a set of defined parameters. We also rigorously elaborated various testing domains and approaches followed by the extant TSR frameworks. The results reveal that majority of TSR frameworks focused on randomized unit testing, and a considerable number of frameworks lacks in supporting multi-objective optimization problems. Moreover, there is no generalized framework, effective for testing applications developed in any programming domain. Conversely, Integer Linear Programming (ILP) based TSR frameworks provide an optimal solution for multi-objective optimization problems and improve execution time by running multiple ILP in parallel. The study concludes with new insights and provides an unbiased view of the state-of-the-art TSR frameworks. Finally, we present potential research issues for further investigation to anticipate efficient TSR frameworks

Javan yksikkö- ja integraatiotestaus: JVM:n käyttäytymisvetoiset testaustyökalut vastaan JUnit

This master’s thesis studied how do Behavior-Driven Development testing frameworks change the testing of Java-code compared to JUnit. The research was done as a case study. The case study was conducted in industry context at Vincit Plc, were two projects changed new unit and integration tests classes to use a new BDD-testing framework instead of JUnit. Before designing the study methods, related research and their findings were reviewed to guide the study to inspect problematic areas found in unit testing. Case study data collection methods included surveys, interviews and test code analysis. Case study provided promising results for problematic areas highlighted by earlier research. To summarize the developer practice changes, the collected data displayed an increase in unit test case granularity. Results also displayed unanimously that BDD-testing frameworks guide to write more self-documenting tests than JUnit. The structure of BDD tests highlighted better the different parts of the test. Study also revealed that the majority of participants had easier time understanding tests and removing repetition from test code. Developer perception changes in testing included the majority of study participants enjoying writing of tests more than with JUnit. The same majority also perceived that BDD-testing frameworks promote in writing higher quality test code than JUnit. Generally new test code was perceived more understandable and maintainable than tests with JUnit, although this was not unanimous. Learning curve to be effective varied between studied frameworks. Tool support of BDD-testing frameworks for testing Java Spring Framework were found ranging from adequate to good. In conclusion, this thesis results provide small scale evidence that BDD-testing frameworks could potentially ease the maintainability and readability of unit and integration tests while same time rising the enjoyment in testing.Tässä diplomityössä tutkittiin, kuinka käyttäytymisvetoiset testisovelluskehykset muuttavat Java-koodin testausta verrattuna JUnit:iin. Tutkimus suoritettiin tapaustutkimuksen menetelmin Vincit Oy:ssa. Tutkimukseen valittiin kaksi projektia, joissa uudet yksikkö- ja integraatiotestausluokat kirjoitettiin käyttäytymisvetoisilla testaussovelluskehyksillä JUnit:in sijaan. Työhön liittyvät aiempien tutkimusten havainnot ohjasivat työtä tarkastelemaan näissä löydettyjä ongelmallisia alueita. Tiedonkeruukeinoina käytettiin kyselyitä, haastatteluita sekä testikoodin analyysia. Työn tulokset osoittautuivat lupaaviksi ratkaisuksi aiemmin löydettyihin ongelmallisiin seikkoihin. Kokonaisuudessaan sovelluskehittäjien testauskäytännöissä löytyi useita muutoksia. Yksikkötestien rakenne ohjautui aiempaa hienojakoisemmaksi. Tulokset osoittivat myös yksimielisesti, että käyttäytymisvetoiset testaussovelluskehykset ohjaavat kirjoittamaan aiempaa paremmin itseänsä dokumentoivia testejä. Myös testin eri loogiset osat olivat uusien testien rakenteesta helpommin luettavissa. Suurimmalla osalla tutkimukseen osallistuneista testit olivat aiempaa helpompia ymmärtää sekä niistä oli helpompi poistaa toistoa. Suurin osa koki testien kirjoittamisen myös aiempaa nautittavampana. Valtaosa vastaajista koki uusien menetelmien ohjaavan kirjoittamaan laadukkaampaa testikoodia kuin aiemmin. Yleisesti ottaen uutta testikoodia pidettiin ymmärrettävämpänä ja ylläpidettävämpänä kuin JUnit testejä, tosin ei täysin yksimielisesti. Oppimiskäyrä uusien testauskehyksien parissa vaihteli tutkittujen kehysten välillä. Java Spring-sovelluskehyksen testaustuki vaihteli riittävästä tuesta hyvään tukeen. Kokonaisuudessaan työ tarjosi pienessä skaalassa näyttöä siitä, että käyttäytymisvetoiset testaussovelluskehykset voivat mahdollisesti helpottaa yksikkö- ja integraatiotestien ylläpidettävyyttä, luettavuutta sekä koettua nautintoa näiden parissa

Fulfilling a RESTful Commitment: Test-Suite Generation for Swagger-based REST APIs

An application programming interface (API) can be the most integral part of a product or service. APIs drive businesses and innovation in all aspects of technology and they rely on descriptive frameworks to detail what they have to offer. These descriptive frameworks are a rich source of information on a variety of levels, such as usage, best-practices and even basic implications towards the underlying implementation. The motivation of this project during my internship was to exploit this wealth of information in order to create a description-based, or contract-based, unit test generation tool specifically for Swagger-based REST APIs and to encourage contract-based development practices. Beyond my internship, I have developed this tool as my senior project with the goal of broadening its use-case beyond unit testing and into integration testing. Thus, swagger-test-templates is a Node.js based tool designed to consume Swagger-described APIs and generate comprehensive unit tests and specified integration test scenarios

AUnit - a testing framework for alloy

textWriting declarative models of software designs and analyzing them to detect defects is an effective methodology for developing more dependable software systems. However, writing such models correctly can be challenging for practitioners who may not be proficient in declarative programming, and their models themselves may be buggy. We introduce the foundations of a novel test automation framework, AUnit, which we envision for testing declarative models written in Alloy -- a first-order, relational language that is supported by its SAT-based analyzer. We take inspiration from the success of the family of xUnit frameworks that are used widely in practice for test automation, albeit for imperative or object-oriented programs. The key novelty of our work is to define a basis for unit testing for Alloy, specifically, to define the concepts of test case and test coverage as well as coverage criteria for declarative models. We reduce the problems of declarative test execution and coverage computation to partial evaluation without requiring SAT solving. Our vision is to blend how developers write unit tests in commonly used programming languages with how Alloy users formulate their models in Alloy, thereby facilitating the development and testing of Alloy models for both new Alloy users as well as experts. We illustrate our ideas using a small but complex Alloy model. While we focus on Alloy, our ideas generalize to other declarative languages (such as Z, B, ASM).Electrical and Computer Engineerin

Local Asymptotic Equivalence of the Bai and Ng (2004) and Moon and Perron (2004) Frameworks for Panel Unit Root Testing

This paper considers unit-root tests in large n and large T heterogeneous panels with cross-sectional dependence generated by unobserved factors. We reconsider the two prevalent approaches in the literature, that of Moon and Perron (2004) and the PANIC setup proposed in Bai and Ng (2004). While these have been considered as completely different setups, we show that, in case of Gaussian innovations, the frameworks are asymptotically equivalent in the sense that both experiments are locally asymptotically normal (LAN) with the same central sequence. Using Le Cam's theory of statistical experiments we determine the local asymptotic power envelope and derive an optimal test jointly in both setups. We show that the popular Moon and Perron (2004) and Bai and Ng (2010) tests only attain the power envelope in case there is no heterogeneity in the long-run variance of the idiosyncratic components. The new test is asymptotically uniformly most powerful irrespective of possible heterogeneity. Moreover, it turns out that for any test, satisfying a mild regularity condition, the size and local asymptotic power are the same under both data generating processes. Thus, applied researchers do not need to decide on one of the two frameworks to conduct unit root tests. Monte-Carlo simulations corroborate our asymptotic results and document significant gains in finite-sample power if the variances of the idiosyncratic shocks differ substantially among the cross sectional units

Parameterized Unit Testing in the Open Source Wild

With recent advances in test generation research, powerful test generation tools are now at the fingertips of developers in software industry. For example, Microsoft Research Pex, a state-of-the-art tool based on dynamic symbolic execution, has been shipped as IntelliTest in Visual Studio 2015. For test inputs automatically generated by such tool, to supply test oracles (beyond just uncaught runtime exceptions or crashes), developers can write formal specifications such as code contracts in the form of preconditions, postconditions, and class invariants. However, just like writing other types of formal specifications, writing code contracts, especially postconditions, is challenging. In the past decade, parameterized unit testing has emerged as a promising alternative to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. PUTs have been popularly supported by various unit testing frameworks for .NET along with the recent JUnit framework. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, in this paper, we present the first empirical study of parameterized unit testing conducted on open source projects. We study hundreds of parameterized unit tests that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.Ope

Did You Remember to Test Your Tokens?

Authentication is a critical security feature for confirming the identity of a system's users, typically implemented with help from frameworks like Spring Security. It is a complex feature which should be robustly tested at all stages of development. Unit testing is an effective technique for fine-grained verification of feature behaviors that is not widely-used to test authentication. Part of the problem is that resources to help developers unit test security features are limited. Most security testing guides recommend test cases in a "black box" or penetration testing perspective. These resources are not easily applicable to developers writing new unit tests, or who want a security-focused perspective on coverage. In this paper, we address these issues by applying a grounded theory-based approach to identify common (unit) test cases for token authentication through analysis of 481 JUnit tests exercising Spring Security-based authentication implementations from 53 open source Java projects. The outcome of this study is a developer-friendly unit testing guide organized as a catalog of 53 test cases for token authentication, representing unique combinations of 17 scenarios, 40 conditions, and 30 expected outcomes learned from the data set in our analysis. We supplement the test guide with common test smells to avoid. To verify the accuracy and usefulness of our testing guide, we sought feedback from selected developers, some of whom authored unit tests in our dataset.Comment: In 17th International Conference on Mining Software Repositories (MSR) 2020, Technical Track, Virtual. 11 page

A Characteristic Study of Parameterized Unit Tests in .NET Open Source Projects

In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators