How Good Are My Tests?

Background: Test quality is a prerequisite for achieving production system quality. While the concept of quality is multidimensional, most of the effort in testing context hasbeen channelled towards measuring test effectiveness. Objective: While effectiveness of tests is certainly important, we aim to identify a core list of testing principles that also address other quality facets of testing, and to discuss how they can be quantified as indicators of test quality. Method: We have conducted a two-day workshop with our industry partners to come up with a list of relevant principles and best practices expected to result in high quality tests. We then utilised our academic and industrial training materials together with recommendations in practitioner oriented testing books to refine the list. We surveyed existing literature for potential metrics to quantify identified principles. Results: We have identified a list of 15 testing principles to capture the essence of testing goals and best practices from quality perspective. Eight principles do not map toexisting test smells and we propose metrics for six of those. Further, we have identified additional potential metrics for the seven principles that partially map to test smells. Conclusion: We provide a core list of testing principles along with a discussion of possible ways to quantify them for assessing goodness of tests. We believe that our work would be useful for practitioners in assessing the quality of their tests from multiple perspectives including but not limited to maintainability, comprehension and simplicity

On the Effectiveness of Unit Tests in Test-driven Development

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice. Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature. Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.Istanbul Technical University Scientific Research Projects (MGA-2017-40712), and the Academy of Finland (Decision No. 278354)

Test case quality: an empirical study on belief and evidence

Software testing is a mandatory activity in any serious software development process, as bugs are a reality in software development. This raises the question of quality: good tests are effective in finding bugs, but until a test case actually finds a bug, its effectiveness remains unknown. Therefore, determining what constitutes a good or bad test is necessary. This is not a simple task, and there are a number of studies that identify different characteristics of a good test case. A previous study evaluated 29 hypotheses regarding what constitutes a good test case, but the findings are based on developers' beliefs, which are subjective and biased. In this paper we investigate eight of these hypotheses, through an extensive empirical study based on open software repositories. Despite our best efforts, we were unable to find evidence that supports these beliefs. This indicates that, although these hypotheses represent good software engineering advice, they do not necessarily mean that they are enough to provide the desired outcome of good testing code.Comment: 12 pages, 1 figure, 3 table

Exposing Test Analysis Results with DrTests

International audienceTests are getting the cornerstone of continuous development process and software evolution. Tests are the new gold. To improve test quality, a plethora of analyses is proposed such as test smells, mutation testing, test coverage. The problem is that each analysis often needs a particular way to expose its results to the developer. There is a need for an architecture supporting test running and analysis in a modular and extensible way. In this article we present an extensi-ble plugin-based architecture to run and report test results. DrTests is a new test browser that implements such plugin-based architecture. DrTests supports the execution of rotten tests, comments to tests, coverage and profiling tests

Which Software Faults Are Tests Not Detecting?

Context: Software testing plays an important role in assuring the reliability of systems. Assessing the efficacy of testing remains challenging with few established test effectiveness metrics. Those metrics that have been used (e.g. coverage and mutation analysis) have been criticised for insufficiently differentiating between the faults detected by tests. Objective: We investigate how effective tests are at detecting different types of faults and whether some types of fault evade tests more than others. Our aim is to suggest to developers specific ways in which their tests need to be improved to increase fault detection. Method: We investigate seven fault types and analyse how often each goes undetected in 10 open source systems. We statistically look for any relationship between the test set and faults. Results: Our results suggest that the fault detection rates of unit tests are relatively low, typically finding only about a half of all faults. In addition, conditional boundary and method call removals are less well detected by tests than other fault types. Conclusions: We conclude that the testing of these open source systems needs to be improved across the board. In addition, despite boundary cases being long known to attract faults, tests covering boundaries need particular improvement. Overall, we recommend that developers do not rely only on code coverage and mutation score to measure the effectiveness of their tests

Suporte à Evolução de API

A evolução de uma Application Programming Interface (API) é o conjunto das mudanças que ocorrem nessa API ao longo do tempo. Portanto, sempre que ocorre uma mudança evolutiva, existem componentes relacionadas com a API que sofrem alterações, tais como a documentação e as bibliotecas de código cliente. Assim sendo, ao desenhar uma API, especialmente seguindo uma abordagem design-first, é relevante definir um contrato formal, que deve conter toda a informação necessária sobre o comportamento da API, sendo conhecido como especificação de API. Desta forma, todas as componentes relacionadas podem depender do mesmo contrato, o que implica que, caso ocorra uma evolução na API, o contrato é alterado e todas essas componentes podem ser atualizadas dinamicamente com a nova informação. Portanto, este contrato pode ser utilizado para gerar componentes como documentação, código cliente e testes de software, reduzindo custos de desenvolvimento e manutenção. Neste documento, são apresentadas soluções para a API pública do E-goi. O E-goi é uma lataforma de marketing digital multicanal e para alcançar uma solução que resolva os problemas encontrados nesta API, maioritariamente relacionados com evolução e altos custos de manutenção do software, é necessário estudar o estilo arquitetural Representational State Transfer (REST) e linguagens para especificação de API REST. Também devem ser estudadas outras API públicas na área de marketing, diferentes tipos de versionamento e registos de utilização da API que sejam relevantes, uma vez que é através destes registos que se podem encontrar potenciais falhas e melhorias que podem resultar em mudanças evolutivas na API. A arquitetura das soluções desenvolvidas e outras alternativas arquiteturais para geração de ocumentação, código cliente e testes estão descritas e apresentadas utilizando Unified Modelling Language (UML). Seguidamente é apresentado o design a um nível mais baixo, conjuntamente com alguns aspetos de implementação importantes. Por fim é feita a experimentação e avaliação para verificar se esta cumpre todos os objetivos e resolve os problemas identificados.Evolution can be understood as the changes that occur in an Application Programming Interface (API) over time. Therefore, every time an evolutive change occurs, there are components related to the API that also change, such as documentation and client code libraries. Thereby, when designing an API, especially when following a design-first approach, it is relevant to define a formal contract, known as API specification, which must contain every information regarding the API behavior. Being so, every related component can depend on the same contract, meaning that when an evolution on the API occurs, the contract is altered, and every componente may be updated dynamically with the new information. Thus, this contract may be used to generate components like documentation, client code and software tests, reducing development and maintenance costs. In this document, it is presented a set of solutions for E-goi’s public API. Egoi is a multichannel digital marketing platform and, to achieve a solution to solve these API problems, majorly related to evolution and high software maintenance costs, it is necessary to study Representational State Transfer (REST) architectural style and languages for specification of REST API. Also, it must be studied other public API in marketing area, different sorts of versioning and important records of API usage, once these records are used to find potential flaws and improvements that may result in evolutive changes in the API. The architecture of the developed solutions and other architectural alternatives to the generation of documentation, client code and tests are descripted and presented using Unified Modelling Language (UML). Then, it is presented the design in a lower level, together with some important implementation aspects. Finally, it is done the experimentation and evaluation to verify if the solution matches all the required objectives and solves the identified problems

Smells in system user interactive tests

peer reviewedTest smells are known as bad development practices that reflect poor design and implementation choices in software tests. Over the last decade, there are few attempts to study test smells in the context of system tests that interact with the System Under Test through a Graphical User Interface. To fill the gap, we conduct an exploratory analysis of test smells occurring in System User Interactive Tests (SUIT). We thus, compose a catalog of 35 SUIT-specific smells, identified through a multi-vocal literature review, and show how they differ from smells encountered in unit tests. We also conduct an empirical analysis to assess the diffuseness and removal of these smells in 48 industrial repositories and 12 open-source projects. Our results show that the same type of smells tends to appear in both industrial and open-source projects, but they are not addressed in the same way. We also find that smells originating from a combination of multiple code locations appear more often than those that are localized on a single line. This happens because of the difficulty to observe non-local smells without tool support. Furthermore, we find that smell-removing actions are not frequent with less than 50% of the affected tests ever undergoing a smell removal. Interestingly, while smell-removing actions are rare, some smells disappear while discarding tests, i.e., these smells do not appear in follow-up tests that replace the discarded ones