Uma abordagem reativa de construção de linhas de produto de software baseada em TDD e refatoração

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Ciência da Computação, Florianópolis, 2014Linhas de Produto de Software (LPS) trazem diversos benefícios, como a diminuição do tempo de entrada no mercado, a redução dos custos de desenvolvimento, o aumento da produtividade e a melhora da qualidade do produto final. Uma das práticas que auxilia na garantia de qualidade é a prática de testes. No entanto, ainda existem desafios e lacunas na utilização desta prática no desenvolvimento de LPSs. Nem todas as técnicas de testes utilizadas no desenvolvimento de um produto único podem ser aplicadas em LPSs e, portanto, novas adaptações e propostas são necessárias. Além disso, o desenvolvimento tradicional de LPSs também demanda um alto investimento inicial de longo prazo e oferece riscos para mercados dinâmicos, onde mudanças são difíceis de prever. Entretanto, já existem propostas para levar as vantagens de LPSs para mercados dinâmicos por meio da utilização de práticas de desenvolvimento ágil de software, essa união é denominada Engenharia de Linha de Produto Ágil (ELPA). Esta dissertação visa a elaboração de uma abordagem para a construção de LPSs, utilizando a ELPA. Nesta proposta as práticas ágeis de Desenvolvimento Dirigido por Testes (Test-Driven Development - TDD) e Refatoração conduzem a criação de uma LPS de abordagem reativa sem a tentativa de prever variações futuras. Para dar suporte à prática de testes unitários no desenvolvimento reativo de LPSs, foi desenvolvido um framework de testes com a proposta de adaptar padrões de testes unitários que facilitem a verificação da exatidão das aplicações geradas. Os padrõesde teste Framework de Automação de Testes e Testes Dirigidos por Dados fornecem a reutilização da lógica de testes e a automação dos mecanismos de implementação, reduzindo o esforço necessário para testar as variações de cada aplicação. A proposta foi avaliada através de um exemplo que mostrou a aplicação da abordagem e do framework de testes em uma LPS que foi construída de forma reativa a partir de uma aplicação existente. Como resultado foi visto um alto grau de reuso de testes, com 89% de reuso na segunda aplicação, após a modificação de três features, 97% na terceira aplicação, após a adição de uma feature e modificação de outra, e 100% na quarta aplicação onde nenhuma feature foi adicionada ou modificada, e a aplicação foi construída com variantes existentes.Abstract: Software Product Line (SPL) brings benefits such as lower time-to market, less development costs, increased productivity, and improved quality. The quality assurance can be reached through the testing area, however this area still has challenges and gaps in the SPL development. Since not all testing techniques used in a single product development can be applied to SPL, thus some adaptations and new proposals are required. Traditional SPL also requires a high initial investment and offers long-term risks to dynamic markets where changes are difficult to predict. Currently, proposals bring the advantages of SPL for dynamic markets through the use of agile software development practices, which is called Agile Product Line Engineering (APLE). This work presents the development of the necessary steps for building SPL using the APLE. In this proposal the agile practices Test-Driven Development (TDD) and Refactoring lead a reactive development of SPL without attempting to predict future variations. It is also proposed to adapt unit test patterns in the context of SPL. The test patterns Test Automation Framework and Data-Driven Tests provide the reuse of test logic and automation of the implementation mechanisms, reducing the effort required to test variations of each application. The result of this adaptation is a testing framework to be used during application engineering to configure tests through parameterized tests and verify the correctness of generated applications. Thus, this work shows how the agile practices TDD and Refactoring may cause a SPL to evolve and acquire variation points on demand. The proposal was evaluated through an example of a SPL that was built with a reactive approach from an existing application. As a result, it was obtained a high degree of tests reuse, with 89% of reuse in the second application after modifying three features, 97% in the third application after adding one feature and modifying another one, and 100% of reuse in the fourth application where no feature was added or modified, and the application was built with existing variants

Applying test case prioritization to software microbenchmarks

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative

Capturing Performance Assumptions using Stochastic Performance Logic

Performance testing is a mean used to evaluate speed of software projects. In an ideal state a project has a set of tests attached to it and such set may be repeat- edly executed in order to verify that all performance expectations are satisfied. The most widespread method of constructing these tests nowadays is based on measuring absolute time values. A test executes a chosen application unit and then compares the time it took to complete with a precise bound, which has been determined in advance. However, this approach has several disadvantages that affect reliability of such tests. First of all, the way in which those precise bounds should be established is not clear. And even if it is, then the bounds are tied to a certain hardware configuration. As a remedy, this thesis demonstrates a whole another approach, which is based on relative performance comparison. Using a logic built on top of a research published by the issuing department, chosen application units are compared together in a manner that makes results of such tests more reliable even to a change of hardware configuration. The presented theory is also implemented and verified on selected use cases.

Capturing Performance Assumptions using Stochastic Performance Logic

Compared to functional unit testing, automated performance testing is difficult, partially because correctness criteria are more difficult to express for performance than for functionality. Where existing approaches rely on absolute bounds of the execution time, we aim to express assertions on code performance in relative, hardware-independent terms. To this end, we introduce Stochastic Performance Logic (SPL), which allows making statements about relative method performance. Since SPL interpretation is based on statistical tests applied to performance measurements, it allows (for a special class of formulas) calculating the minimum probability at which a particular SPL formula holds. We prove basic properties of the logic and present an algorithm for SAT-solver-guided evaluation of SPL formulas, which allows optimizing the number of performance measurements that need to be made. Finally, we propose integration of SPL formulas with Java code using higher-level performance annotations, for performance testing and documentation purposes