Selection of heterogeneous test environments for the execution of automated tests

As software complexity grows so does the size of automated test suites that enable us to validate the expected behavior of the system under test. When that occurs, problems emerge for developers in the form of increased effort to manage the test process and longer execution time of test suites. Manual managing automated tests is especially problematic, as the recurring costa of guaranteeing that the automated tests (e.g.: thousands) are correctly configured to execute on the available test environments (e.g.: dozens or hundreds), on a regular basis and during the products lifetime may become huge, with unbearable human effort involved. This problem increases substantially when the system under test is one highly configurable product, requiring to be validated in heterogeneous environments, especially when these target test environments also evolve frequently (e.g.: new operating systems, new browsers, new mobile devices, ...). Being an integral part of software development, testing needs to evolve and break free from the conventional methods. This dissertation presents a technique that extends one existent algorithm to reduce the number of test executions, and extend it, enabling to perform the test case distribution over multiples heterogeneous test environments. The development, implementation and validation of the technique presented in this dissertation were conducted in the industrial context of an international software house. Real development scenarios were used to conduct experiments and validations, and the results demonstrated that the proposed technique is effective in terms of eliminating the human effort involved in test distribution.À medida que a complexidade do software aumenta o mesmo acontece com a dimensão das suites de testes automizados que permitem validar o comportamento esperado do sistema que está a ser testado. Quando isso ocorre, aparecem problemas para os programadores sob a forma de aumento de esforço necessário para gerir o processo de teste e maior tempo de execução das suites de teste. Gerir manualmente milhares de testes automatizados é especialmente problemático uma vez que os custos recorrentes de garantir que os testes automatizados (ex: milhares) estão corretamente configurados para executar nos ambientes de testes disponíveis (ex: dezenas ou centennas), durante o tempo de vida dos produtos pode tornar-se gigantesco. Este problema aumenta substancialmente quando o sistema que está a ser testado é um produto altamente configurável, precisando de ser validado em ambientes heterogéneos, especialmente quando também estes ambientes destino de testes também evoluem frequentemente (ex: novos sistemas operativos, novos browsers, novos devices móveis, ...). O tempo de execução destas suites de testes torna-se também um problema enorme, dado que não é viável executar todos as suites de testes em todas as configurações possiveis. Sendo uma parte integral do desenvolvimento de software, a forma de testar precisa de evoluir e libertar-se dos métodos convencionais. Esta dissertação apresenta uma técnica que estende um algoritmo existente que permite reduzir o número de execuções de testes, e estende-o, permitindo fazer a distribuição de casos de teste sobre múltiplos ambientes de teste heterogéneos. O desenvolvimento, implementação e validação da técnica proposta na presente dissertação foram conduzidos no contexto industrial de uma empresa internacional de desenvolvimento de software. Foram utilizados cenários de desenvolvimento de software reais para conduzir experiências e validações, e os resultados demonstraram que a técnica proposta é eficaz em termos de eliminar o esforo humano envolvido na distribuição de testes

An investigation of distributed computing for combinatorial testing

Combinatorial test generation, also calledt-way testing, is the process ofgenerating sets of input parameters for a system under test, by consideringinteractions between values of multiple parameters. In order to decrease totaltesting time, there is an interest in techniques that generate smaller test suites.In our previous work, we used graph techniques to produce high-quality testsuites. However, these techniques require a lot of computing power andmemory, which is why this paper investigates distributed computing fort-waytesting. We first introduce our distributed graph colouring method, with newalgorithms for building the graph and for colouring it. Second, we present ourdistributed hypergraph vertex covering method and a new heuristic. Third, weshow how to build a distributed IPOG algorithm by leveraging either graphcolouring or hypergraph vertex covering as vertical growth algorithms.Finally, we test these new methods on a computer cluster and compare themto existingt-way testing tools

Artificial table testing dynamically adaptive systems

Dynamically Adaptive Systems (DAS) are systems that modify their behavior and structure in response to changes in their surrounding environment. Critical mission systems increasingly incorporate adaptation and response to the environment; examples include disaster relief and space exploration systems. These systems can be decomposed in two parts: the adaptation policy that specifies how the system must react according to the environmental changes and the set of possible variants to reconfigure the system. A major challenge for testing these systems is the combinatorial explosions of variants and envi-ronment conditions to which the system must react. In this paper we focus on testing the adaption policy and propose a strategy for the selection of envi-ronmental variations that can reveal faults in the policy. Artificial Shaking Table Testing (ASTT) is a strategy inspired by shaking table testing (STT), a technique widely used in civil engineering to evaluate building's structural re-sistance to seismic events. ASTT makes use of artificial earthquakes that simu-late violent changes in the environmental conditions and stresses the system adaptation capability. We model the generation of artificial earthquakes as a search problem in which the goal is to optimize different types of envi-ronmental variations

Many-objective test suite generation for software product lines

A Software Product Line (SPL) is a set of products built from a number of features, the set of valid products being defined by a feature model. Typically, it does not make sense to test all products defined by an SPL and one instead chooses a set of products to test (test selection) and, ideally, derives a good order in which to test them (test prioritisation). Since one cannot know in advance which products will reveal faults, test selection and prioritisation are normally based on objective functions that are known to relate to likely effectiveness or cost. This paper introduces a new technique, the grid-based evolution strategy (GrES), which considers several objective functions that assess a selection or prioritisation and aims to optimise on all of these. The problem is thus a many-objective optimisation problem. We use a new approach, in which all of the objective functions are considered but one (pairwise coverage) is seen as the most important. We also derive a novel evolution strategy based on domain knowledge. The results of the evaluation, on randomly generated and realistic feature models, were promising, with GrES outperforming previously proposed techniques and a range of many-objective optimisation algorithms

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing