671 research outputs found

    A Survey of Constrained Combinatorial Testing

    Get PDF
    Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques

    Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search

    Get PDF
    The surge of search based software engineering research has been hampered by the need to develop customized search algorithms for different classes of the same problem. For instance, two decades of bespoke Combinatorial Interaction Testing (CIT) algorithm development, our exemplar problem, has left software engineers with a bewildering choice of CIT techniques, each specialized for a particular task. This paper proposes the use of a single hyperheuristic algorithm that learns search strategies across a broad range of problem instances, providing a single generalist approach. We have developed a Hyperheuristic algorithm for CIT, and report experiments that show that our algorithm competes with known best solutions across constrained and unconstrained problems: For all 26 real-world subjects, it equals or outperforms the best result previously reported in the literature. We also present evidence that our algorithm's strong generic performance results from its unsupervised learning. Hyperheuristic search is thus a promising way to relocate CIT design intelligence from human to machine

    Optimization Techniques for Automated Software Test Data Generation

    Get PDF
    Esta tesis propone una variedad de contribuciones al campo de pruebas evolutivas. Hemos abarcados un amplio rango de aspectos relativos a las pruebas de programas: código fuente procedimental y orientado a objetos, paradigmas estructural y funcional, problemas mono-objetivo y multi-objetivo, casos de prueba aislados y secuencias de pruebas, y trabajos teóricos y experimentales. En relación a los análisis llevados a cabo, hemos puesto énfasis en el análisis estadístico de los resultados para evaluar la significancia práctica de los resultados. En resumen, las principales contribuciones de la tesis son: Definición de una nueva medida de distancia para el operador instanceof en programas orientados a objetos: En este trabajo nos hemos centrado en un aspecto relacionado con el software orientado a objetos, la herencia, para proponer algunos enfoques que pueden ayudar a guiar la búsqueda de datos de prueba en el contexto de las pruebas evolutivas. En particular, hemos propuesto una medida de distancia para computar la distancia de ramas en presencia del operador instanceof en programas Java. También hemos propuesto dos operadores de mutación que modifican las soluciones candidatas basadas en la medida de distancia definida. Definición de una nueva medida de complejidad llamada ``Branch Coverage Expectation'': En este trabajo nos enfrentamos a la complejidad de pruebas desde un punto de vista original: un programa es más complejo si es más difícil de probar de forma automática. Consecuentemente, definimos la ``Branch Coverage Expectation'' para proporcionar conocimiento sobre la dificultad de probar programas. La fundación de esta medida se basa en el modelo de Markov del programa. El modelo de Markov proporciona fundamentos teóricos. El análisis de esta medida indica que está más correlacionada con la cobertura de rama que las otras medidas de código estáticas. Esto significa que esto es un buen modo de estimar la dificultad de probar un programa. Predicción teórica del número de casos de prueba necesarios para cubrir un porcentaje concreto de un programa: Nuestro modelo de Markov del programa puede ser usado para proporcionar una estimación del número de casos de prueba necesarios para cubrir un porcentaje concreto del programa. Hemos comparado nuestra predicción teórica con la media de las ejecuciones reales de un generador de datos de prueba. Este modelo puede ayudar a predecir la evolución de la fase de pruebas, la cual consecuentemente puede ahorrar tiempo y coste del proyecto completo. Esta predicción teórica podría ser también muy útil para determinar el porcentaje del programa cubierto dados un número de casos de prueba. Propuesta de enfoques para resolver el problema de generación de datos de prueba multi-objetivo: En ese capítulo estudiamos el problema de la generación multi-objetivo con el fin de analizar el rendimiento de un enfoque directo multi-objetivo frente a la aplicación de un algoritmo mono-objetivo seguido de una selección de casos de prueba. Hemos evaluado cuatro algoritmos multi-objetivo (MOCell, NSGA-II, SPEA2, y PAES) y dos algoritmos mono-objetivo (GA y ES), y dos algoritmos aleatorios. En términos de convergencia hacía el frente de Pareto óptimo, GA y MOCell han sido los mejores resolutores en nuestra comparación. Queremos destacar que el enfoque mono-objetivo, donde se ataca cada rama por separado, es más efectivo cuando el programa tiene un grado de anidamiento alto. Comparativa de diferentes estrategias de priorización en líneas de productos y árboles de clasificación: En el contexto de pruebas funcionales hemos tratado el tema de la priorización de casos de prueba con dos representaciones diferentes, modelos de características que representan líneas de productos software y árboles de clasificación. Hemos comparado cinco enfoques relativos al método de clasificación con árboles y dos relativos a líneas de productos, cuatro de ellos propuestos por nosotros. Los resultados nos indican que las propuestas para ambas representaciones basadas en un algoritmo genético son mejores que el resto en la mayoría de escenarios experimentales, es la mejor opción cuando tenemos restricciones de tiempo o coste. Definición de la extensión del método de clasificación con árbol para la generación de secuencias de pruebas: Hemos definido formalmente esta extensión para la generación de secuencias de pruebas que puede ser útil para la industria y para la comunidad investigadora. Sus beneficios son claros ya que indudablemente el coste de situar el artefacto bajo pruebas en el siguiente estado no es necesario, a la vez que reducimos significativamente el tamaño de la secuencia utilizando técnicas metaheurísticas. Particularmente nuestra propuesta basada en colonias de hormigas es el mejor algoritmo de la comparativa, siendo el único algoritmo que alcanza la cobertura máxima para todos los modelos y tipos de cobertura. Exploración del efecto de diferentes estrategias de seeding en el cálculo de frentes de Pareto óptimos en líneas de productos: Estudiamos el comportamiento de algoritmos clásicos multi-objetivo evolutivos aplicados a las pruebas por pares de líneas de productos. El grupo de algoritmos fue seleccionado para cubrir una amplia y diversa gama de técnicas. Nuestra evaluación indica claramente que las estrategias de seeding ayudan al proceso de búsqueda de forma determinante. Cuanta más información se disponga para crear esta población inicial, mejores serán los resultados obtenidos. Además, gracias al uso de técnicas multi-objetivo podemos proporcionar un conjunto de pruebas adecuado mayor o menor, en resumen, que mejor se adapte a sus restricciones económicas o tecnológicas. Propuesta de técnica exacta para la computación del frente de Pareto óptimo en líneas de productos software: Hemos propuesto un enfoque exacto para este cálculo en el caso multi-objetivo con cobertura paiwise. Definimos un programa lineal 0-1 y un algoritmo basado en resolutores SAT para obtener el frente de Pareto verdadero. La evaluación de los resultados nos indica que, a pesar de ser un fantástico método para el cálculo de soluciones óptimas, tiene el inconveniente de la escalabilidad, ya que para modelos grandes el tiempo de ejecución sube considerablemente. Tras realizar un estudio de correlaciones, confirmamos nuestras sospechas, existe una alta correlación entre el tiempo de ejecución y el número de productos denotado por el modelo de características del programa

    An orchestrated survey of available algorithms and tools for Combinatorial Testing

    Get PDF
    For functional testing based on the input domain of a functionality, parameters and their values are identified and a test suite is generated using a criterion exercising combinations of those parameters and values. Since software systems are large, resulting in large numbers of parameters and values, a technique based on combinatorics called Combinatorial Testing (CT) is used to automate the process of creating those combinations. CT is typically performed with the help of combinatorial objects called Covering Arrays. The goal of the present work is to determine available algorithms/tools for generating a combinatorial test suite. We tried to be as complete as possible by using a precise protocol for selecting papers describing those algorithms/tools. The 75 algorithms/tools we identified are then categorized on the basis of different comparison criteria, including: the test suite generation technique, the support for selection (combination) criteria, mixed covering array, the strength of coverage, and the support for constraints between parameters. Results can be of interest to researchers or software companies who are looking for a CT algorithm/tool suitable for their needs

    An empirical comparison of fixed-strength and mixed-strength for interaction coverage based prioritization

    Get PDF
    Test case prioritization (TCP) plays an important role in identifying, characterizing, diagnosing and correcting faults quickly. TCP has been widely used to order test cases of different types, including model inputs (also called abstract test cases). Model inputs are constructed by modeling the program according to its input parameters, values, and constraints, and has been used in different testing methods, such as combinatorial interaction testing, and software product line testing. Interaction coveragebased test case prioritization (ICTCP) uses interaction coverage information derived from the model input to order inputs. Previous studies have focused generally on the fixed-strength ICTCP, which adopts a fixed strength(i.e.,thelevelofparameterinteractions)tosupporttheICTCPprocess.Itisgenerallyacceptedthat using more strengths for ICTCP, i.e., mixed-strength ICTCP, may give better ordering than fixed-strength. To confirm whether mixed-strength is better than fixed-strength, in this paper we report on an extensive empirical study using five real-world programs (written in C), each of which has six versions. The results oftheempiricalstudiesshowthatmixed-strengthhasbetterratesofinteractioncoverageoverallthanfixedstrength, but they have very similar rates of fault detection. Our results also show that fixed-strength should be used instead of the mixed-strength at the later stage of software testing. Finally, we offer some practical guidelinesfortesterswhenusinginteractioncoverageinformationtoprioritizemodelinputs,underdifferent testing scenarios and resources

    Test generation for high coverage with abstraction refinement and coarsening (ARC)

    Get PDF
    Testing is the main approach used in the software industry to expose failures. Producing thorough test suites is an expensive and error prone task that can greatly benefit from automation. Two challenging problems in test automation are generating test input and evaluating the adequacy of test suites: the first amounts to producing a set of test cases that accurately represent the software behavior, the second requires defining appropriate metrics to evaluate the thoroughness of the testing activities. Structural testing addresses these problems by measuring the amount of code elements that are executed by a test suite. The code elements that are not covered by any execution are natural candidates for generating further test cases, and the measured coverage rate can be used to estimate the thoroughness of the test suite. Several empirical studies show that test suites achieving high coverage rates exhibit a high failure detection ability. However, producing highly covering test suites automatically is hard as certain code elements are executed only under complex conditions while other might be not reachable at all. In this thesis we propose Abstraction Refinement and Coarsening (ARC), a goal oriented technique that combines static and dynamic software analysis to automatically generate test suites with high code coverage. At the core of our approach there is an abstract program model that enables the synergistic application of the different analysis components. In ARC we integrate Dynamic Symbolic Execution (DSE) and abstraction refinement to precisely direct test generation towards the coverage goals and detect infeasible elements. ARC includes a novel coarsening algorithm for improved scalability. We implemented ARC-B, a prototype tool that analyses C programs and produces test suites that achieve high branch coverage. Our experiments show that the approach effectively exploits the synergy between symbolic testing and reachability analysis outperforming state of the art test generation approaches. We evaluated ARC-B on industry relevant software, and exposed previously unknown failures in a safety-critical software component

    Um método e uma ferramenta para testes baseados em modelos para linhas de produto software

    Get PDF
    Orientador: Eliane MartinsDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: As linhas de produtos de software (LPS) estão ganhando interesse devido à crescente demanda por produtos personalizáveis. Tal se deve, em parte, por que as LPS são um meio eficiente e efetivo de entregar produtos com maior qualidade a um custo menor. Em uma LPS, produtos têm requisitos em comum e também, características específicas a cada um. Testar se um produto implementa os requisitos comuns e específicos é um importante passo para garantir uma boa qualidade. No entanto, o teste de uma LPS é uma tarefa complexa, uma vez que a variedade de produtos que podem ser derivados a partir da combinação de características comuns e específicas é enorme. Mesmo que se escolha apenas alguns produtos selecionados, o esforço para testá-los ainda assim é grande, dado que os produtos variam em termos das características específicas selecionadas. Portanto, reutilizar casos de teste de um produto para o outro para determinar se satisfazem os requisitos funcionais, pode não ser possível. Os testes baseados em modelos (MBT) podem ser úteis neste caso, nos quais um modelo de comportamento pode ser obtido a partir dos requisitos e este modelo pode ser usado para a geração automática de casos de teste. Neste trabalho é apresentada uma abordagem em que os requisitos SPL são centrados em casos de uso. Casos de uso (UC) são um formato popular para representar os requisitos. A partir das descrições de casos de uso escritas em um formato semi-estruturado e contendo a especificação de variabilidade, os modelos de comportamento são gerados automaticamente para um produto sob teste, na forma de um modelo de máquina de estado. Construir uma máquina de estado não é trivial para a maioria dos profissionais, que estão mais habituados com descrições textuais e informais dos requisitos. Em geral, a criação manual de modelos de máquinas de estado a partir de UCs pode ser demorado e propenso a erros. O objetivo é fornecer aos engenheiros de teste um método que os guie na criação dos artefatos necessários para que uma versão preliminar de um modelo de estado seja extraída automaticamente dos requisitos. Este modelo preliminar pode ser refinado para tornar-se adequado para uma ferramenta de geração de casos de teste. Para esse processo de refinamento também são fornecidas algumas diretrizes. Como prova de conceito, desenvolveu-se um protótipo de uma ferramenta, MARITACA, que utiliza técnicas de processamento de língua natural para extrair as máquinas de estado a partir das descrições dos casos de uso. O texto apresenta o uso do método e da ferramenta em um exemplo ilustrativo, obtido da literatura, e em uma família de aplicações distribuídas tolerantes a falhas. Este estudo mostrou a aplicabilidade do método proposto. Uma das preocupações nos testes de SPL é a geração de casos de teste redundantes de um produto para outro. Os resultados, embora preliminares, mostraram que a maioria dos casos de teste gerados para um novo produto não são redundantes, pois envolvem características específicas de cada produtoAbstract: Software product lines (SPL) are gaining interest because of the increasing demand for customizable products. This is partly because SPLs are an efficient and effective means of delivering products with higher quality at a lower cost. In SPL, products have common requirements and also, specific features for each one. Testing whether a product implements common and specific requirements is an important step to ensure good quality of the derived products. However, testing a SPL is a complex task, since the variety of products that can be derived from the combination of common and specific features is huge. Even if only a few specific products are selected, the effort to test them is still significant, since the products vary in terms of the specific features that are selected. Therefore, reusing test cases from one product to another to determine whether they satisfy the functional requirements may not be possible. Model-based testing (MBT) may be useful in this case, in which a behavior model can be obtained from the requirements and this model can be used for automatic test cases generation. This work presents model-based product testing approach (MBPTA) for software product lines, in which requirements are centered on use cases. Use Cases (UC) are a popular format for representing requirements. From the use case descriptions written in the form of a semi-structured format and containing the variability specification, the behavior models are automatically generated for a product under test, in the form of a state machine model. Building a state machine is not a trivial task for most practitioners, who are more familiarized with textual and informal descriptions of requirements. In general, the manual creation of state machine models from UCs can be time-consuming and prone to errors. The goal is to provide the test engineers with a method that guides them in the creation of artifacts necessary to extract a preliminary version of a state model from the requirements. This preliminary model can be refined to become suitable for a test case generation tool. MBPTA also provides guidelines for the refinement process of the preliminary model. As proof of concept, a prototype of a tool was developed, MARITACA, which uses natural language processing techniques to extract state machines from the use case descriptions. The text presents the use of the method and the tool in an illustrative example, obtained from the literature, and in a family of distributed fault-tolerant applications. This study showed the applicability of the proposed method. One of the concerns in SPL testing is the generation of redundant test cases from one product to another. The results, though preliminary, showed that most of the test cases generated for a new product are not redundant because they involve specific features of each productMestradoCiência da ComputaçãoMestra em Ciência da ComputaçãoCAPE

    Model based test suite minimization using metaheuristics

    Get PDF
    Software testing is one of the most widely used methods for quality assurance and fault detection purposes. However, it is one of the most expensive, tedious and time consuming activities in software development life cycle. Code-based and specification-based testing has been going on for almost four decades. Model-based testing (MBT) is a relatively new approach to software testing where the software models as opposed to other artifacts (i.e. source code) are used as primary source of test cases. Models are simplified representation of a software system and are cheaper to execute than the original or deployed system. The main objective of the research presented in this thesis is the development of a framework for improving the efficiency and effectiveness of test suites generated from UML models. It focuses on three activities: transformation of Activity Diagram (AD) model into Colored Petri Net (CPN) model, generation and evaluation of AD based test suite and optimization of AD based test suite. Unified Modeling Language (UML) is a de facto standard for software system analysis and design. UML models can be categorized into structural and behavioral models. AD is a behavioral type of UML model and since major revision in UML version 2.x it has a new Petri Nets like semantics. It has wide application scope including embedded, workflow and web-service systems. For this reason this thesis concentrates on AD models. Informal semantics of UML generally and AD specially is a major challenge in the development of UML based verification and validation tools. One solution to this challenge is transforming a UML model into an executable formal model. In the thesis, a three step transformation methodology is proposed for resolving ambiguities in an AD model and then transforming it into a CPN representation which is a well known formal language with extensive tool support. Test case generation is one of the most critical and labor intensive activities in testing processes. The flow oriented semantic of AD suits modeling both sequential and concurrent systems. The thesis presented a novel technique to generate test cases from AD using a stochastic algorithm. In order to determine if the generated test suite is adequate, two test suite adequacy analysis techniques based on structural coverage and mutation have been proposed. In terms of structural coverage, two separate coverage criteria are also proposed to evaluate the adequacy of the test suite from both perspectives, sequential and concurrent. Mutation analysis is a fault-based technique to determine if the test suite is adequate for detecting particular types of faults. Four categories of mutation operators are defined to seed specific faults into the mutant model. Another focus of thesis is to improve the test suite efficiency without compromising its effectiveness. One way of achieving this is identifying and removing the redundant test cases. It has been shown that the test suite minimization by removing redundant test cases is a combinatorial optimization problem. An evolutionary computation based test suite minimization technique is developed to address the test suite minimization problem and its performance is empirically compared with other well known heuristic algorithms. Additionally, statistical analysis is performed to characterize the fitness landscape of test suite minimization problems. The proposed test suite minimization solution is extended to include multi-objective minimization. As the redundancy is contextual, different criteria and their combination can significantly change the solution test suite. Therefore, the last part of the thesis describes an investigation into multi-objective test suite minimization and optimization algorithms. The proposed framework is demonstrated and evaluated using prototype tools and case study models. Empirical results have shown that the techniques developed within the framework are effective in model based test suite generation and optimizatio
    corecore