Automatic testing of software with structurally complex inputs

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.Includes bibliographical references (p. 123-132).Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties of structurally complex test inputs; presents a technique that automates generation of such inputs; describes the Korat tool that implements this technique for Java; and evaluates the effectiveness of Korat in testing a set of data-structure implementations. Our approach allows the developer to describe the properties of valid test inputs using a familiar implementation language such as Java. Specifically, the user provides an imperative predicate--a piece of code that returns a truth value--that returns true if the input satisfies the required property and false otherwise. Korat implements our technique for solving imperative predicates: given a predicate and a bound on the size of the predicate's inputs, Korat automatically generates the bounded-exhaustive test suite that consists of all inputs, within the given bound, that satisfy the property identified by the predicate. To generate these inputs, Korat systematically searches the bounded input space by executing the predicate on the candidate inputs. Korat does this efficiently by pruning the search based on the predicate's executions and by generating only nonisomorphic inputs. Bounded-exhaustive testing is a methodology for testing the code on all inputs within the given small bound.(cont.) Our experiments on a set of ten linked and array- based data structures show that Korat can efficiently generate bounded-exhaustive test suites from imperative predicates even for very large input spaces. Further, these test suites can achieve high statement, branch, and mutation coverage. The use of our technique for generating structurally complex test inputs also enabled testers in industry to detect faults in real, production-quality applications.by Darko Marinov.Ph.D

Criterios de cobertura sobre RepOK para reducir test suites exhaustivas acotadas: estudio de casos

La generación exhaustiva acotada de casos de test es una técnica de generación de entradas para programas que consiste en construir todas las posibles entradas válidas hasta cierta cota dada. Aunque el tamaño de la test suite generada está ligado a la cota provista, inclusive para cotas muy pequeñas las test suites obtenidas resultan considerablemente grandes, haciendo al uso de las mismas algo prohibitivo. En este trabajo revisaremos, adicionando nuevos casos de estudios, un enfoque para reducir el tamaño de las test suites generadas exhaustivamente presentado previamente. Este enfoque está basado en el uso de criterios de cobertura de código sobre el invariante de representación de la estructura sobre la cual la test suite es producida. La implementación de este invariante es utilizada para decidir cuándo dos entradas válidas pueden ser consideradas equivalentes, lo cual sucede si éstas ejercitan el código del invariante de representación de manera similar de acuerdo algún criterio de cobertura de código de caja blanca. Esta relación de equivalencia entre las entradas válidas es aprovechada para descartar casos de test que cubren clases de equivalencias ya cubiertas por algún otro caso de test presente en la suite. En este trabajo, se adicionan nuevos casos de estudios que muestran que, a medida que las cotas crecen, reducir las test suites exhaustivas aplicando la técnica presentada, produce resultados similares a las test suites exhaustivas, en cuanto a habilidad para matar mutantes.Eje: Workshop Ingeniería de Software (WIS).Red de Universidades con Carreras en Informática (RedUNCI

Mejoras al testing exhaustivo acotado

Tesis (Doctor en Cs. de la Computación)--Universidad Nacional de Córdoba, Facultad de Matemática, Astronomía y Física, 2015.El Testing consiste en ejecutar una pieza de software con diferentes entradas para luego chequear si el resultado obtenido se corresponde con el resultado esperado. Se estima que esta actividad ocupa más de la mitad del costo total del desarrollo de software, por lo que es fundamental su automatización. La construcción automática de entradas para probar el software bajo análisis reduce significativamente los costos de hacer testing. El testing exhaustivo acotado es una técnica muy efectiva para encontrar errores, que consiste en generar automáticamente todas las entradas válidas dentro de ciertas cotas, y posteriormente usar estas entradas para probar el código bajo análisis. La efectividad de los conjuntos de entradas producidos, crece a medida que las cotas crecen, pero además, el tiempo consumido durante la generación y posterior ejecución del programa bajo las entradas producidas, se incrementa de manera exponencial cuando las cotas crecen, y como consecuencia, muchas veces, esta tarea resulta inviable. En este trabajo se presentan un grupo de técnicas destinadas a disminuir el tiempo empleado para realizar testing exhaustivo acotado, para lo cual, se propone reducir el tiempo de generación de entradas, así como también el tiempo de ejecución de las mismas mediante la reducción `adecuada' del tamaño de los conjuntos producidos

UNIT-LEVEL ISOLATION AND TESTING OF BUGGY CODE

In real-world software development, maintenance plays a major role and developers spend 50-80% of their time in maintenance-related activities. During software maintenance, a significant amount of effort is spent on ending and fixing bugs. In some cases, the fix does not completely eliminate the buggy behavior; though it addresses the reported problem, it fails to account for conditions that could lead to similar failures. There could be many possible reasons: the conditions may have been overlooked or difficult to reproduce, e.g., when the components that invoke the code or the underlying components it interacts with can not put it in a state where latent errors appear. We posit that such latent errors can be discovered sooner if the buggy section can be tested more thoroughly in a separate environment, a strategy that is loosely analogous to the medical procedure of performing a biopsy where tissue is removed, examined and subjected to a battery of tests to determine the presence of a disease. In this thesis, we propose a process in which the buggy code is extracted and isolated in a test framework. Test drivers and stubs are added to exercise the code and observe its interactions with its dependencies. We lay the groundwork for the creation of an automated tool for isolating code by studying its feasibility and investigating existing testing technologies that can facilitate the creation of such drivers and stubs. We investigate mocking frameworks, symbolic execution and model checking tools and test their capabilities by examining real bugs from the Apache Tomcat project. We demonstrate the merits of performing unit-level symbolic execution and model checking to discover runtime exceptions and logical errors. The process is shown to have high coverage and able to uncover latent errors due to insufficient fixes

Systematic techniques for more effective fault localization and program repair

Debugging faulty code is a tedious process that is often quite expensive and can require much manual effort. Developers typically perform debugging in two key steps: (1) fault localization, i.e., identifying the location of faulty line(s) of code; and (2) program repair, i.e., modifying the code to remove the fault(s). Automating debugging to reduce its cost has been the focus of a number of research projects during the last decade, which have introduced a variety of techniques. However, existing techniques suffer from two basic limitations. One, they lack accuracy to handle real programs. Two, they focus on automating only one of the two key steps, thereby leaving the other key step to the developer. Our thesis is that an approach that integrates systematic search based on state-of-the-art constraint solvers with techniques to analyze artifacts that describe application specific properties and behaviors, provides the basis for developing more effective debugging techniques. We focus on faults in programs that operate on structurally complex inputs, such as heap-allocated data or relational databases. Our approach lays the foundation for a unified framework for localization and repair of faults in programs. We embody our thesis in a suite of integrated techniques based on propositional satisfiability solving, correctness specifications analysis, test-spectra analysis, and rule-learning algorithms from machine learning, implement them as a prototype tool-set, and evaluate them using several subject programs.Electrical and Computer Engineerin

Invariant detection meets Random test case generation

Dissertação de mestrado em Engenharia de InformáticaFull fledged verification of software ensures correction to a level that no other technique can reach. However it requires precise and unambiguous specifications of requirements, functionality and technical aspects of the software to be verified. Furthermore, it requires that these specifications together with the produced models and code be checked for conformity. This represents beyond doubt an investment that most developers and companies are neither able nor willing to make. Although testing can not reach the same level of assurance as full fledged verification, it is the most widely accepted and used technique to validate expectations about software products. Testing is the most natural way of checking that a piece of software is doing what the developers expect it to do. Improvements to test case generation have the potential to produce a great impact in the state of the art of software engineering, by putting Software Testing closer to Formal Software Verification. This is an exploratory project, aimed at surveying the current state of the art in the field of test case generation and related techniques for the Java language, eventually suggesting possible advancements in the field.A verdadeira verificação de software garante correcção de software a um nível que nenhuma outra técnica consegue igualar. No entanto, exige especificações precisas e inequívocas de requisitos de funcionalidade e aspectos técnicos do software a ser verificado. Além disso, é necessário que as especificações, juntamente com os modelos produzidos e código sejam verificados quanto à sua conformidade. Isto representa indubitavelmente um investimento que a maioria dos profissionais e empresas não são capazes, nem estão dispostos a fazer. Embora os testes não alcancem o mesmo nível de garantia como a verificação completa, é a técnica mais amplamente aceite e usada para validar as especificações sobre produtos de software. O teste é a forma mais natural de verificar que um pedaço de software cumpre o que os programadores esperam que faça. Melhorias na geração de boletins de teste têm o potencial de produzir um grande impacto no estado da arte da engenharia de software, colocando o teste de software mais perto da Verificação Formal de Software. Este projecto é de carácter exploratório, visando o levantamento do estado actual da área de geração de casos de teste para a linguagem Java e técnicas relacionadas, sugerindo avanços possíveis nesta área de validação de softwar

Fundamental Approaches to Software Engineering

This open access book constitutes the proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering, FASE 2021, which took place during March 27–April 1, 2021, and was held as part of the Joint Conferences on Theory and Practice of Software, ETAPS 2021. The conference was planned to take place in Luxembourg but changed to an online format due to the COVID-19 pandemic. The 16 full papers presented in this volume were carefully reviewed and selected from 52 submissions. The book also contains 4 Test-Comp contributions

Tagungsband zum 21. Kolloquium Programmiersprachen und Grundlagen der Programmierung

Das 21. Kolloquium Programmiersprachen und Grundlagen der Programmierung (KPS 2021) setzt eine traditionelle Reihe von Arbeitstagungen fort, die 1980 von den Forschungsgruppen der Professoren Friedrich L. Bauer (TU München), Klaus Indermark (RWTH Aachen) und Hans Langmaack(CAU Kiel) ins Leben gerufen wurde.Die Veranstaltung ist ein offenes Forum für alle interessierten deutschsprachigen Wissenschaftlerinnen und Wissenschaftler zum zwanglosen Austausch neuer Ideen und Ergebnisse aus den Forschungsbereichen Entwurf und Implementierung von Programmiersprachen sowie Grundlagen und Methodik des Programmierens. Dieser Tagungsband enthält die wissenschaftlichen Beiträge,die bei dem 21. Kolloquium dieser Tagungsreihe präsentiert wurden, welches vom 27. bis 29. September 2021 in Kiel stattfand und von der Arbeitsgruppe Programmiersprachen und Übersetzerkonstruktion der Christian-Albrechts-Universität zu Kiel organisiert wurde

Weighted Statistical Testing based on Active Learning and Formal Verification Techniques for Software Reliability Assessment

This work developed an automatic approach for the assessment of software reliability which is both theoretical sound and practical. The developed approach extends and combines theoretical sound approaches in a novel manner to systematically reduce the overhead of reliability assessment

Directed random testing

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 155-162).Random testing can quickly generate many tests, is easy to implement, scales to large software applications, and reveals software errors. But it tends to generate many tests that are illegal or that exercise the same parts of the code as other tests, thus limiting its effectiveness. Directed random testing is a new approach to test generation that overcomes these limitations, by combining a bottom-up generation of tests with runtime guidance. A directed random test generator takes a collection of operations under test and generates new tests incrementally, by randomly selecting operations to apply and finding arguments from among previously-constructed tests. As soon as it generates a new test, the generator executes it, and the result determines whether the test is redundant, illegal, error-revealing, or useful for generating more tests. The technique outputs failing tests pointing to potential errors that should be corrected, and passing tests that can be used for regression testing. The thesis also contributes auxiliary techniques that post-process the generated tests, including a simplification technique that transforms a, failing test into a smaller one that better isolates the cause of failure, and a branch-directed test generation technique that aims to increase the code coverage achieved by the set of generated tests. Applied to 14 widely-used libraries (including the Java JDK and the core .NET framework libraries), directed random testing quickly reveals many serious, previously unknown errors in the libraries. And compared with other test generation tools (model checking, symbolic execution, and traditional random testing), it reveals more errors and achieves higher code coverage.(cont.) In an industrial case study, a test team at Microsoft using the technique discovered in fifteen hours of human effort as many errors as they typically discover in a person-year of effort using other testing methods.by Carlos Pacheco.Ph.D