Tool support for statistical testing of software components

We describe the "STSC" prototype tool that supports the statistical testing of software components. The tool supports a wide range of operational profiles and test oracles for test case generation and output evaluation. The tool also generates appropriate values for different types of input parameters of operations. STSC automatically generates a test driver from an operational profile. This test driver invokes a test oracle that is implemented as a behaviour-checking version of the implementation. To evaluate the flexibility and usability of the tool, it has been applied to several case studies using different types of operational profiles and test oracles

A Passive Test Oracle Using a Component's API

A test oracle is a mechanism that is used during testing to determine whether a software component behaves correctly or not. The test oracle problem is widely acknowledged in the software testing literature and many methods for test oracle development have been proposed. Most of these methods use specifications or other resources to develop test oracles. A passive test oracle checks the behaviour of the component, but does not reproduce this behaviour. In this paper, we present a technique that develops passive test oracles for components using their APIs. This simple technique can be applied to any software component that is accessed through an API. In an initial experiment, we found that test oracles developed this way were more effective at finding faults with a relatively small number of test cases than test oracles developed from a formal specification and developed as a parallel implementation

Automatic Generation of Acceptance Test Cases from Use Case Specifications: an NLP-based Approach

Acceptance testing is a validation activity performed to ensure the conformance of software systems with respect to their functional requirements. In safety critical systems, it plays a crucial role since it is enforced by software standards, which mandate that each requirement be validated by such testing in a clearly traceable manner. Test engineers need to identify all the representative test execution scenarios from requirements, determine the runtime conditions that trigger these scenarios, and finally provide the input data that satisfy these conditions. Given that requirements specifications are typically large and often provided in natural language (e.g., use case specifications), the generation of acceptance test cases tends to be expensive and error-prone. In this paper, we present Use Case Modeling for System-level, Acceptance Tests Generation (UMTG), an approach that supports the generation of executable, system-level, acceptance test cases from requirements specifications in natural language, with the goal of reducing the manual effort required to generate test cases and ensuring requirements coverage. More specifically, UMTG automates the generation of acceptance test cases based on use case specifications and a domain model for the system under test, which are commonly produced in many development environments. Unlike existing approaches, it does not impose strong restrictions on the expressiveness of use case specifications. We rely on recent advances in natural language processing to automatically identify test scenarios and to generate formal constraints that capture conditions triggering the execution of the scenarios, thus enabling the generation of test data. In two industrial case studies, UMTG automatically and correctly translated 95% of the use case specification steps into formal constraints required for test data generation; furthermore, it generated test cases that exercise not only all the test scenarios manually implemented by experts, but also some critical scenarios not previously considered

Fostering design by contract by exploiting the relationship between code commentary and contracts.

Embora contratos no estilo de programação por contratos (DBC) tragam precisão para expressar o comportamento do código, desenvolvedores são resistentes ao seu uso. Há várias razões para isto, tais como a dificuldade na produção de contratos ou o trabalho de manter os contratos consistentes com o código em evolução. Por outro lado, Javadoc é uma abordagem comumente usada para documentar programas Java. Mesmo assim, comentários Javadoc não servem para a checagem automática de conformidade devido à ambiguidade inerente à linguagem natural. Neste trabalho, procuramos minimizar a distância entre contratos e Javadoc, estimulando a adoção de DBC a partir de duas contribuições principais; primeiro, propomos uma extensão ao sistema de tags do Javadoc (CONTRACTJ DOC) para possibilitar a integração de contratos na notação de comentários; então, propomos uma abordagem para geração de contratos a partir de comentários em linguagem natural (CONTRACT SUGGESTOR). Nós realizamos três avaliações: primeiro, avaliamos a aplicabilidade e a compreensibilidade de CONTRACTJ DOC. Como resultados, detectamos inconsistências entre a documentação Javadoc e o código fonte. A maioria dos contratos que escrevemos foram checagens de valores limítrofes para parâmetros e repetições de expressões de retorno de métodos. Além disso, a legibilidade dos comentários percebida pelos desenvolvedores não diferiu significativamente entre as abordagens, o que é promissor, dado que contratos são usualmente classificados como difíceis de ler. Segundo, avaliamos a qualidade dos contratos gerados por CONTRACT SUGGESTOR verificando a taxa de falsos positivos gerados. Como resultado, são gerados mais contratos corretos para non-null do que para relational, devido a quantidade de instâncias de comentários para cada propriedade. Por fim, realizamos estudos de caso com JMLOK2 e C ONTRACTOK – CONTRACTOK é uma extensão da abordagem de J MLOK2 para o contexto C#/Code Contracts. Primeiro, usamos JMLOK2 para verificar os contratos gerados automaticamente por CONTRACT SUGGESTOR; depois usamos as ferramentas para verificar 24 sistemas de código aberto (12 para cada ferramenta). As ferramentas detectaram 188 não-conformidades, sendo 72 problemas de pós-condição e 61 de invariante; as causas prováveis mais comuns foram Pré-condição fraca (91) e Erro de código (56). Com isso, objetivamos motivar a adoção de DBC como forma de aprimorar o projeto dos programas, e por consequência, sua qualidade geral.Contracts in Design by Contract style bring about preciseness for expressing the code behavior; however, developers are resistant to their use. There are several likely reasons for this, such as the trouble to conceive good, useful contracts, or the burden of maintaining contracts consistent with the evolving code. On the other hand, Javadoc is a common way of documenting Java programs. Nevertheless, Javadoc comments do not serve to an automated conformance checking due to ambiguity issues inherent to the natural languages. In this work, we try to minimize the distance between contracts and Javadoc, fostering DBC adoption by means of two main contributions; first, we propose an extension to the Javadoc tagging system (C ONTRACTJD OC) for allowing the integration of contracts into the comments notation; then, we propose an approach for automatically generating contracts based on natural language code commentary (CONTRACTSUGGESTOR). We perform three evaluations: first, we evaluate the applicability and comprehensibility of C ONTRACTJD OC. As results, we detected inconsistencies between the documentation available by means of Javadoc comments and the source code. The majority of the contracts we could write from the comments remains between common-case and repetitive with the code. Moreover, developers’ impression about the readability of comments did not differ significantly, which is promising, as contracts are usually regarded as hard to read – one reason for its non-adoption. Then, we evaluate the quality of contracts generated by CONTRACTSUGGESTOR by analyzing the false positives rate. As result, the approach generates more correct contracts for non-null than for relational, due to the number of comment instances for each property. Finally, we perform case studies with J MLOK2 and C ONTRACTOK – C ONTRACTOK is an extension of JMLOK2 for C#/Code Contracts context. First, we used J MLOK2 for conformance checking the contracts automatically generated; then, we run the tools over 24 open-source systems (12 with each tool). The tools detected 188 nonconformances. From those, 72 are postcondition and 61 are invariant problems; with respect to likely causes manually established, Weak precondition (91) and Code error (56) are the most commons. With this, we aim to promote DBC adoption as a way for improving the design of the projects, and consequently, their quality in general.Cape

Using Metamorphic Testing at Runtime to Detect Defects in Applications without Test Oracles

First, we will present an approach called Automated Metamorphic System Testing. This will involve automating system-level metamorphic testing by treating the application as a black box and checking that the metamorphic properties of the entire application hold after execution. This will allow for metamorphic testing to be conducted in the production environment without affecting the user, and will not require the tester to have access to the source code. The tests do not require an oracle upon their creation; rather, the metamorphic properties act as built-in test oracles. We will also introduce an implementation framework called Amsterdam. Second, we will present a new type of testing called Metamorphic Runtime Checking. This involves the execution of metamorphic tests from within the application, i.e., the application launches its own tests, within its current context. The tests execute within the application's current state, and in particular check a function's metamorphic properties. We will also present a system called Columbus that supports the execution of the Metamorphic Runtime Checking from within the context of the running application. Like Amsterdam, it will conduct the tests with acceptable performance overhead, and will ensure that the execution of the tests does not affect the state of the original application process from the users' perspective; however, the implementation of Columbus will be more challenging in that it will require more sophisticated mechanisms for conducting the tests without pre-empting the rest of the application, and for comparing the results which may conceivably be in different processes or environments. Third, we will describe a set of metamorphic testing guidelines that can be followed to assist in the formulation and specification of metamorphic properties that can be used with the above approaches. These will categorize the different types of properties exhibited by many applications in the domain of machine learning and data mining in particular (as a result of the types of applications we will investigate), but we will demonstrate that they are also generalizable to other domains as well. This set of guidelines will also correlate to the different types of defects that we expect the approaches will be able to find