VERDICTS: Visual Exploratory Requirements Discovery and Injection for Comprehension and Testing of Software

We introduce a methodology and research tools for visual exploratory software analysis. VERDICTS combines exploratory testing, tracing, visualization, dynamic discovery and injection of requirements specifications into a live quick-feedback cycle, without recompilation or restart of the system under test. This supports discovery and verification of software dynamic behavior, software comprehension, testing, and locating the defect origin. At its core, VERDICTS allows dynamic evolution and testing of hypotheses about requirements and behavior, by using contracts as automated component verifiers. We introduce Semantic Mutation Testing as an approach to evaluate concordance of automated verifiers and the functional specifications they represent with respect to existing implementation. Mutation testing has promise, but also has many known issues. In our tests, both black-box and white-box variants of our Semantic Mutation Testing approach performed better than traditional mutation testing as a measure of quality of automated verifiers

Classifying the Correctness of Generated White-Box Tests: An Exploratory Study

White-box test generator tools rely only on the code under test to select test inputs, and capture the implementation's output as assertions. If there is a fault in the implementation, it could get encoded in the generated tests. Tool evaluations usually measure fault-detection capability using the number of such fault-encoding tests. However, these faults are only detected, if the developer can recognize that the encoded behavior is faulty. We designed an exploratory study to investigate how developers perform in classifying generated white-box test as faulty or correct. We carried out the study in a laboratory setting with 54 graduate students. The tests were generated for two open-source projects with the help of the IntelliTest tool. The performance of the participants were analyzed using binary classification metrics and by coding their observed activities. The results showed that participants incorrectly classified a large number of both fault-encoding and correct tests (with median misclassification rate 33% and 25% respectively). Thus the real fault-detection capability of test generators could be much lower than typically reported, and we suggest to take this human factor into account when evaluating generated white-box tests.Comment: 13 pages, 7 figure

Engineering simulations for cancer systems biology

Computer simulation can be used to inform in vivo and in vitro experimentation, enabling rapid, low-cost hypothesis generation and directing experimental design in order to test those hypotheses. In this way, in silico models become a scientific instrument for investigation, and so should be developed to high standards, be carefully calibrated and their findings presented in such that they may be reproduced. Here, we outline a framework that supports developing simulations as scientific instruments, and we select cancer systems biology as an exemplar domain, with a particular focus on cellular signalling models. We consider the challenges of lack of data, incomplete knowledge and modelling in the context of a rapidly changing knowledge base. Our framework comprises a process to clearly separate scientific and engineering concerns in model and simulation development, and an argumentation approach to documenting models for rigorous way of recording assumptions and knowledge gaps. We propose interactive, dynamic visualisation tools to enable the biological community to interact with cellular signalling models directly for experimental design. There is a mismatch in scale between these cellular models and tissue structures that are affected by tumours, and bridging this gap requires substantial computational resource. We present concurrent programming as a technology to link scales without losing important details through model simplification. We discuss the value of combining this technology, interactive visualisation, argumentation and model separation to support development of multi-scale models that represent biologically plausible cells arranged in biologically plausible structures that model cell behaviour, interactions and response to therapeutic interventions

Vulnerabilities preservation using code mutation

Dissertação de mestrado integrado em Informatics EngineeringThe main goal of software security testing is to assess the security risks of an application so that programmers can eliminate all vulnerabilities, as early as possible, before they are exploited by attackers. There are several tools on the market that allow to perform these tests during the software development life cycle to ensure that there are no security flaws in the final product. However, like all tools, these can also have imperfections, one of them being unable to detect weaknesses in vulnerable software. The project of this dissertation aims to tackle this problem, so that it is possible to find and correct flaws in security tests in order to, consequently, increase the effectiveness of the tools that intend to certify the security of applications. For this, the solution studied in this document is to apply syntactic transformations in vulnerable code samples without interfering in the presence of the vulnerabilities that should later be detected. This process is based on: ) code refactoring techniques that allow improving the internal quality of the software; ) the mutation testing system used to evaluate the quality of software testing. To implement this idea, a tool called VSG was developed with the functionality of producing new code samples with security flaws. This document describes the whole development process, from the architecture to the implementation of the tool. In the end, there is an analysis with the results obtained when trying to detect the vulnerabilities present in the samples produced through the CxSAST application of the company Checkmarx, from which this dissertation emerged.O objetivo principal de testes de segurança de software consiste em avaliar os riscos de segurança de uma aplicação para que os programadores possam eliminar todas as vulnerabilidades o mais cedo possível, antes que sejam exploradas por atacantes. Existem várias ferramentas no mercado que permitem realizar estes testes durante o processo de desenvolvimento de software para garantir que não existam falhas de segurança no produto final. Porém, tal como todas as ferramentas, estas também podem apresentar imperfeições, sendo uma delas não conseguir detetar fraquezas em software vulnerável. O projeto desta dissertação pretende combater este problema, de modo a que seja possível encontrar e corrigir falhas nos testes de segurança para, consequentemente, aumentar a eficácia das ferramentas que pretendem certificar a segurança das aplicações. Para isto, a solução estudada neste documento passa por aplicar transformações sintáticas em amostras de código vulneráveis sem interferir na presença das vulnerabilidades que deverão, posteriormente, ser detetadas. Este processo baseia-se: ) nas técnicas de refatoração de código que permitem melhorar a qualidade interna do software; ) no sistema de testes de mutação usado para avaliar a qualidade de testes de software. Para implementar esta ideia, uma ferramenta chamada VSG foi desenvolvida com a funcionalidade de produzir novas amostras de código com falhas de segurança. Neste documento é descrito todo o processo de desenvolvimento, desde a arquitetura até à implementação da ferramenta. No final, existe uma análise com os resultados obtidos ao tentar detetar as vulnerabilidades presentes nas amostras produzidas através da aplicação CxSAST da empresa Checkmarx, da qual esta dissertação surgiu

Improving Function Coverage with Munch: A Hybrid Fuzzing and Directed Symbolic Execution Approach

Fuzzing and symbolic execution are popular techniques for finding vulnerabilities and generating test-cases for programs. Fuzzing, a blackbox method that mutates seed input values, is generally incapable of generating diverse inputs that exercise all paths in the program. Due to the path-explosion problem and dependence on SMT solvers, symbolic execution may also not achieve high path coverage. A hybrid technique involving fuzzing and symbolic execution may achieve better function coverage than fuzzing or symbolic execution alone. In this paper, we present Munch, an open source framework implementing two hybrid techniques based on fuzzing and symbolic execution. We empirically show using nine large open-source programs that overall, Munch achieves higher (in-depth) function coverage than symbolic execution or fuzzing alone. Using metrics based on total analyses time and number of queries issued to the SMT solver, we also show that Munch is more efficient at achieving better function coverage.Comment: To appear at 33rd ACM/SIGAPP Symposium On Applied Computing (SAC). To be held from 9th to 13th April, 201

Evolutionary improvement of programs

Most applications of genetic programming (GP) involve the creation of an entirely new function, program or expression to solve a specific problem. In this paper, we propose a new approach that applies GP to improve existing software by optimizing its non-functional properties such as execution time, memory usage, or power consumption. In general, satisfying non-functional requirements is a difficult task and often achieved in part by optimizing compilers. However, modern compilers are in general not always able to produce semantically equivalent alternatives that optimize non-functional properties, even if such alternatives are known to exist: this is usually due to the limited local nature of such optimizations. In this paper, we discuss how best to combine and extend the existing evolutionary methods of GP, multiobjective optimization, and coevolution in order to improve existing software. Given as input the implementation of a function, we attempt to evolve a semantically equivalent version, in this case optimized to reduce execution time subject to a given probability distribution of inputs. We demonstrate that our framework is able to produce non-obvious optimizations that compilers are not yet able to generate on eight example functions. We employ a coevolved population of test cases to encourage the preservation of the function's semantics. We exploit the original program both through seeding of the population in order to focus the search, and as an oracle for testing purposes. As well as discussing the issues that arise when attempting to improve software, we employ rigorous experimental method to provide interesting and practical insights to suggest how to address these issues