DeepSaucer: Unified Environment for Verifying Deep Neural Networks

In recent years, a number of methods for verifying DNNs have been developed. Because the approaches of the methods differ and have their own limitations, we think that a number of verification methods should be applied to a developed DNN. To apply a number of methods to the DNN, it is necessary to translate either the implementation of the DNN or the verification method so that one runs in the same environment as the other. Since those translations are time-consuming, a utility tool, named DeepSaucer, which helps to retain and reuse implementations of DNNs, verification methods, and their environments, is proposed. In DeepSaucer, code snippets of loading DNNs, running verification methods, and creating their environments are retained and reused as software assets in order to reduce cost of verifying DNNs. The feasibility of DeepSaucer is confirmed by implementing it on the basis of Anaconda, which provides virtual environment for loading a DNN and running a verification method. In addition, the effectiveness of DeepSaucer is demonstrated by usecase examples

The complexity of asynchronous model based testing

This is the post-print version of the final paper published in Theoretical Computer Science. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2012 Elsevier B.V.In model based testing (MBT), testing is based on a model MM that typically is expressed using a state-based language such as an input output transition system (IOTS). Most approaches to MBT assume that communications between the system under test (SUT) and its environment are synchronous. However, many systems interact with their environment through asynchronous channels and the presence of such channels changes the nature of testing. In this paper we investigate the situation in which the SUT interacts with its environment through asynchronous channels and the problems of producing test cases to reach a state, execute a transition, or to distinguish two states. In addition, we investigate the Oracle Problem. All four problems are explored for both FIFO and non-FIFO channels. It is known that the Oracle Problem can be solved in polynomial time for FIFO channels but we also show that the three test case generation problems can also be solved in polynomial time in the case where the IOTS is observable but the general test generation problems are EXPTIME-hard. For non-FIFO channels we prove that all of the test case generation problems are EXPTIME-hard and the Oracle Problem in NP-hard, even if we restrict attention to deterministic IOTSs

Metamorphic testing for cybersecurity

Metamorphic testing (MT) can enhance security testing by providing an alternative to using a testing oracle, which is often unavailable or impractical. The authors report how MT detected previously unknown bugs in real-world critical applications such as code obfuscators, giving evidence that software testing requires diverse perspectives to achieve greater cybersecurity

A Comparative Study of GUI Testing Approaches

A maioria das aplicações de software modernas apresentam uma Interface Gráfica de Utilizador (GUI), que torna a aplicação mais simples de usar, promovendo maior produtividade e melhor acessibilidade, e oferecendo flexibilidade na forma como os utilizadores podem executar tarefas. No entanto, devido à complexidade das GUIs, o processo de teste de GUI pode ser moroso e intensivo. Assim, automatizar o processo tanto quanto possível é indispensável para testar qualquer interface gráfica mais complexa e evoluída.Existem diferentes abordagens para testes de GUI automatizados, no entanto, a maioria delas requerer esforços manuais substanciais, outras apenas são capazes de encontrar erros específicos ou não permitem a reutilização de casos de teste após alterações de sistema ou GUI. Muitos investigadores afirmam que devem ser utilizadas diferentes técnicas/abordagens para um bom processo de teste.Uma nova abordagem baseada em modelos, denominada de Pattern-Based GUI Testing, foi implementada a fim de aumentar a sistematização, reutilização e diminuir o esforço da modelação e teste de GUIs. Baseia-se no conceito de Padrões de Teste de Interface de Utilizador (UITP), que contêm estratégias de teste genéricas para testar características recorrentes e comuns (UI Patterns) em GUIs. É apoiada pela ferramenta PBGT, que integra um ambiente de modelação e execução de testes de modo a suportar a criação de modelos de teste com base em UITPs, com recurso a uma linguagem específica de domínio (PARADIGM) para modelação da GUI.Como a abordagem é recente, é relevante submetê-la a experiências e testes sistematizados a fim de avaliar o seu bom desempenho/comportamento e compará-la com outras técnicas. Assim, este trabalho de dissertação baseia-se na avaliação e comparação da abordagem PBGT em relação a outras ferramentas e técnicas, no que diz respeito à detecção de falhas, facilidade de utilização, e aos esforços necessários para testar a aplicação.Para a realização de experiências, foram introduzidas mutações em três aplicações web diferentes - iAddressBook , TaskFreak e Tudu - de modo a abranger um maior número de casos de uso, e cada mutante foi, por sua vez, testado por cada uma das ferramentas selecionadas ou desenvolvidas e que implementam as diferentes abordagens de teste consideradas.Most of the modern software applications feature a Graphical User Interface (GUI), which turns the application easier to use, promoting higher productivity and better accessibility, and offering flexibility in how users perform tasks. However, due to GUI's complexity, the GUI testing process can be a time-consuming and intensive process. Therefore, automate the process as much as possible is indispensable to test any more evolved graphic user interface.There are some common automated GUI testing approaches, but while most of them require substantial manual efforts, others lack reusability or are only able to find specific types of errors. Many researchers state that a variety of techniques should be used.A new model-based testing approach, called Pattern- Based GUI Testing, was implemented in order to increase systematization, reusability and diminish the effort in modelling and testing. It is based on the concept of User Interface Test Patterns (UITP), which contain generic test strategies for testing common recurrent behavior (UI Patterns) on GUIs, and supported by the PBGT Tool which provides an integrated modeling and testing environment that supports the crafting of test models based on UI Test Patterns, using a GUI modeling DSL (PARADIGM).As a novel proposal, it is entirely relevant to submit it to systematized experiments and tests in order to assess its good performance/behavior and compare it with other techniques. Thus, this dissertation work mainly addresses PBGT's approach, aiming to compare it with different testing approaches/tools in what concerns to errors/fault detection, ease of use, and overall efforts required to test the application.To perform the experiments, mutations were introduced in each of three different web applications - iAddressBook, TaskFreak and Tudu - to cover a greater number of use cases, and each mutant was tested by each of the selected or developed testing tools which implement the considered approaches. Those approaches' benefits and problems are then conveniently described

Fault-Based Testing in the Absence of an Oracle

Although testing is the most popular method for assuring software quality, there are two recognized limitations, known as the reliable test set problem and the oracle problem. Fault-based testing is an attempt by Morell to alleviate the reliable test set problem. In this paper, we propose to enhance fault-based testing to address the oracle problem as well. We present an integrated method that combines metamorphic testing with fault-based testing using real and symbolic inputs