Achievements, open problems and challenges for search based software testing

Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda, focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach

Achievements, Open Problems and Challenges for Search Based Software Testing

testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda1, focusing on the open problems and chal-lenges of testing non-functional properties, in particular a topic we call ‘Search Based Energy Testing ’ (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach. I

Applying search in an automatic contract-based testing tool

ISSN:0302-9743ISSN:1611-334

Using contracts to guide the search-based verification of concurrent programs

ISSN:0302-9743ISSN:1611-334

Specialising Software for Different Downstream Applications Using Genetic Improvement and Code Transplantation

OAPA Genetic improvement uses computational search to improve existing software while retaining its partial functionality. Genetic improvement has previously been concerned with improving a system with respect to all possible usage scenarios. In this paper, we show how genetic improvement can also be used to achieve specialisation to a specific set of usage scenarios. We use genetic improvement to evolve faster versions of a C++ program, a Boolean satisfiability solver called MiniSAT, specialising it for three applications. Our specialised solvers achieve between 4% and 36% execution time improvement, which is commensurate with efficiency gains achievable using human expert optimisation for the general solver. We also use genetic improvement to evolve faster versions of an image processing tool called ImageMagick, utilising code from GraphicsMagick, another image processing tool which was forked from it. We specialise the format conversion functionality to black & amp; white images and colour images only. Our specialised versions achieve up to 3% execution time improvement

Testing web enabled simulation at scale using metamorphic testing

We report on Facebook's deployment of MIA (Metamorphic Interaction Automaton). MIA is used to test Facebook's Web Enabled Simulation, built on a web infrastructure of hundreds of millions of lines of code. MIA tackles the twin problems of test flakiness and the unknowable oracle problem. It uses metamorphic testing to automate continuous integration and regression test execution. MIA also plays the role of a test bot, automatically commenting on all relevant changes submitted for code review. It currently uses a suite of over 40 metamorphic test cases. Even at this extreme scale, a non-trivial metamorphic test suite subset yields outcomes within 20 minutes (sufficient for continuous integration and review processes). Furthermore, our offline mode simulation reduces test flakiness from approximately 50% (of all online tests) to 0% (offline). Metamorphic testing has been widely-studied for 22 years. This paper is the first reported deployment into an industrial continuous integration system

Using Genetic Improvement and Code Transplants to Specialise a C++ Program to a Problem Class

Genetic Improvement (GI) is a form of Genetic Programming that improves an existing program. We use GI to evolve a faster version of a C++ program, a Boolean satisfiability (SAT) solver called MiniSAT, specialising it for a particular problem class, namely Combinatorial Interaction Testing (CIT), using automated code transplantation. Our GI-evolved solver achieves overall 17 percent improvement, making it comparable with average expert human performance. Additionally, this automatically evolved solver is faster than any of the human-improved solvers for the CIT problem

Information Retrieval and Spectrum Based Bug Localization: Better Together

Debugging often takes much effort and resources. To help developers debug, numerous information retrieval (IR)-based and spectrum-based bug localization techniques have been proposed. IR-based techniques process textual infor-mation in bug reports, while spectrum-based techniques pro-cess program spectra (i.e., a record of which program el-ements are executed for each test case). Both eventually generate a ranked list of program elements that are likely to contain the bug. However, these techniques only con-sider one source of information, either bug reports or pro-gram spectra, which is not optimal. To deal with the limita-tion of existing techniques, in this work, we propose a new multi-modal technique that considers both bug reports and program spectra to localize bugs. Our approach adaptively creates a bug-specific model to map a particular bug to its possible location, and introduces a novel idea of suspicious words that are highly associated to a bug. We evaluate our approach on 157 real bugs from four software systems, and compare it with a state-of-the-art IR-based bug localization method, a state-of-the-art spectrum-based bug localization method, and three state-of-the-art multi-modal feature loca-tion methods that are adapted for bug localization. Experi-ments show that our approach can outperform the baselines by at least 47.62%, 31.48%, 27.78%, and 28.80 % in terms of number of bugs successfully localized when a developer in

Genetic Improvement of Software: From Program Landscapes to the Automatic Improvement of a Live System

In today’s technology driven society, software is becoming increasingly important in more areas of our lives. The domain of software extends beyond the obvious domain of computers, tablets, and mobile phones. Smart devices and the internet-of-things have inspired the integra- tion of digital and computational technology into objects that some of us would never have guessed could be possible or even necessary. Fridges and freezers connected to social media sites, a toaster activated with a mobile phone, physical buttons for shopping, and verbally asking smart speakers to order a meal to be delivered. This is the world we live in and it is an exciting time for software engineers and computer scientists. The sheer volume of code that is currently in use has long since outgrown beyond the point of any hope for proper manual maintenance. The rate of which mobile application stores such as Google’s and Apple’s have expanded is astounding. The research presented here aims to shed a light on an emerging field of research, called Genetic Improvement ( GI ) of software. It is a methodology to change program code to improve existing software. This thesis details a framework for GI that is then applied to explore fitness landscape of bug fixing Python software, reduce execution time in a C ++ program, and integrated into a live system. We show that software is generally not fragile and although fitness landscapes for GI are flat they are not impossible to search in. This conclusion applies equally to bug fixing in small programs as well as execution time improvements. The framework’s application is shown to be transportable between programming languages with minimal effort. Additionally, it can be easily integrated into a system that runs a live web service. The work within this thesis was funded by EPSRC grant EP/J017515/1 through the DAASE project

Uma abordagem de otimização multiobjetivo para projeto arquitetural de linha de produto de software

Resumo: A indústria de software tem adotado a abordagem de Linha de Produto de Software (LPS) com o objetivo de aumentar o reúso de software e diminuir o tempo de produção e os custos de desenvolvimento dos produtos. Nessa abordagem, o principal artefato e a arquitetura de LPS (PLA - Product Line Architecture). No entanto, obter uma PLA modular, extensível e reusável e uma tarefa não trivial. O arquiteto pode se apoiar em métricas arquiteturais para definir e melhorar o projeto da PLA. Contudo, essa tarefa pode envolver vários fatores, muitas vezes conflitantes entre si, e encontrar o melhor trade-off entre as métricas utilizadas para avaliar o projeto transforma o projeto de PLA em uma tarefa que demanda grande esforço humano. Nesse contexto, o projeto de PLA pode ser formulado como um problema de otimização com varios fatores. Porém, elaborar um projeto que atenda a todos os fatores envolvidos pode ser mais difícil do que reconhecer um bom projeto. Problemas da Engenharia de Software similares a esse tem sido eficientemente resolvidos com algoritmos de busca em um campo de pesquisa conhecido como Engenharia de Software Baseada em Busca (SBSE - Search Based Software Engineering). Entretanto, as abordagens existentes utilizadas para otimizar arquiteturas de software nãao são apropriadas para projeto de PLAs, pois não consideram características específicas de LPS. Desse modo, este trabalho propõe uma abordagem de otimização multiobjetivo automatizada para avaliar e melhorar um projeto de PLA no que tange a modularização de características, estabilidade do projeto e extensibilidade de LPS. A abordagem proposta inclui: (a) um processo sistemático para conduzir a otimização de projeto de PLA por meio de algoritmos de busca; (b) um metamodelo que permite que esses algoritmos manipulem projetos de PLA; (c) novos operadores de busca para evoluir projetos de PLA em termos de modularização de características; e (d) um tratamento multiobjetivo para o problema de projeto de PLA. Esse tratamento multiobjetivo engloba métricas que indicam a modularização de características e a extensibilidade de LPS, além de métricas convencionais para medir princípios básicos de projeto como coesão e acoplamento. Ao final do processo de otimização, um conjunto de possíveis soluções de projeto de PLA que representam os melhores trade-off entre os objetivos otimizados e retornado. O arquiteto deve selecionar uma solução de acordo com as suas prioridades. A ferramenta OPLA-Tool foi desenvolvida para instanciar a abordagem usando algoritmos evolutivos multiobjetivos, os quais tem sido usados com sucesso na área de SBSE. Utilizando a OPLA-Tool, quatro estudos empíricos foram realizados com nove PLAs para avaliar: os operadores de busca propostos; o uso das métricas de LPS; e os algoritmos escolhidos. Em comparação às PLAs originais, os resultados mostraram que a abordagem proposta consegue gerar projetos mais estáveis, mais elegantes e com melhor modularização de características