18 research outputs found
Achievements, open problems and challenges for search based software testing
Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda, focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach
Applying search in an automatic contract-based testing tool
ISSN:0302-9743ISSN:1611-334
Achievements, Open Problems and Challenges for Search Based Software Testing
testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda1, focusing on the open problems and chal-lenges of testing non-functional properties, in particular a topic we call ‘Search Based Energy Testing ’ (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach. I
Using contracts to guide the search-based verification of concurrent programs
ISSN:0302-9743ISSN:1611-334
An Empirical Study of Meta- and Hyper-Heuristic Search for Multi-Objective Release Planning
A variety of meta-heuristic search algorithms have been introduced for optimising software release planning. However, there has been no comprehensive empirical study of different search algorithms across multiple different real-world datasets. In this article, we present an empirical study of global, local, and hybrid meta- and hyper-heuristic search-based algorithms on 10 real-world datasets. We find that the hyper-heuristics are particularly effective. For example, the hyper-heuristic genetic algorithm significantly outperformed the other six approaches (and with high effect size) for solution quality 85% of the time, and was also faster than all others 70% of the time. Furthermore, correlation analysis reveals that it scales well as the number of requirements increases
Specialising Software for Different Downstream Applications Using Genetic Improvement and Code Transplantation
OAPA Genetic improvement uses computational search to improve existing software while retaining its partial functionality. Genetic improvement has previously been concerned with improving a system with respect to all possible usage scenarios. In this paper, we show how genetic improvement can also be used to achieve specialisation to a specific set of usage scenarios. We use genetic improvement to evolve faster versions of a C++ program, a Boolean satisfiability solver called MiniSAT, specialising it for three applications. Our specialised solvers achieve between 4% and 36% execution time improvement, which is commensurate with efficiency gains achievable using human expert optimisation for the general solver. We also use genetic improvement to evolve faster versions of an image processing tool called ImageMagick, utilising code from GraphicsMagick, another image processing tool which was forked from it. We specialise the format conversion functionality to black & amp; white images and colour images only. Our specialised versions achieve up to 3% execution time improvement
Testing web enabled simulation at scale using metamorphic testing
We report on Facebook's deployment of MIA (Metamorphic Interaction Automaton). MIA is used to test Facebook's Web Enabled Simulation, built on a web infrastructure of hundreds of millions of lines of code. MIA tackles the twin problems of test flakiness and the unknowable oracle problem. It uses metamorphic testing to automate continuous integration and regression test execution. MIA also plays the role of a test bot, automatically commenting on all relevant changes submitted for code review. It currently uses a suite of over 40 metamorphic test cases. Even at this extreme scale, a non-trivial metamorphic test suite subset yields outcomes within 20 minutes (sufficient for continuous integration and review processes). Furthermore, our offline mode simulation reduces test flakiness from approximately 50% (of all online tests) to 0% (offline). Metamorphic testing has been widely-studied for 22 years. This paper is the first reported deployment into an industrial continuous integration system
Using Genetic Improvement and Code Transplants to Specialise a C++ Program to a Problem Class
Genetic Improvement (GI) is a form of Genetic Programming that improves an existing program. We use GI to evolve a faster version of a C++ program, a Boolean satisfiability (SAT) solver called MiniSAT, specialising it for a particular problem class, namely Combinatorial Interaction Testing (CIT), using automated code transplantation. Our GI-evolved solver achieves overall 17 percent improvement, making it comparable with average expert human performance. Additionally, this automatically evolved solver is faster than any of the human-improved solvers for the CIT problem
Genetic Improvement of Software: From Program Landscapes to the Automatic Improvement of a Live System
In today’s technology driven society, software is becoming increasingly important in more
areas of our lives. The domain of software extends beyond the obvious domain of computers,
tablets, and mobile phones. Smart devices and the internet-of-things have inspired the integra-
tion of digital and computational technology into objects that some of us would never have
guessed could be possible or even necessary. Fridges and freezers connected to social media
sites, a toaster activated with a mobile phone, physical buttons for shopping, and verbally
asking smart speakers to order a meal to be delivered. This is the world we live in and it is an
exciting time for software engineers and computer scientists. The sheer volume of code that is
currently in use has long since outgrown beyond the point of any hope for proper manual
maintenance. The rate of which mobile application stores such as Google’s and Apple’s have
expanded is astounding.
The research presented here aims to shed a light on an emerging field of research, called
Genetic Improvement ( GI ) of software. It is a methodology to change program code to improve
existing software. This thesis details a framework for GI that is then applied to explore fitness
landscape of bug fixing Python software, reduce execution time in a C ++ program, and
integrated into a live system.
We show that software is generally not fragile and although fitness landscapes for GI are
flat they are not impossible to search in. This conclusion applies equally to bug fixing in small
programs as well as execution time improvements. The framework’s application is shown to
be transportable between programming languages with minimal effort. Additionally, it can be
easily integrated into a system that runs a live web service.
The work within this thesis was funded by EPSRC grant EP/J017515/1 through the DAASE
project