Achievements, Open Problems and Challenges for Search Based Software Testing

testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda1, focusing on the open problems and chal-lenges of testing non-functional properties, in particular a topic we call ‘Search Based Energy Testing ’ (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach. I

Performance evaluation metrics for multi-objective evolutionary algorithms in search-based software engineering: Systematic literature review

Many recent studies have shown that various multi-objective evolutionary algorithms have been widely applied in the field of search-based software engineering (SBSE) for optimal solutions. Most of them either focused on solving newly re-formulated problems or on proposing new approaches, while a number of studies performed reviews and comparative studies on the performance of proposed algorithms. To evaluate such performance, it is necessary to consider a number of performance metrics that play important roles during the evaluation and comparison of investigated algorithms based on their best-simulated results. While there are hundreds of performance metrics in the literature that can quantify in performing such tasks, there is a lack of systematic review conducted to provide evidence of using these performance metrics, particularly in the software engineering problem domain. In this paper, we aimed to review and quantify the type of performance metrics, number of objectives, and applied areas in software engineering that reported in primary studies-this will eventually lead to inspiring the SBSE community to further explore such approaches in depth. To perform this task, a formal systematic review protocol was applied for planning, searching, and extracting the desired elements from the studies. After considering all the relevant inclusion and exclusion criteria for the searching process, 105 relevant articles were identified from the targeted online databases as scientific evidence to answer the eight research questions. The preliminary results show that remarkable studies were reported without considering performance metrics for the purpose of algorithm evaluation. Based on the 27 performance metrics that were identified, hypervolume, inverted generational distance, generational distance, and hypercube-based diversity metrics appear to be widely adopted in most of the studies in software requirements engineering, software design, software project management, software testing, and software verification. Additionally, there are increasing interest in the community in re-formulating many objective problems with more than three objectives, yet, currently are dominated in re-formulating two to three objectives

A multi-objective evolutionary approach for automatic generation of test cases from state machines

Orientador: Eliane MartinsTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: A geração automática de casos de teste contribui tanto para melhorar a produtividade quanto para reduzir esforço e custo no processo de desenvolvimento de software. Neste trabalho é proposta uma abordagem, denominada MOST (Multi-Objective Search-based Testing approach from EFSM), para gerar casos de teste a partir de Máquina de Estados Finitos Estendida (MEFE) com a aplicação de uma técnica de otimização. No teste baseado em MEFE, é necessário encontrar uma sequência de entrada para exercitar um caminho no modelo, a fim de cobrir um critério de teste (e.g. todas as transições). Como as sequências podem ter diferentes tamanhos, motivou-se o desenvolvimento do algoritmo M-GEOvsl (Multi-Objective Generalized Extremal Optimization with variable string length) que permite gerar soluções de diferentes tamanhos. Além disso, por ser um algoritmo multiobjetivo, M-GEOvsl também possibilita que mais de um critério seja usado para avaliar as soluções. Com a aplicação desse algoritmo em MOST, tanto a cobertura da transição alvo quanto o tamanho da sequência são levados em consideração na geração de casos de teste. Para guiar a busca, são utilizadas informações das dependências do modelo. O algoritmo gera as sequências de entrada, incluindo os valores de seus parâmetros. Em MOST, um modelo executável da MEFE recebe como entrada os dados gerados pelo M-GEOvsl e produz dinamicamente os caminhos percorridos. Uma vez que os aspectos de controle e dados do modelo são considerados durante a execução do modelo, evita-se o problema de geração de caminhos infactíveis. Um caminho pode ser sintaticamente possível, mas semanticamente infactível, devido aos conitos de dados envolvidos no modelo. Para avaliar a abordagem proposta foram realizados vários experimentos com modelos da literatura e de aplicações reais. Os resultados da abordagem também foram comparados com os casos de teste obtidos em um trabalho relacionado.Abstract: Automated test case generation can improve the productivity as well as reduce effort and cost in the software development process. In this work an approach, named MOST (Multi- Objective Search-based Testing approach from EFSM), is proposed to generate test cases from Extended Finite State Machine (EFSM) using an optimization technique. In EFSM based testing, an input sequence should be found to sensitize a path in the model, in order to cover a test criterion (e.g. all transitions). As the sequences can have different lengths, it motivates the development of the M-GEOvsl (Multi-Objective Generalized Extremal Optimization with variable string length) algorithm that makes possible the generation of solutions with different lengths. Moreover, as a multiobjective algorithm, M-GEOvsl also allows to use more than one criterion to evaluate the solutions. Using this algorithm in MOST, the coverage of the target transition as well as the sequence length are taken into account in the test case generation. To guide the search, the information about the model dependences is used. The algorithm generates the input sequences, including the values of their parameters. In MOST, an executable model of the EFSM receives as input the data generated by M-GEOvsl and produces the traversed paths dynamically. Since the control and data aspects are considered during model execution, the problem of infeasible path generation is avoided. A path can be syntatically possible, but semantically infeasible, due to the data conicts in the model. In order to evaluate the proposed approach, experiments were performed with models of the literature and real-world applications. The results were also compared to the test cases obtained in a related workDoutoradoCiência da ComputaçãoDoutor em Ciência da Computaçã