Performance evaluation metrics for multi-objective evolutionary algorithms in search-based software engineering: Systematic literature review

Many recent studies have shown that various multi-objective evolutionary algorithms have been widely applied in the field of search-based software engineering (SBSE) for optimal solutions. Most of them either focused on solving newly re-formulated problems or on proposing new approaches, while a number of studies performed reviews and comparative studies on the performance of proposed algorithms. To evaluate such performance, it is necessary to consider a number of performance metrics that play important roles during the evaluation and comparison of investigated algorithms based on their best-simulated results. While there are hundreds of performance metrics in the literature that can quantify in performing such tasks, there is a lack of systematic review conducted to provide evidence of using these performance metrics, particularly in the software engineering problem domain. In this paper, we aimed to review and quantify the type of performance metrics, number of objectives, and applied areas in software engineering that reported in primary studies-this will eventually lead to inspiring the SBSE community to further explore such approaches in depth. To perform this task, a formal systematic review protocol was applied for planning, searching, and extracting the desired elements from the studies. After considering all the relevant inclusion and exclusion criteria for the searching process, 105 relevant articles were identified from the targeted online databases as scientific evidence to answer the eight research questions. The preliminary results show that remarkable studies were reported without considering performance metrics for the purpose of algorithm evaluation. Based on the 27 performance metrics that were identified, hypervolume, inverted generational distance, generational distance, and hypercube-based diversity metrics appear to be widely adopted in most of the studies in software requirements engineering, software design, software project management, software testing, and software verification. Additionally, there are increasing interest in the community in re-formulating many objective problems with more than three objectives, yet, currently are dominated in re-formulating two to three objectives

The relationship between search based software engineering and predictive modeling

Search Based Software Engineering (SBSE) is an approach to software engineering in which search based optimization algorithms are used to identify optimal or near optimal solutions and to yield insight. SBSE techniques can cater for multiple, possibly competing objectives and/or constraints and applications where the potential solution space is large and complex. This paper will provide a brief overview of SBSE, explaining some of the ways in which it has already been applied to construction of predictive models. There is a mutually beneficial relationship between predictive models and SBSE. The paper sets out eleven open problem areas for Search Based Predictive Modeling and describes how predictive models also have role to play in improving SBSE

Evolutionary Search Techniques with Strong Heuristics for Multi-Objective Feature Selection in Software Product Lines

Software design is a process of trading off competing objectives. If the user objective space is rich, then we should use optimizers that can fully exploit that richness. For example, this study configures software product lines (expressed as feature models) using various search-based software engineering methods. Our main result is that as we increase the number of optimization objectives, the methods in widespread use (e.g. NSGA-II, SPEA2) perform much worse than IBEA (Indicator-Based Evolutionary Algorithm). IBEA works best since it makes most use of user preference knowledge. Hence it does better on the standard measures (hypervolume and spread) but it also generates far more products with 0 violations of domain constraints. We also present significant improvements to IBEA\u27s performance by employing three strong heuristic techniques that we call PUSH, PULL, and seeding. The PUSH technique forces the evolutionary search to respect certain rules and dependencies defined by the feature models, while the PULL technique gives higher weight to constraint satisfaction as an optimization objective and thus achieves a higher percentage of fully-compliant configurations within shorter runtimes. The seeding technique helps in guiding very large feature models to correct configurations very early in the optimization process. Our conclusion is that the methods we apply in search-based software engineering need to be carefully chosen, particularly when studying complex decision spaces with many optimization objectives. Also, we conclude that search methods must be customized to fit the problem at hand. Specifically, the evolutionary search must respect domain constraints

A gaussian mixture-based approach to synthesizing nonlinear feature functions for automated object detection

Feature design is an important part to identify objects of interest into a known number of categories or classes in object detection. Based on the depth-first search for higher order feature functions, the technique of automated feature synthesis is generally considered to be a process of creating more effective features from raw feature data during the run of the algorithms. This dynamic synthesis of nonlinear feature functions is a challenging problem in object detection. This thesis presents a combinatorial approach of genetic programming and the expectation maximization algorithm (GP-EM) to synthesize nonlinear feature functions automatically in order to solve the given tasks of object detection. The EM algorithm investigates the use of Gaussian mixture which is able to model the behaviour of the training samples during an optimal GP search strategy. Based on the Gaussian probability assumption, the GP-EM method is capable of performing simultaneously dynamic feature synthesis and model-based generalization. The EM part of the approach leads to the application of the maximum likelihood (ML) operation that provides protection against inter-cluster data separation and thus exhibits improved convergence. Additionally, with the GP-EM method, an innovative technique, called the histogram region of interest by thresholds (HROIBT), is introduced for diagnosing protein conformation defects (PCD) from microscopic imagery. The experimental results show that the proposed approach improves the detection accuracy and efficiency of pattern object discovery, as compared to single GP-based feature synthesis methods and also a number of other object detection systems. The GP-EM method projects the hyperspace of the raw data onto lower-dimensional spaces efficiently, resulting in faster computational classification processes