Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution

It is not uncommon that meta-heuristic algorithms contain some intrinsic parameters, the optimal configuration of which is crucial for achieving their peak performance. However, evaluating the effectiveness of a configuration is expensive, as it involves many costly runs of the target algorithm. Perhaps surprisingly, it is possible to build a cheap-to-evaluate surrogate that models the algorithm's empirical performance as a function of its parameters. Such surrogates constitute an important building block for understanding algorithm performance, algorithm portfolio/selection, and the automatic algorithm configuration. In principle, many off-the-shelf machine learning techniques can be used to build surrogates. In this paper, we take the differential evolution (DE) as the baseline algorithm for proof-of-concept study. Regression models are trained to model the DE's empirical performance given a parameter configuration. In particular, we evaluate and compare four popular regression algorithms both in terms of how well they predict the empirical performance with respect to a particular parameter configuration, and also how well they approximate the parameter versus the empirical performance landscapes

Finding High-Dimensional D-OptimalDesigns for Logistic Models via Differential Evolution

D-optimal designs are frequently used in controlled experiments to obtain the most accurateestimate of model parameters at minimal cost. Finding them can be a challenging task, especially whenthere are many factors in a nonlinear model. As the number of factors becomes large and interact withone another, there are many more variables to optimize and the D-optimal design problem becomes highdimensionaland non-separable. Consequently, premature convergence issues arise. Candidate solutions gettrapped in local optima and the classical gradient-based optimization approaches to search for the D-optimaldesigns rarely succeed. We propose a specially designed version of differential evolution (DE) which is arepresentative gradient-free optimization approach to solve such high-dimensional optimization problems.The proposed specially designed DE uses a new novelty-based mutation strategy to explore the variousregions in the search space. The exploration of the regions will be carried out differently from the previouslyexplored regions and the diversity of the population can be preserved. The proposed novelty-based mutationstrategy is collaborated with two common DE mutation strategies to balance exploration and exploitationat the early or medium stage of the evolution. Additionally, we adapt the control parameters of DE as theevolution proceeds. Using logistic models with several factors on various design spaces as examples, oursimulation results show our algorithm can find D-optimal designs efficiently and the algorithm outperformsits competitors. As an application, we apply our algorithm and re-design a 10-factor car refueling experimentwith discrete and continuous factors and selected pairwise interactions. Our proposed algorithm was able toconsistently outperform the other algorithms and find a more efficient D-optimal design for the problem

Evolutionary Multiobjective Optimization Driven by Generative Adversarial Networks (GANs)

Recently, increasing works have proposed to drive evolutionary algorithms using machine learning models. Usually, the performance of such model based evolutionary algorithms is highly dependent on the training qualities of the adopted models. Since it usually requires a certain amount of data (i.e. the candidate solutions generated by the algorithms) for model training, the performance deteriorates rapidly with the increase of the problem scales, due to the curse of dimensionality. To address this issue, we propose a multi-objective evolutionary algorithm driven by the generative adversarial networks (GANs). At each generation of the proposed algorithm, the parent solutions are first classified into real and fake samples to train the GANs; then the offspring solutions are sampled by the trained GANs. Thanks to the powerful generative ability of the GANs, our proposed algorithm is capable of generating promising offspring solutions in high-dimensional decision space with limited training data. The proposed algorithm is tested on 10 benchmark problems with up to 200 decision variables. Experimental results on these test problems demonstrate the effectiveness of the proposed algorithm

Diffusion-Driven Domain Adaptation for Generating 3D Molecules

Can we train a molecule generator that can generate 3D molecules from a new domain, circumventing the need to collect data? This problem can be cast as the problem of domain adaptive molecule generation. This work presents a novel and principled diffusion-based approach, called GADM, that allows shifting a generative model to desired new domains without the need to collect even a single molecule. As the domain shift is typically caused by the structure variations of molecules, e.g., scaffold variations, we leverage a designated equivariant masked autoencoder (MAE) along with various masking strategies to capture the structural-grained representations of the in-domain varieties. In particular, with an asymmetric encoder-decoder module, the MAE can generalize to unseen structure variations from the target domains. These structure variations are encoded with an equivariant encoder and treated as domain supervisors to control denoising. We show that, with these encoded structural-grained domain supervisors, GADM can generate effective molecules within the desired new domains. We conduct extensive experiments across various domain adaptation tasks over benchmarking datasets. We show that our approach can improve up to 65.6% in terms of success rate defined based on molecular validity, uniqueness, and novelty compared to alternative baselines.Comment: 11 pages, 3 figures, and 3 table

BiLO-CPDP: Bi-Level Programming for Automated Model Discovery in Cross-Project Defect Prediction

Cross-Project Defect Prediction (CPDP), which borrows data from similar projects by combining a transfer learner with a classifier, have emerged as a promising way to predict software defects when the available data about the target project is insufficient. How-ever, developing such a model is challenge because it is difficult to determine the right combination of transfer learner and classifier along with their optimal hyper-parameter settings. In this paper, we propose a tool, dubbedBiLO-CPDP, which is the first of its kind to formulate the automated CPDP model discovery from the perspective of bi-level programming. In particular, the bi-level programming proceeds the optimization with two nested levels in a hierarchical manner. Specifically, the upper-level optimization routine is designed to search for the right combination of transfer learner and classifier while the nested lower-level optimization routine aims to optimize the corresponding hyper-parameter settings.To evaluateBiLO-CPDP, we conduct experiments on 20 projects to compare it with a total of 21 existing CPDP techniques, along with its single-level optimization variant and Auto-Sklearn, a state-of-the-art automated machine learning tool. Empirical results show that BiLO-CPDP champions better prediction performance than all other 21 existing CPDP techniques on 70% of the projects, while be-ing overwhelmingly superior to Auto-Sklearn and its single-level optimization variant on all cases. Furthermore, the unique bi-level formalization inBiLO-CPDP also permits to allocate more budget to the upper-level, which significantly boosts the performance

Evolutionary methods for modelling and control of linear and nonlinear systems

The aim of this work is to explore the potential and enhance the capability of evolutionary computation for the development of novel and advanced methodologies for engineering system modelling and controller design automation. The key to these modelling and design problems is optimisation. Conventional calculus-based methods currently adopted in engineering optimisation are in essence local search techniques, which require derivative information and lack of robustness in solving practical engineering problems. One objective of this research is thus to develop an effective and reliable evolutionary algorithm for engineering applications. For this, a hybrid evolutionary algorithm is developed, which combines the global search power of a "generational" EA with the interactive local fine-tuning of Boltzmann learning. It overcomes the weakness in local exploration and chromosome stagnation usually encountered in pure EAs. A novel one-integer-one-parameter coding scheme is also developed to significantly reduce the quantisation error, chromosome length and processing overhead time. An "Elitist Direct Inheritance" technique is developed to incorporate with Bolzmann learning for reducing the control parameters and convergence time of EAs. Parallelism of the hybrid EA is also realised in this thesis with nearly linear pipelinability. Generic model reduction and linearisation techniques in L2 and L∞ norms are developed based on the hybrid EA technique. They are applicable to both discrete and continuous-time systems in both the time and the frequency domains. Superior to conventional model reduction methods, the EA based techniques are capable of simultaneously recommending both an optimal order number and optimal parameters by a control gene used as a structural switch. This approach is extended to MIMO system linearisation from both a non-linear model and I/O data of the plant. It also allows linearisation for an entire operating region with the linear approximate-model network technique studied in this thesis. To build an original model, evolutionary black-box and clear-box system identification techniques are developed based on the L2 norm. These techniques can identify both the system parameters and transport delay in the same evolution process. These open-loop identification methods are further extended to closed-loop system identification. For robust control, evolutionary L∞ identification techniques are developed. Since most practical systems are nonlinear in nature and it is difficult to model the dominant dynamics of such a system while retaining neglected dynamics for accuracy, evolutionary grey-box modelling techniques are proposed. These techniques can utilise physical law dominated global clearbox structure, with local black-boxes to include unmeasurable nonlinearities as the coefficient models of the clear-box. This unveils a new way of engineering system modelling. With an accurately identified model, controller design problems still need to be overcome. Design difficulties by conventional analytical and numerical means are discussed and a design automation technique is then developed. This is again enabled by the hybrid evolutionary algorithm in this thesis. More importantly, this technique enables the unification of linear control system designs in both the time and the frequency domains under performance satisfaction. It is also extended to control along a trajectory of operating points for nonlinear systems. In addition, a multi-objective evolutionary algorithm is developed to make the design more transparent and visible. To achieve a step towards autonomy in building control systems, a technique for direct designs from plant step response data is developed, which bypasses the system identification phase. These computer-automated intelligent design methodologies are expected to offer added productivity and quality of control systems

Unleashing the Potential of Spiking Neural Networks for Sequential Modeling with Contextual Embedding

The human brain exhibits remarkable abilities in integrating temporally distant sensory inputs for decision-making. However, existing brain-inspired spiking neural networks (SNNs) have struggled to match their biological counterpart in modeling long-term temporal relationships. To address this problem, this paper presents a novel Contextual Embedding Leaky Integrate-and-Fire (CE-LIF) spiking neuron model. Specifically, the CE-LIF model incorporates a meticulously designed contextual embedding component into the adaptive neuronal firing threshold, thereby enhancing the memory storage of spiking neurons and facilitating effective sequential modeling. Additionally, theoretical analysis is provided to elucidate how the CE-LIF model enables long-term temporal credit assignment. Remarkably, when compared to state-of-the-art recurrent SNNs, feedforward SNNs comprising the proposed CE-LIF neurons demonstrate superior performance across extensive sequential modeling tasks in terms of classification accuracy, network convergence speed, and memory capacity