Investigating benchmark correlations when comparing algorithms with parameter tuning: detailed experiments and results.

Benchmarks are important to demonstrate the utility of optimisation algorithms, but there is controversy about the practice of benchmarking; we could select instances that present our algorithm favourably, and dismiss those on which our algorithm underperforms. Several papers highlight the pitfalls concerned with benchmarking, some of which concern the context of the automated design of algorithms, where we use a set of problem instances (benchmarks) to train our algorithm. As with machine learning, if the training set does not reflect the test set, the algorithm will not generalize. This raises some open questions concerning the use of test instances to automatically design algorithms. We use differential evolution and sweep the parameter settings to investigate the practice of benchmarking using the BBOB benchmarks. We make three key findings. Firstly, several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances, possibly introducing unwanted bias to a resulting automatically designed algorithm. Secondly, the number of evaluations can have a large effect on the conclusion. Finally, a systematic sweep of the parameters shows how performance varies with time across the space of algorithm configurations. The datasets, including all computed features, the evolved policies and their performances, and the visualisations for all feature sets are available from the University of Stirling Data Repository (http://hdl.handle.net/11667/109)

Investigating benchmark correlations when comparing algorithms with parameter tuning.

Benchmarks are important for comparing performance of optimisation algorithms, but we can select instances that present our algorithm favourably, and dismiss those on which our algorithm under-performs. Also related are automated design of algorithms, which use problem instances (benchmarks) to train an algorithm: careful choice of instances is needed for the algorithm to generalise. We sweep parameter settings of differential evolution to applied to the BBOB benchmarks. Several benchmark functions are highly correlated. This may lead to the false conclusion that an algorithm performs well in general, when it performs poorly on a few key instances. These correlations vary with the number of evaluations

Towards explainable metaheuristics: feature extraction from trajectory mining.

Explaining the decisions made by population-based metaheuristics can often be considered difficult due to the stochastic nature of the mechanisms employed by these optimisation methods. As industries continue to adopt these methods in areas that increasingly require end-user input and confirmation, the need to explain the internal decisions being made has grown. In this article, we present our approach to the extraction of explanation supporting features using trajectory mining. This is achieved through the application of principal components analysis techniques to identify new methods of tracking population diversity changes post-runtime. The algorithm search trajectories were generated by solving a set of benchmark problems with a genetic algorithm and a univariate estimation of distribution algorithm and retaining all visited candidate solutions which were then projected to a lower dimensional sub-space. We also varied the selection pressure placed on high fitness solutions by altering the selection operators. Our results show that metrics derived from the projected sub-space algorithm search trajectories are capable of capturing key learning steps and how solution variable patterns that explain the fitness function may be captured in the principal component coefficients. A comparative study of variable importance rankings derived from a surrogate model built on the same dataset was also performed. The results show that both approaches are capable of identifying key features regarding variable interactions and their influence on fitness in a complimentary fashion

Localized Axion Photon States in a Strong Magnetic Field

We consider the axion field and electromagnetic waves with rapid time dependence, coupled to a strong time independent, asymptotically approaching a constant at infinity "mean" magnetic field, which takes into account the back reaction from the axion field and electromagnetic waves with rapid time dependence in a time averaged way. The direction of the self consistent mean field is orthogonal to the common direction of propagation of the axion and electromagnetic waves with rapid time dependence and parallel to the polarization of these electromagnetic waves. Then, there is an effective U(1) symmetry mixing axions and photons. Using the natural complex variables that this U(1) symmetry suggests we find localized planar soliton solutions. These solutions appear to be stable since they produce a different magnetic flux than the state with only a constant magnetic field, which we take as our "ground state". The solitons also have non trivial U(1) charge defined before, different from the uncharged vacuum.Comment: 9 pages, Latex, pacs:11.30.Fs, 14.80.Mz, 14.70.Bh. A small change in text introduce

Generating easy and hard problems using the proximate optimality principle.

We present an approach to generating problems of variable difficulty based on the well-known Proximate Optimality Principle (POP), often paraphrased as similar solutions have similar fitness. We explore definitions of this concept in terms of metrics in objective space and in representation space and define POP in terms of coherence of these metrics. We hypothesise that algorithms will perform well when the neighbourhoods they explore in representation space are coherent with the natural metric induced by fitness on objective space. We develop an explicit method of problem generation which creates bit string problems where the natural fitness metric is coherent or anti-coherent with Hamming neighbourhoods. We conduct experiments to show that coherent problems are easy whereas anti-coherent problems are hard for local hill climbers using the Hamming neighbourhoods

Structural coherence of problem and algorithm: an analysis for EDAs on all 2-bit and 3-bit problems.

Metaheuristics assume some kind of coherence between decision and objective spaces. Estimation of Distribution algorithms approach this by constructing an explicit probabilistic model of high fitness solutions, the structure of which is intended to reflect the structure of the problem. In this context, 'structure' means the dependencies or interactions between problem variables in a probabilistic graphical model. There are many approaches to discovering these dependencies, and existing work has already shown that often these approaches discover 'unnecessary' elements of structure - that is, elements which are not needed to correctly rank solutions. This work performs an exhaustive analysis of all 2 and 3 bit problems, grouped into classes based on mononotic invariance. It is shown in [1] that each class has a minimal Walsh structure that can be used to solve the problem. We compare the structure discovered by different structure learning approaches to the minimal Walsh structure for each class, with summaries of which interactions are (in)correctly identified. Our analysis reveals a large number of symmetries that may be used to simplify problem solving. We show that negative selection can result in improved coherence between discovered and necessary structure, and conclude with some directions for a general programme of study building on this work

First-principles study of nucleation, growth, and interface structure of Fe/GaAs

We use density-functional theory to describe the initial stages of Fe film growth on GaAs(001), focusing on the interplay between chemistry and magnetism at the interface. Four features appear to be generic: (1) At submonolayer coverages, a strong chemical interaction between Fe and substrate atoms leads to substitutional adsorption and intermixing. (2) For films of several monolayers and more, atomically abrupt interfaces are energetically favored. (3) For Fe films over a range of thicknesses, both Ga- and As-adlayers dramatically reduce the formation energies of the films, suggesting a surfactant-like action. (4) During the first few monolayers of growth, Ga or As atoms are likely to be liberated from the interface and diffuse to the Fe film surface. Magnetism plays an important auxiliary role for these processes, even in the dilute limit of atomic adsorption. Most of the films exhibit ferromagnetic order even at half-monolayer coverage, while certain adlayer-capped films show a slight preference for antiferromagnetic order.Comment: 11 two-column pages, 12 figures, to appear in Phys. Rev.

Embedding a Native State into a Random Heteropolymer Model: The Dynamic Approach

We study a random heteropolymer model with Langevin dynamics, in the supersymmetric formulation. Employing a procedure similar to one that has been used in static calculations, we construct an ensemble in which the affinity of the system for a native state is controlled by a "selection temperature" T0. In the limit of high T0, the model reduces to a random heteropolymer, while for T0-->0 the system is forced into the native state. Within the Gaussian variational approach that we employed previously for the random heteropolymer, we explore the phases of the system for large and small T0. For large T0, the system exhibits a (dynamical) spin glass phase, like that found for the random heteropolymer, below a temperature Tg. For small T0, we find an ordered phase, characterized by a nonzero overlap with the native state, below a temperature Tn \propto 1/T0 > Tg. However, the random-globule phase remains locally stable below Tn, down to the dynamical glass transition at Tg. Thus, in this model, folding is rapid for temperatures between Tg and Tn, but below Tg the system can get trapped in conformations uncorrelated with the native state. At a lower temperature, the ordered phase can also undergo a dynamical glass transition, splitting into substates separated by large barriers.Comment: 19 pages, revtex, 6 figure