Competent Program Evolution, Doctoral Dissertation, December 2006

Heuristic optimization methods are adaptive when they sample problem solutions based on knowledge of the search space gathered from past sampling. Recently, competent evolutionary optimization methods have been developed that adapt via probabilistic modeling of the search space. However, their effectiveness requires the existence of a compact problem decomposition in terms of prespecified solution parameters. How can we use these techniques to effectively and reliably solve program learning problems, given that program spaces will rarely have compact decompositions? One method is to manually build a problem-specific representation that is more tractable than the general space. But can this process be automated? My thesis is that the properties of programs and program spaces can be leveraged as inductive bias to reduce the burden of manual representation-building, leading to competent program evolution. The central contributions of this dissertation are a synthesis of the requirements for competent program evolution, and the design of a procedure, meta-optimizing semantic evolutionary search (MOSES), that meets these requirements. In support of my thesis, experimental results are provided to analyze and verify the effectiveness of MOSES, demonstrating scalability and real-world applicability

Unifying a Geometric Framework of Evolutionary Algorithms and Elementary Landscapes Theory

Evolutionary algorithms (EAs) are randomised general-purpose strategies, inspired by natural evolution, often used for finding (near) optimal solutions to problems in combinatorial optimisation. Over the last 50 years, many theoretical approaches in evolutionary computation have been developed to analyse the performance of EAs, design EAs or measure problem difficulty via fitness landscape analysis. An open challenge is to formally explain why a general class of EAs perform better, or worse, than others on a class of combinatorial problems across representations. However, the lack of a general unified theory of EAs and fitness landscapes, across problems and representations, makes it harder to characterise pairs of general classes of EAs and combinatorial problems where good performance can be guaranteed provably. This thesis explores a unification between a geometric framework of EAs and elementary landscapes theory, not tied to a specific representation nor problem, with complementary strengths in the analysis of population-based EAs and combinatorial landscapes. This unification organises around three essential aspects: search space structure induced by crossovers, search behaviour of population-based EAs and structure of fitness landscapes. First, this thesis builds a crossover classification to systematically compare crossovers in the geometric framework and elementary landscapes theory, revealing a shared general subclass of crossovers: geometric recombination P-structures, which covers well-known crossovers. The crossover classification is then extended to a general framework for axiomatically analysing the population behaviour induced by crossover classes on associated EAs. This shows the shared general class of all EAs using geometric recombination P-structures, but no mutation, always do the same abstract form of convex evolutionary search. Finally, this thesis characterises a class of globally convex combinatorial landscapes shared by the geometric framework and elementary landscapes theory: abstract convex elementary landscapes. It is formally explained why geometric recombination P-structure EAs expectedly can outperform random search on abstract convex elementary landscapes related to low-order graph Laplacian eigenvalues. Altogether, this thesis paves a way towards a general unified theory of EAs and combinatorial fitness landscapes

Optimisation of multiplier-less FIR filter design techniques

This thesis is concerned with the design of multiplier-less (ML) finite impulse response (FIR) digital filters. The use of multiplier-less digital filters results in simplified filtering structures, better throughput rates and higher speed. These characteristics are very desirable in many DSP systems. This thesis concentrates on the design of digital filters with power-of-two coefficients that result in simplified filtering structures. Two distinct classesof ML FIR filter design algorithms are developed and compared with traditional techniques. The first class is based on the sensitivity of filter coefficients to rounding to power-of-two. Novel elements include extending of the algorithm for multiple-bands filters and introducing mean square error as the sensitivity criterion. This improves the performance of the algorithm and reduces the complexity of resulting filtering structures. The second class of filter design algorithms is based on evolutionary techniques, primarily genetic algorithms. Three different algorithms based on genetic algorithm kernel are developed. They include simple genetic algorithm, knowledge-based genetic algorithm and hybrid of genetic algorithm and simulated annealing. Inclusion of the additional knowledge has been found very useful when re-designing filters or refining previous designs. Hybrid techniques are useful when exploring large, N-dimensional searching spaces. Here, the genetic algorithm is used to explore searching space rapidly, followed by fine search using simulated annealing. This approach has been found beneficial for design of high-order filters. Finally, a formula for estimation of the filter length from its specification and complementing both classes of design algorithms, has been evolved using techniques of symbolic regression and genetic programming. Although the evolved formula is very complex and not easily understandable, statistical analysis has shown that it produces more accurate results than traditional Kaiser's formula. In summary, several novel algorithms for the design of multiplier-less digital filters have been developed. They outperform traditional techniques that are used for the design of ML FIR filters and hence contributed to the knowledge in the field of ML FIR filter design

Prescriptive formalism for constructing domain-specific evolutionary algorithms

It has been widely recognised in the computational intelligence and machine learning communities that the key to understanding the behaviour of learning algorithms is to understand what representation is employed to capture and manipulate knowledge acquired during the learning process. However, traditional evolutionary algorithms have tended to employ a fixed representation space (binary strings), in order to allow the use of standardised genetic operators. This approach leads to complications for many problem domains, as it forces a somewhat artificial mapping between the problem variables and the canonical binary representation, especially when there are dependencies between problem variables (e.g. problems naturally defined over permutations). This often obscures the relationship between genetic structure and problem features, making it difficult to understand the actions of the standard genetic operators with reference to problem-specific structures. This thesis instead advocates m..

Immunology as a metaphor for computational information processing : fact or fiction?

The biological immune system exhibits powerful information processing capabilities, and therefore is of great interest to the computer scientist. A rapidly expanding research area has attempted to model many of the features inherent in the natural immune system in order to solve complex computational problems. This thesis examines the metaphor in detail, in an effort to understand and capitalise on those features of the metaphor which distinguish it from other existing methodologies. Two problem domains are considered — those of scheduling and data-clustering. It is argued that these domains exhibit similar characteristics to the environment in which the biological immune system operates and therefore that they are suitable candidates for application of the metaphor. For each problem domain, two distinct models are developed, incor-porating a variety of immunological principles. The models are tested on a number of artifical benchmark datasets. The success of the models on the problems considered confirms the utility of the metaphor

Minimal models of evolution: germline fitness effects of cancer mutations and stochastic tunneling under strong recombination

In a time where data on the genetic make-up of organisms is available in abundance, the theory of evolution is of immediate importance to answer key questions of biology: How can one explain the variation seen in the DNA of different organisms and species? What are the effects of changes in the DNA on the function of cells? What are the driving mechanisms of diseases with a genetic component such as cancer? Minimal mathematical models of evolution provide a basis for the interpretation of DNA data. The explanations they offer are concrete and testable, their assumptions and limitations explicit. The application and further development of minimal evolution models is the main theme of this work. In the first part, the functional effects of mutations found in cancer cells are analyzed from the perspective of germline evolution. This is the process that produced the DNA of organisms as we see it today. Mutations have an effect on the fitness of healthy cells. This impact can be estimated from the variation seen in the sequences of protein domains. It is found that this evolutionarily informed conservation score has utility to identify cancer driver genes, especially if they are tumor suppressor genes. The relevance of this fitness scale for cancer mutations is demonstrated on a data set of mutations in protein kinase genes. This analysis is followed by an application of Hidden Markov Models (HMM) to the detection of signals of positive selection in cancer mutation data. Cancer as an evolutionary process of cells is markedly different from the process of germline evolution. Cancer-specific selection can be seen in genes, whose activity or lack thereof is essential for the progress of cancer. These cancer genes exhibit an increased rate of amino acid changing mutations, beyond the level expected by chance. The identification of these genes is a statistical task for which HMM are shown to be most suitable. Finally, an extended mathematical model of evolution is analyzed which describes the adaptation of a sexually reproducing population to a global fitness maximum via compensatory mutations. In a two-locus/two-allele model, the compound effects of mutation, selection, genetic drift, recombination and sign epistasis lead to the interesting situation of adaption via the crossing of a fitness valley in genotype space. This bottleneck can be overcome by rare large fluctuations in the allele frequencies overcoming the effect of recombinatorial reshuffling. The relevant time scales are derived for a parameter regime that includes large recombination