We argue that molecular self-organisation during embryonic development allows evolution to perform highly nonlinear combinatorial optimisation. A structured approach to architectural optimisation of large-scale Artificial Neural Networks using this principle is presented. We also present simulation results demonstrating the evolution of an edge detecting retina using the proposed methodology

Adams, R.G.

Bolouri, H.

George, S.

Rust, A.G.

English

University of Hertfordshire Research Archive

Molecular Self-Organisation in a Developmental Model for the Evolution
of Large-Scale Artificial Neural Networks
Hamid Bolouri1,2, Rod Adams3, Stella George3, Alistair G Rust1,3
1Engineering R & D Centre, University of Hertfordshire, AL10 9AB, UK
2Division of Biology, California Institute of Technology, USA
3
 Dept. Computer Science, University of Hertfordshire, AL10 9AB, UK
email: H.Bolouri, R.G.Adams, S.J.George, A.G.Rust@herts.ac.uk
Abstract
We argue that molecular self-organisation during embryonic
development allows evolution to perform highly nonlinear
combinatorial optimisation. A structured approach to
architectural optimisation of large-scale Artificial Neural
Networks using this principle is presented. We also present
simulation results demonstrating the evolution of an edge
detecting retina using the proposed methodology.
Introduction
One of the attractions of Artificial Neural Networks (ANNs)
has been the possibility of designing intelligent systems
capable of optimising their functionality according to
application requirements.
Adapting the architecture of an ANN (number of neurons,
connectivity pattern, and neuron function) to a given
application can be viewed as a combinatorial optimisation
problem. For sophisticated applications, the problem tends to
be high-dimensional and highly nonlinear. Direct applications
of current combinatorial optimisation methods to such
problems tend to be unacceptably inefficient.
Approaches to ANN architecture optimisation
   	
    ﬀﬁﬃﬂ
! "$#%'&(#*)%'+,! ).-
/1023'4576,/849 :,;
<= >.?A@ BC<EDFB*G H
>ID
H
?B>= J
KLNMOQPR S
L7T!U,V UK
WX,YFZ*[Y]\ ^_ `abc dCc e8fbc `g hi*jFikml.nljpo.q
rIs.t.uNvIrwr.t.xzyIs{uN|]}~s{ 87|} sI.uNr. r1mu$t. s1|E}*vrIrIsI7  ﬃ{}rIs{t.uﬃvrz.8{|' v. vII
Ł8 7F 
Figure 1, approaches to ANN architecture optimisation
Figure 1 presents an overview of current approaches to ANN
architecture optimisation. Much research in self-adapting
ANNs has focused on the design of unsupervised learning
algorithms, and the use of algorithmic (generative) techniques
to ‘grow’ or ‘prune’ neurons and their connections.
Unsupervised learning, and generative algorithms both make
strong assumptions about the ANN architecture and the
characteristics of the target application domain. To alleviate
these restrictions, a number of researchers have exploited
evolutionary methods to design adaptive generative
algorithms (see [Rust98a] for a review). The generative
algorithm is encoded in a genome. Its execution (the mapping
from genotype to phenotype) mimics embryonic development.
Current implementations of this approach limit the
architectural search space through apriori assumptions and
algorithmic restrictions. However, restricting the range of
genotype to phenotype mappings can hinder rather than help
the optimisation process: nonoptimal mappings result in more
nonlinear search spaces which are more difficult to search.
We argue that it is possible to evolve unconstrained optimal
mappings through the use of two strategies:
1) Cellular (neuron) characteristics should be defined in terms
of interacting molecular processes.   These molecular
interactions result in nonlinear genotype-phenotype mappings.
Their evolutionary optimisation reduces the degree of
nonlinearity in the genotype search space and simplifies the
optimisation process.
2) Evolution should be staged, mimicking the process of
speciation. Specifically, evolution should start with the
simplest set of generative rules, search for the best achievable
phenotype, then incorporate new (more sophisticated) rules
(mimicking the emergence of a new species), and search
again; repeating this procedure until satisfactory phenotypes
emerge.
The paper presents simulation results demonstrating
molecular, staged evolution of a simple edge detecting retina.
Methodology
Our strategy is based on a molecular model of the evolution
of embryonic development with the following characteristics:
1) Neurons are modelled as cells. Each distinct cell type is
defined by the interactions of a specific subset of genes within
the genome.
2) The genome, and the evolutionary operators mutation and
cross-over, are defined at the molecular level (i.e. a much
finer level of detail than cell characteristics such as receptive
field size).
3) The translation from genetic encoding to neural network
(genotype to phenotype mapping) is performed using
sequences of operations defined by individual genes.  These
operations mimic gene regulation  (the sequential activation
and interaction of genes) and embryonic development in
biological systems.
4) Although cells of the same type (ensembles of neurons)
share the same genetic description, individual neurons within
an ensemble may differ from each other if differences in cell-
cell or cell- environment interactions lead to different
developmental histories.
5) Hierarchic network structures can be defined with nested
loops of gene interactions. Thus, similar cells in different parts
of a network may utilise common segments of the genome to
define the parts they have in common.
6) A gene (more accurately a gene product) has two aspects:
the definition of what it will interact with, and the definition of
the nature of its interactions  (affinity and  function
respectively).  In the simulations presented here, the space of
all possible protein functions was predefined by mimicking
known biological functions. We carried out numerous
developmental simulations to ensure that the set of functions
included in our repertoire was sufficient to generate a wide
variety of neuron morphologies [Rust98a,b]. Similarly, each
affinity value (interaction rate) is discretised to a finite range
of possible values. Molecular evolution thus becomes a
combinatorial optimisation task. A developmental program is
evolved by constructing sequences of gene interactions where
each gene  is selected from among a large, but finite ensemble
of predefined genes.
7) The genome is defined as a (variable) number of distinct
chromosomes each comprising a set of genes which may, in
principle, interact with each other. During cross-over only
alike chromosomes can exchange parts.
8) The neural system is evolved gradually and in stages
reminiscent of the emergence of new species. Speciation is
achieved by allowing the size of the genome to vary through
addition of new genes to the genome by the user when a
genome of a certain size is deemed close to its peak fitness.
9) Evolution starts by using only the simplest of developmental
programs. In this case, neuron connectivity is specified using
only intrinsic growth rules. When this type of genome is
deemed to have been adequately optimised, additional genes
describing potentially more powerful developmental processes
are added to the genome and a new cycle of evolutionary
optimisation starts. Some examples of the developmental
processes modelled are given in figure 2.
Example developmental rules
• Intrinsic
• length +=   l
• branch@(time)
• direction += 
• Interactive
• direction += func(    attractive -     repulsive)
• concentration = 1/X
• branch = prob(|opposing attractant gradients|)
• growth = 1-  (synaptic activity)
• pruning = func(RC integrate & fire physiology +
    competitive Hebbian adaptation)

Figure 2, example developmental processes modelled
10) No information other than the performance of the evolved
networks is used to guide the evolutionary process.
Example self-organising molecular evolution
We present the modelling of a simple edge-detecting retina as
a testbed application. Starting from a random genome, we
evolve a retina with on-centre/off-surround bipolar cells
performing edge detection. The choice of a retina as our
testbed application has a number of advatages, summarised in
figure 3. In particular, the simple, layered structure of the
retina allows us to simplify embryonic development to the
formation of connections between growing axons and
dendrites.
Advantages of evolving of an edge-detecting retina
– No learning + deterministic behaviour + no feedback
• fast fitness evaluation
– Layered structure
• simple cell differentiation and migration
• can focus on connectivity formation
– 3 types of neurons + regular, repeated, modular structure
• evolve developmental rules for a small network, grow a large system
• good example of a heterogenous, modular network
– Highly stereotyped + simple neuron physiology + well studied
• exemplary solutions provided by biology!
• much data on retinal development
– Potential to expand towards
•  HNN-style hierarchical feature clustering & shape recognition
• feedback for attention and gain control
Figure 3, the selected testbed application allows
investigations to focus on the most pertinent issues.
Experimental details
The edge-detecting retina comprises 3 types of cells: cones,
horizontal cells, and bipolar cells. Cones are photodetectors
which transmit the input signal to specialized synapses named
triad junctions. Horizontal cells average the signal value they
receive through their connections to triad junctions. Bipolar
cells are inhibited by input from horizontal cells (surround
activity) and excited by cones (centre activity). Figure 4 shows
a single triad junction and its connections. Figure 5 shows plan
views of the target network structure and illustrates the desired
network functionality. Note that the ANN structures we evolve
are 3 dimensional in order to allow neurites to pass across each
other without colliding. This is necessary because our long
term objective is to evolve a Neocognitron-like image
recognition system [Sabisch97, 98]. The retina reported here
may be viewed as the first stage of  such a network. Figure 6
shows the 3 dimensional structure of an example evolved
network. Figure 7 shows a plan view of the same network so
that it can be related to figure 6.
Evolution was performed using the GENESIS Genetic
Algorithm package [Grefenstette90]. In all cases, the
population size was 50, and the cross-over rate was 0.6. In the
first 2 stages of evolution a fixed mutation rate of 0.001 was
used, in the final stage the mutation was initially set to 0.01
and was reduced gradually to 0.001. Selection was rank based
using elitist criteria. Five visually salient images (edges at
various orientations)  were used to evaluate the fitness of all
evolved networks.
  
	  
  ﬀﬂﬁﬃ 
 ﬀ!ﬃﬁ"ﬃ# $ﬀ
%'&$( )*,+.-/0%$( 1/
Figure 4, a single triad junction and its connections
To reduce computational load, we took advantage of the
modular, repeated structure of our retinas to evolve networks
responding to 8x8 retinal cones (i.e. 64 cones, 36 horizontal
cells, 25 bipolar cells). The developmental parameters thus
evolved were then used to grow larger retinas (1024 cones,
900 horizontal cells, and 481 bipolar cells).
Results
As can be seen from figure 8(a), intrinsic developmental
processes alone are sufficient to evolve perfect functionality
for a network of deterministically placed neurons. This is
similar to results achieved by other researchers in the field (see
[Rust98a] for a review). However, when the starting positions
of the neurons are perturbed (neuron positions varied in any of
three directions by 10% with a 25% probability), intrinsic
rules result in very poor evolved perfomance (figure 8(b)).
243
56798 :; <3= >?!7 243
56798A@3?$; B35CD>= 7 EGFH HIDJ#K.L MN
OPQSRUT VXWYP
Z\[].^[]
_`Sa bScSdfe ghc iSjk k l
m$n"oho pSn"qSrs
tu vfwfx ySz {|wS}'~!

 SS'
 

SŁSh Ł"' " 
SŁSh Ł"' " |Ł"'
   
Ł"S'
 




Figure 5, the target retinal structure and function. Note
that cones and horizontal cells occupy the same grid point
in plan view, but are spatially distinct in 3D.
In the next stage of evolution, interactive parameters were
added to the genome permitting neurites to self organise by
interacting with each other and producing extra branches
determined by local needs. The new population was seeded
using the best parameters from the intrinsic growth stage. See
figure 8(c) for resultant edge detection functionality. Note that
performance from intrinsic only growth has been improved,
but is still not optimal .
Figure 6, 3D view of an evolved retina
In the final stage of evolution, parameters for self-organising
pruning mechanisms (see [Rust97] for details) were added to
the genome. The new population was seeded using the best
parameters from the previous two stages. Multiple retinas
were again grown and evaluated. The final set of evolved
parameters were used to grow a 32x32 retina where the
positions of the neurons were perturbed as before. Figure 8(d)
shows the resultant edge detection behaviour. Note that all 4
edges in the image are now fully detected. Figure 9 shows the
results of another run  of our staged evolution algorithm. Note
the the evovled network behaviours are different, but exhibit
the same progressive improvement in functionality as more
self-organising developmental interactions are added.
Figure 7, plan view of the 3D retina structure
       (a)     (b)
                  (c)     (d)
Figure 8, evolved functionality for a 32x32 retina. The input
image and desired output are shown in figure 5. (a)-(e) show
outputs of  retinas grown using the parameters from the best
evolved individuals. (a) Intrinsic growth rules, symmetric retina.
(b) Intrinsic growth rules, perturbed retina. (c) Intrinsic and
interactive rules, perturbed retina. (d) Intrinsic, interactive and
pruning rules, perturbed retina.
Conclusions
A methodology to evolve complex ANN structures using
molecular self-organisation during embryonic development
was presented. Our simulations of the evolution of an edge-
detecting retina demonstrate the benefits of molecular self-
organisation and staged speciation in providing increased
developmental robustness and functional fitness.
  	
 	
ﬀ ﬁﬃﬂ

! "ﬀ ﬁ
ﬀ #%$&"	' ()
'%ﬀ  (	*ﬀ ﬁﬃﬂ
+,-./ ,	0
/ 132
4 56789	:;64 <*5*=
>
4 6?5<*4 @ﬃ7
A5684 5	@
4 :3B
4 56789!:;64 <*5CB
D	EF*GIHGID	JLKM
NPOQSR&TU T	V*W
X
U YZT	[*U \ﬃ]
^_` acbd efﬃg hi&j
Figure 9, another example of  the benefits of staged
evolution (in this case of an edge detecting retina)
References
[Grefenstette 90], Grefenstette, JJ, GENESIS from:
 ftp.aic.nrl.navy.mil:/pub/galist/src/1990
[Rust98a] A. G. Rust, Developmental self organisation in
Artificial Neural Networks, PhD Thesis, Dept. of Computer
Science, University of Hertfordshire, June 1998.
 [Rust98b] Rust A G, Adams R, George S, and Bolouri H,
Evolution of developmental ontogeny for robustly
reproducible phenotypes, submitted to the 5th Int. Conf. on
Parallel Problem Solving from Nature, Amsterdam, Sept
27-30, 1998.
[Rust97] Rust A G, Adams R, George S & Bolouri H,
Activity-Based Pruning in Developmental Artificial Neural
Networks, Proc. 4th European Conf. on Artificial Life
(ECAL97), Eds. Husbands P & Harvey I, MIT Press, 224-
233, 1997
[Sabisch98] Sabisch T, Ferguson A & Bolouri H, Automatic
Registration of Complex Images Using a Self Organizing
Neural System , Proc. 1998 IEEE International Joint
Conference on Neural Networks (IJCNN'98), Achorage,
Alaska, 165-170, 4-9 May 1998.
[Sabisch97] Sabisch T, Ferguson A & Bolouri H, Rotation,
Translation and Scaling Tolerant Recognition of Complex
Shapes using a Hierarchical Self-Organising Neural
Network, Proc. Fourth International Conference on Neural
Information Processing (ICONIP'97), 1174-1179, Dunedin,
New Zealand, November 24-28 1997.


Activity-Based Pruning

Automatic Registration of Complex Images Using a Self Organizing Neural System ,

Developmental self organisation in Artificial Neural Networks,

Evolution of developmental ontogeny for robustly reproducible phenotypes,

Translation and Scaling Tolerant Recognition of Complex Shapes using a Hierarchical Self-Organising Neural Network,

Molecular self-organisation in a developmental model for the evolution of large-scale artificial neural networks

We argue that molecular self-organisation during embryonic development allows evolution to perform highly nonlinear combinatorial optimisation. A structured approach to architectural optimisation of large-scale Artificial Neural Networks using this principle is presented. We also present simulation results demonstrating the evolution of an edge detecting retina using the proposed methodology.  Introduction  One of the attractions of Artificial Neural Networks (ANNs) has been the possibility of designing intelligent systems capable of optimising their functionality according to application requirements. Adapting the architecture of an ANN (number of neurons, connectivity pattern, and neuron function) to a given application can be viewed as a combinatorial optimisation problem. For sophisticated applications, the problem tends to be high-dimensional and highly nonlinear. Direct applications of current combinatorial optimisation methods to such problems tend to be unacceptably inefficient..

Hamid Bolouri

Rod Adams

Stella George

Alistair G Rust

CiteSeerX

Molecular Self-Organisation in a Developmental Model for the Evolution of Large-Scale Artificial Neural Networks

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.57.2538

Molecular self-organisation in a developmental model for the evolution of large-scale artificial neural networks

Abstract

Similar works

Full text

Available Versions

University of Hertfordshire Research Archive

CiteSeerX