Incremental construction of LSTM recurrent neural network

Long Short--Term Memory (LSTM) is a recurrent neural network that uses structures called memory blocks to allow the net remember significant events distant in the past input sequence in order to solve long time lag tasks, where other RNN approaches fail. Throughout this work we have performed experiments using LSTM networks extended with growing abilities, which we call GLSTM. Four methods of training growing LSTM has been compared. These methods include cascade and fully connected hidden layers as well as two different levels of freezing previous weights in the cascade case. GLSTM has been applied to a forecasting problem in a biomedical domain, where the input/output behavior of five controllers of the Central Nervous System control has to be modelled. We have compared growing LSTM results against other neural networks approaches, and our work applying conventional LSTM to the task at hand.Postprint (published version

Global adaptation in networks of selfish components: emergent associative memory at the system scale

In some circumstances complex adaptive systems composed of numerous self-interested agents can self-organise into structures that enhance global adaptation, efficiency or function. However, the general conditions for such an outcome are poorly understood and present a fundamental open question for domains as varied as ecology, sociology, economics, organismic biology and technological infrastructure design. In contrast, sufficient conditions for artificial neural networks to form structures that perform collective computational processes such as associative memory/recall, classification, generalisation and optimisation, are well-understood. Such global functions within a single agent or organism are not wholly surprising since the mechanisms (e.g. Hebbian learning) that create these neural organisations may be selected for this purpose, but agents in a multi-agent system have no obvious reason to adhere to such a structuring protocol or produce such global behaviours when acting from individual self-interest. However, Hebbian learning is actually a very simple and fully-distributed habituation or positive feedback principle. Here we show that when self-interested agents can modify how they are affected by other agents (e.g. when they can influence which other agents they interact with) then, in adapting these inter-agent relationships to maximise their own utility, they will necessarily alter them in a manner homologous with Hebbian learning. Multi-agent systems with adaptable relationships will thereby exhibit the same system-level behaviours as neural networks under Hebbian learning. For example, improved global efficiency in multi-agent systems can be explained by the inherent ability of associative memory to generalise by idealising stored patterns and/or creating new combinations of sub-patterns. Thus distributed multi-agent systems can spontaneously exhibit adaptive global behaviours in the same sense, and by the same mechanism, as the organisational principles familiar in connectionist models of organismic learning

Neural Network Modelling of Constrained Spatial Interaction Flows

Fundamental to regional science is the subject of spatial interaction. GeoComputation - a new research paradigm that represents the convergence of the disciplines of computer science, geographic information science, mathematics and statistics - has brought many scholars back to spatial interaction modeling. Neural spatial interaction modeling represents a clear break with traditional methods used for explicating spatial interaction. Neural spatial interaction models are termed neural in the sense that they are based on neurocomputing. They are clearly related to conventional unconstrained spatial interaction models of the gravity type, and under commonly met conditions they can be understood as a special class of general feedforward neural network models with a single hidden layer and sigmoidal transfer functions (Fischer 1998). These models have been used to model journey-to-work flows and telecommunications traffic (Fischer and Gopal 1994, Openshaw 1993). They appear to provide superior levels of performance when compared with unconstrained conventional models. In many practical situations, however, we have - in addition to the spatial interaction data itself - some information about various accounting constraints on the predicted flows. In principle, there are two ways to incorporate accounting constraints in neural spatial interaction modeling. The required constraint properties can be built into the post-processing stage, or they can be built directly into the model structure. While the first way is relatively straightforward, it suffers from the disadvantage of being inefficient. It will also result in a model which does not inherently respect the constraints. Thus we follow the second way. In this paper we present a novel class of neural spatial interaction models that incorporate origin-specific constraints into the model structure using product units rather than summation units at the hidden layer and softmax output units at the output layer. Product unit neural networks are powerful because of their ability to handle higher order combinations of inputs. But parameter estimation by standard techniques such as the gradient descent technique may be difficult. The performance of this novel class of spatial interaction models will be demonstrated by using the Austrian interregional traffic data and the conventional singly constrained spatial interaction model of the gravity type as benchmark. References Fischer M M (1998) Computational neural networks: A new paradigm for spatial analysis Environment and Planning A 30 (10): 1873-1891 Fischer M M, Gopal S (1994) Artificial neural networks: A new approach to modelling interregional telecommunciation flows, Journal of Regional Science 34(4): 503-527 Openshaw S (1993) Modelling spatial interaction using a neural net. In Fischer MM, Nijkamp P (eds) Geographical information systems, spatial modelling, and policy evaluation, pp. 147-164. Springer, Berlin

Transformations in the Scale of Behaviour and the Global Optimisation of Constraints in Adaptive Networks

The natural energy minimisation behaviour of a dynamical system can be interpreted as a simple optimisation process, finding a locally optimal resolution of problem constraints. In human problem solving, high-dimensional problems are often made much easier by inferring a low-dimensional model of the system in which search is more effective. But this is an approach that seems to require top-down domain knowledge; not one amenable to the spontaneous energy minimisation behaviour of a natural dynamical system. However, in this paper we investigate the ability of distributed dynamical systems to improve their constraint resolution ability over time by self-organisation. We use a ‘self-modelling’ Hopfield network with a novel type of associative connection to illustrate how slowly changing relationships between system components can result in a transformation into a new system which is a low-dimensional caricature of the original system. The energy minimisation behaviour of this new system is significantly more effective at globally resolving the original system constraints. This model uses only very simple, and fully-distributed positive feedback mechanisms that are relevant to other ‘active linking’ and adaptive networks. We discuss how this neural network model helps us to understand transformations and emergent collective behaviour in various non-neural adaptive networks such as social, genetic and ecological networks

Evolution of Neural Networks for Helicopter Control: Why Modularity Matters

The problem of the automatic development of controllers for vehicles for which the exact characteristics are not known is considered in the context of miniature helicopter flocking. A methodology is proposed in which neural network based controllers are evolved in a simulation using a dynamic model qualitatively similar to the physical helicopter. Several network architectures and evolutionary sequences are investigated, and two approaches are found that can evolve very competitive controllers. The division of the neural network into modules and of the task into incremental steps seems to be a precondition for success, and we analyse why this might be so