Identification of robotic manipulators' inverse dynamics coefficients via model-based adaptive networks

The values of a given manipulator's dynamics coefficients need to be accurately identified in order to employ model-based algorithms in the control of its motion. This thesis details the development of a novel form of adaptive network which is capable of accurately learning the coefficients of systems, such as manipulator inverse dynamics, where the algebraic form is known but the coefficients' values are not. Empirical motion data from a pair of PUMA 560s has been processed by the Context-Sensitive Linear Combiner (CSLC) network developed, and the coefficients of their inverse dynamics identified. The resultant precision of control is shown to be superior to that achieved from employing dynamics coefficients derived from direct measurement. As part of the development of the CSLC network, the process of network learning is examined. This analysis reveals that current network architectures for processing analogue output systems with high input order are highly unlikely to produce solutions that are good estimates throughout the entire problem space. In contrast, the CSLC network is shown to generalise intrinsically as a result of its structure, whilst its training is greatly simplified by the presence of only one minima in the network's error hypersurface. Furthermore, a fine-tuning algorithm for network training is presented which takes advantage of the CSLC network's single adaptive layer structure and does not rely upon gradient descent of the network error hypersurface, which commonly slows the later stages of network training

Coding Strategies for Genetic Algorithms and Neural Nets

The interaction between coding and learning rules in neural nets (NNs), and between coding and genetic operators in genetic algorithms (GAs) is discussed. The underlying principle advocated is that similar things in "the world" should have similar codes. Similarity metrics are suggested for the coding of images and numerical quantities in neural nets, and for the coding of neural network structures in genetic algorithms. A principal component analysis of natural images yields receptive fields resembling horizontal and vertical edge and bar detectors. The orientation sensitivity of the "bar detector" components is found to match a psychophysical model, suggesting that the brain may make some use of principal components in its visual processing. Experiments are reported on the effects of different input and output codings on the accuracy of neural nets handling numeric data. It is found that simple analogue and interpolation codes are most successful. Experiments on the coding of image data demonstrate the sensitivity of final performance to the internal structure of the net. The interaction between the coding of the target problem and reproduction operators of mutation and recombination in GAs are discussed and illustrated. The possibilities for using GAs to adapt aspects of NNs are considered. The permutation problem, which affects attempts to use GAs both to train net weights and adapt net structures, is illustrated and methods to reduce it suggested. Empirical tests using a simulated net design problem to reduce evaluation times indicate that the permutation problem may not be as severe as has been thought, but suggest the utility of a sorting recombination operator, that matches hidden units according to the number of connections they have in common. A number of experiments using GAs to design network structures are reported, both to specify a net to be trained from random weights, and to prune a pre-trained net. Three different coding methods are tried, and various sorting recombination operators evaluated. The results indicate that appropriate sorting can be beneficial, but the effects are problem-dependent. It is shown that the GA tends to overfit the net to the particular set of test criteria, to the possible detriment of wider generalisation ability. A method of testing the ability of a GA to make progress in the presence of noise, by adding a penalty flag, is described

The application of b-mode ultrasonography for analysis of human skeletal muscle

Skeletal muscles control the joints of the skeletal system and they allow human movement and interaction with the environment. They are vital for stability in balance, walking and running, and many other skilled motor tasks. To understand how muscles operate in general and specific situations there are a variety of tools at the disposal of research scientists and clinicians for analysing muscle function. Strain gauges for example allow the quantification of forces exerted during joint rotation. However, skeletal muscles are multilayer systems and often different muscles are responsible for the overall force generated during joint rotation. Therefore, strain gauges do not reveal the extent of the contribution of individual muscles during muscle function. The most widely-used and accepted muscle analysis tool is electromyography (EMG), which can measure the activation level of individual muscles by measuring the electrical potential propagating through muscle resulting from local activations of motor units. However, EMG does not linearly relate to any real physical forces, meaning that without prior knowledge of the force exertion on the level of the muscle, force cannot be estimated. EMG can measure superficial layers of muscle non-invasively by attaching surface electrodes (surface EMG) to the skin over the belly of the muscle. To measure the activity of individual muscle beneath the superficial muscle, a needle or thin-wire electrode must be inserted through the skin and into the muscle volume (intramuscular EMG), which is invasive and not practical in many situations. Furthermore, intramuscular EMG can only provide measurement of a very small volume (<1mm3) which can have varying amounts of active motor units. Ultrasonography is a powerful cost-effective non-invasive imaging technology which allows real-time observation of cross-sections of multiple layers of dynamic skeletal muscle. Recent advances in automated skeletal muscle ultrasound analysis techniques, and advances in image processing techniques make ultrasound a valuable line of investigation for analysis of dynamic skeletal muscle. This aim of this thesis is to study and develop advanced image analysis techniques applicable to the analysis of dynamic skeletal muscle. The broader aim is to understand the capacity/limits of ultrasound as a skeletal muscle analysis tool. The ideas presented within offer new approaches to modelling complex muscle architecture and function via ultrasound. Tools have also been developed here that will contribute to, and promote ultrasound skeletal muscle analysis as a new and emerging technology which may be used by clinicians and research scientists to develop our understanding of skeletal muscle function. The main findings of this thesis are that automated segmentation of architecturally simple and complex skeletal muscle groups is possible and accurate, and that information about joint angles and muscle activity/force can be automatically extracted directly from ultrasound images without the explicit knowledge of how to extract it. The techniques used offer new possibilities for non-invasive information extraction from complex muscle groups such as the muscles in the human posterior neck

On the induction of temporal structure by recurrent neural networks

Language acquisition is one of the core problems in artificial intelligence (AI) and it is generally accepted that any successful AI account of the mind will stand or fall depending on its ability to model human language. Simple Recurrent Networks (SRNs) are a class of so-called artificial neural networks that have a long history in language modelling via learning to predict the next word in a sentence. However, SRNs have also been shown to suffer from catastrophic forgetting, lack of syntactic systematicity and an inability to represent more than three levels of centre-embedding, due to the so-called 'vanishing gradients' problem. This problem is caused by the decay of past input information encoded within the error-gradients which vanish exponentially as additional input information is encountered and passed through the recurrent connections. That said, a number of architectural variations have been applied which may compensate for this issue, such as the Nonlinear Autoregressive Network with exogenous inputs (NARX) network and the multi-recurrent network (MRN). In addition to this, Echo State Networks (ESNs) are a relatively new class of recurrent neural network that do not suffer from the vanishing gradients problem and have been shown to exhibit state-of-the-art performance in tasks such as motor control, dynamic time series prediction, and more recently language processing. This research re-explores the class of SRNs and evaluates them against the state-of-the-art ESN to identify which model class is best able to induce the underlying finite-state automaton of the target grammar implicitly through the next word prediction task. In order to meet its aim, the research analyses the internal representations formed by each of the different models and explores the conditions under which they are able to carry information about long term sequential dependencies beyond what is found in the training data. The findings of the research are significant. It reveals that the traditional class of SRNs, trained with backpropagation through time, are superior to ESNs for the grammar prediction task. More specifically, the MRN, with its state-based memory of varying rigidity, is more able to learn the underlying grammar than any other model. An analysis of the MRN’s internal state reveals that this is due to its ability to maintain a constant variance within its state-based representation of the embedded aspects (or finite state machines) of the target grammar. The investigations show that in order to successfully induce complex context free grammars directly from sentence examples, then not only are a hidden layer and output layer recurrency required, but so is self-recurrency on the context layer to enable varying degrees of current and past state information, that are integrated over time

An investigation of the design and use of feed-forward artificial neural networks in the classification of remotely sensed images

Artificial neural networks (ANNs) have attracted the attention of researchers in many fields, and have been used to solve a wide range of problems. In the field of remote sensing they have been used in a variety of applications, including land cover mapping, image compression, geological mapping and meteorological image classification, and have generally proved to be more powerful than conventional statistical classifiers, especially when training data are limited and the data in each class are not normally distributed. The use of ANNs requires some critical decisions on the part of the user. These decisions, which are mainly concerned with the determinations of the components of the network structure and the parameters defined for the learning algorithm, can significantly affect the accuracy of the resulting classification. Although there are some discussions in the literature regarding the issues that affect network performance, there is no standard method or approach that is universally accepted to determine the optimum values of these parameters for a particular problem. In this thesis, a feed-forward network structure that learns the characteristics of the training data through the backpropagation learning algorithm is employed to classify land cover features using multispectral, multitemporal, and multisensory image data. The thesis starts with a review and discussion of general principles of classification and the use of artificial neural networks. Special emphasis is put on the issue of feature selection, due to the availability of hyperspectral image data from recent sensors. The primary aims of this research are to comprehensively investigate the impact of the choice of network architecture and initial parameter estimates, and to compare a number of heuristics developed by researchers. The most effective heuristics are identified on the basis of a large number of experiments employing two real-world datasets, and the superiority of the optimum settings using the 'best' heuristics is then validated using an independent dataset. The results are found to be promising in terms of ease of design and use of ANNs, and in producing considerably higher classification accuracies than either the maximum likelihood or neural network classifiers constructed using ad hoc design and implementation strategies. A number of conclusions are drawn and later used to generate a comprehensive set of guidelines that will facilitate the process of design and use of artificial neural networks in remote sensing image classification. This study also explores the use of visualisation techniques in understanding the behaviour of artificial neural networks and the results produced by them. A number of visual analysis techniques are employed to examine the internal characteristics of the training data. For this purpose, a toolkit allowing the analyst to perform a variety of visualisation and analysis procedures was created using the MATLAB software package, and is available in the accompanying CD-ROM. This package was developed during the course of this research, and contains the tools used during the investigations reported in this thesis. The contribution to knowledge of the research work reported in this thesis lies in the identification of optimal strategies for the use of ANNs in land cover classifications based on remotely sensed data. Further contributions include an indepth analysis of feature selection methods for use with high-dimensional datasets, and the production of a MATLAB toolkit that implements the methods used in this study

Neuroinformatics in Functional Neuroimaging

This Ph.D. thesis proposes methods for information retrieval in functional neuroimaging through automatic computerized authority identification, and searching and cleaning in a neuroscience database. Authorities are found through cocitation analysis of the citation pattern among scientific articles. Based on data from a single scientific journal it is shown that multivariate analyses are able to determine group structure that is interpretable as particular “known ” subgroups in functional neuroimaging. Methods for text analysis are suggested that use a combination of content and links, in the form of the terms in scientific documents and scientific citations, respectively. These included context sensitive author ranking and automatic labeling of axes and groups in connection with multivariate analyses of link data. Talairach foci from the BrainMap ™ database are modeled with conditional probability density models useful for exploratory functional volumes modeling. A further application is shown with conditional outlier detection where abnormal entries in the BrainMap ™ database are spotted using kernel density modeling and the redundancy between anatomical labels and spatial Talairach coordinates. This represents a combination of simple term and spatial modeling. The specific outliers that were found in the BrainMap ™ database constituted among others: Entry errors, errors in the article and unusual terminology

An investigation of the design and use of feed-forward artificial neural networks in the classification of remotely sensed images

Artificial neural networks (ANNs) have attracted the attention of researchers in many fields, and have been used to solve a wide range of problems. In the field of remote sensing they have been used in a variety of applications, including land cover mapping, image compression, geological mapping and meteorological image classification, and have generally proved to be more powerful than conventional statistical classifiers, especially when training data are limited and the data in each class are not normally distributed. The use of ANNs requires some critical decisions on the part of the user. These decisions, which are mainly concerned with the determinations of the components of the network structure and the parameters defined for the learning algorithm, can significantly affect the accuracy of the resulting classification. Although there are some discussions in the literature regarding the issues that affect network performance, there is no standard method or approach that is universally accepted to determine the optimum values of these parameters for a particular problem. In this thesis, a feed-forward network structure that learns the characteristics of the training data through the backpropagation learning algorithm is employed to classify land cover features using multispectral, multitemporal, and multisensory image data. The thesis starts with a review and discussion of general principles of classification and the use of artificial neural networks. Special emphasis is put on the issue of feature selection, due to the availability of hyperspectral image data from recent sensors. The primary aims of this research are to comprehensively investigate the impact of the choice of network architecture and initial parameter estimates, and to compare a number of heuristics developed by researchers. The most effective heuristics are identified on the basis of a large number of experiments employing two real-world datasets, and the superiority of the optimum settings using the 'best' heuristics is then validated using an independent dataset. The results are found to be promising in terms of ease of design and use of ANNs, and in producing considerably higher classification accuracies than either the maximum likelihood or neural network classifiers constructed using ad hoc design and implementation strategies. A number of conclusions are drawn and later used to generate a comprehensive set of guidelines that will facilitate the process of design and use of artificial neural networks in remote sensing image classification. This study also explores the use of visualisation techniques in understanding the behaviour of artificial neural networks and the results produced by them. A number of visual analysis techniques are employed to examine the internal characteristics of the training data. For this purpose, a toolkit allowing the analyst to perform a variety of visualisation and analysis procedures was created using the MATLAB software package, and is available in the accompanying CD-ROM. This package was developed during the course of this research, and contains the tools used during the investigations reported in this thesis. The contribution to knowledge of the research work reported in this thesis lies in the identification of optimal strategies for the use of ANNs in land cover classifications based on remotely sensed data. Further contributions include an indepth analysis of feature selection methods for use with high-dimensional datasets, and the production of a MATLAB toolkit that implements the methods used in this study

The Shallow and the Deep:A biased introduction to neural networks and old school machine learning

The Shallow and the Deep is a collection of lecture notes that offers an accessible introduction to neural networks and machine learning in general. However, it was clear from the beginning that these notes would not be able to cover this rapidly changing and growing field in its entirety. The focus lies on classical machine learning techniques, with a bias towards classification and regression. Other learning paradigms and many recent developments in, for instance, Deep Learning are not addressed or only briefly touched upon.Biehl argues that having a solid knowledge of the foundations of the field is essential, especially for anyone who wants to explore the world of machine learning with an ambition that goes beyond the application of some software package to some data set. Therefore, The Shallow and the Deep places emphasis on fundamental concepts and theoretical background. This also involves delving into the history and pre-history of neural networks, where the foundations for most of the recent developments were laid. These notes aim to demystify machine learning and neural networks without losing the appreciation for their impressive power and versatility