Training issues and learning algorithms for feedforward and recurrent neural networks

Ph.DDOCTOR OF PHILOSOPH

Microelectronic circuits for noninvasive ear type assistive devices

An ear type system and its circuit realization with application as new assistive devices are investigated. The auditory brainstem responses obtained from clinical hearing measurements are utilized for which the ear type systems mimicking the physical and behavioral characteristics of the individual auditory system are developed. In the case that effects from the hearing loss and disorder can be detected via the measured responses, differentiations between normal and impaired characteristics of the human auditory system are made possible from which the new noninvasive way of correcting these undesired effects is proposed. The ear type system of auditory brainstem response is developed using an adaptation of the nonlinear neural network architecture and the system for making a correction is realized using the derived inverse of neural network. Microelectronic circuits of the systems are designed and simulated showing a possibility of developing into a hearing aid type device which potentially helps hearing impaired patients in an alternate and noninvasive useful way

The hardware implementation of an artificial neural network using stochastic pulse rate encoding principles

In this thesis the development of a hardware artificial neuron device and artificial neural network using stochastic pulse rate encoding principles is considered. After a review of neural network architectures and algorithmic approaches suitable for hardware implementation, a critical review of hardware techniques which have been considered in analogue and digital systems is presented. New results are presented demonstrating the potential of two learning schemes which adapt by the use of a single reinforcement signal. The techniques for computation using stochastic pulse rate encoding are presented and extended with new novel circuits relevant to the hardware implementation of an artificial neural network. The generation of random numbers is the key to the encoding of data into the stochastic pulse rate domain. The formation of random numbers and multiple random bit sequences from a single PRBS generator have been investigated. Two techniques, Simulated Annealing and Genetic Algorithms, have been applied successfully to the problem of optimising the configuration of a PRBS random number generator for the formation of multiple random bit sequences and hence random numbers. A complete hardware design for an artificial neuron using stochastic pulse rate encoded signals has been described, designed, simulated, fabricated and tested before configuration of the device into a network to perform simple test problems. The implementation has shown that the processing elements of the artificial neuron are small and simple, but that there can be a significant overhead for the encoding of information into the stochastic pulse rate domain. The stochastic artificial neuron has the capability of on-line weight adaption. The implementation of reinforcement schemes using the stochastic neuron as a basic element are discussed

Towards a Brain-inspired Information Processing System: Modelling and Analysis of Synaptic Dynamics: Towards a Brain-inspired InformationProcessing System: Modelling and Analysis ofSynaptic Dynamics

Biological neural systems (BNS) in general and the central nervous system (CNS) specifically exhibit a strikingly efficient computational power along with an extreme flexible and adaptive basis for acquiring and integrating new knowledge. Acquiring more insights into the actual mechanisms of information processing within the BNS and their computational capabilities is a core objective of modern computer science, computational sciences and neuroscience. Among the main reasons of this tendency to understand the brain is to help in improving the quality of life of people suffer from loss (either partial or complete) of brain or spinal cord functions. Brain-computer-interfaces (BCI), neural prostheses and other similar approaches are potential solutions either to help these patients through therapy or to push the progress in rehabilitation. There is however a significant lack of knowledge regarding the basic information processing within the CNS. Without a better understanding of the fundamental operations or sequences leading to cognitive abilities, applications like BCI or neural prostheses will keep struggling to find a proper and systematic way to help patients in this regard. In order to have more insights into these basic information processing methods, this thesis presents an approach that makes a formal distinction between the essence of being intelligent (as for the brain) and the classical class of artificial intelligence, e.g. with expert systems. This approach investigates the underlying mechanisms allowing the CNS to be capable of performing a massive amount of computational tasks with a sustainable efficiency and flexibility. This is the essence of being intelligent, i.e. being able to learn, adapt and to invent. The approach used in the thesis at hands is based on the hypothesis that the brain or specifically a biological neural circuitry in the CNS is a dynamic system (network) that features emergent capabilities. These capabilities can be imported into spiking neural networks (SNN) by emulating the dynamic neural system. Emulating the dynamic system requires simulating both the inner workings of the system and the framework of performing the information processing tasks. Thus, this work comprises two main parts. The first part is concerned with introducing a proper and a novel dynamic synaptic model as a vital constitute of the inner workings of the dynamic neural system. This model represents a balanced integration between the needed biophysical details and being computationally inexpensive. Being a biophysical model is important to allow for the abilities of the target dynamic system to be inherited, and being simple is needed to allow for further implementation in large scale simulations and for hardware implementation in the future. Besides, the energy related aspects of synaptic dynamics are studied and linked to the behaviour of the networks seeking for stable states of activities. The second part of the thesis is consequently concerned with importing the processing framework of the dynamic system into the environment of SNN. This part of the study investigates the well established concept of binding by synchrony to solve the information binding problem and to proposes the concept of synchrony states within SNN. The concepts of computing with states are extended to investigate a computational model that is based on the finite-state machines and reservoir computing. Biological plausible validations of the introduced model and frameworks are performed. Results and discussions of these validations indicate that this study presents a significant advance on the way of empowering the knowledge about the mechanisms underpinning the computational power of CNS. Furthermore it shows a roadmap on how to adopt the biological computational capabilities in computation science in general and in biologically-inspired spiking neural networks in specific. Large scale simulations and the development of neuromorphic hardware are work-in-progress and future work. Among the applications of the introduced work are neural prostheses and bionic automation systems

Bio-inspired Dynamic Control Systems with Time Delays

The world around us exhibits a rich and ever changing environment of startling, bewildering and fascinating complexity. Almost everything is never as simple as it seems, but through the chaos we may catch fleeting glimpses of the mechanisms within. Throughout the history of human endeavour we have mimicked nature to harness it for our own ends. Our attempts to develop truly autonomous and intelligent machines have however struggled with the limitations of our human ability. This has encouraged some to shirk this responsibility and instead model biological processes and systems to do it for us. This Thesis explores the introduction of continuous time delays into biologically inspired dynamic control systems. We seek to exploit rich temporal dynamics found in physical and biological systems for modelling complex or adaptive behaviour through the artificial evolution of networks to control robots. Throughout, arguments have been presented for the modelling of delays not only to better represent key facets of physical and biological systems, but to increase the computational potential of such systems for the synthesis of control. The thorough investigation of the dynamics of small delayed networks with a wide range of time delays has been undertaken, with a detailed mathematical description of the fixed points of the system and possible oscillatory modes developed to fully describe the behaviour of a single node. Exploration of the behaviour for even small delayed networks illustrates the range of complex behaviour possible and guides the development of interesting solutions. To further exploit the potential of the rich dynamics in such systems, a novel approach to the 3D simulation of locomotory robots has been developed focussing on minimising the computational cost. To verify this simulation tool a simple quadruped robot was developed and the motion of the robot when undergoing a manually designed gait evaluated. The results displayed a high degree of agreement between the simulation and laser tracker data, verifying the accuracy of the model developed. A new model of a dynamic system which includes continuous time delays has been introduced, and its utility demonstrated in the evolution of networks for the solution of simple learning behaviours. A range of methods has been developed for determining the time delays, including the novel concept of representing the time delays as related to the distance between nodes in a spatial representation of the network. The application of these tools to a range of examples has been explored, from Gene Regulatory Networks (GRNs) to robot control and neural networks. The performance of these systems has been compared and contrasted with the efficacy of evolutionary runs for the same task over the whole range of network and delay types. It has been shown that delayed dynamic neural systems are at least as capable as traditional Continuous Time Recurrent Neural Networks (CTRNNs) and show significant performance improvements in the control of robot gaits. Experiments in adaptive behaviour, where there is not such a direct link between the enhanced system dynamics and performance, showed no such discernible improvement. Whilst we hypothesise that the ability of such delayed networks to generate switched pattern generating nodes may be useful in Evolutionary Robotics (ER) this was not borne out here. The spatial representation of delays was shown to be more efficient for larger networks, however these techniques restricted the search to lower complexity solutions or led to a significant falloff as the network structure becomes more complex. This would suggest that for anything other than a simple genotype, the direct method for encoding delays is likely most appropriate. With proven benefits for robot locomotion and the open potential for adaptive behaviour delayed dynamic systems for evolved control remain an interesting and promising field in complex systems research

Uncertainty Quantification And Economic Dispatch Models For The Power Grid

The modern power grid is constrained by several challenges, such as increased penetration of Distributed Energy Resources (DER), rising demand for Electric Vehicle (EV) integration, and the need to schedule resources in real-time accurately. To address the above challenges, this dissertation offers solutions through data-driven forecasting models, topology-aware economic dispatch models, and efficient optional power flow calculations for large scale grids. Particularly, in chapter 2, a novel microgrid decomposition scheme is proposed to divide the large scale power grids into smaller microgrids. Here, a two-stage Nearest-Generator Girvan-Newman (NGGN) algorithm, a graphicalclustering-based approach, followed by a distributed economic dispatch model, is deployed to yield a 12.64% cost savings. In chapter 3, a deep-learning-based scheduling scheme is intended for the EVs in a household community that uses forecasted demand, consumer preferences and Time-of-use (TOU) pricing scheme to reduce electricity costs for the consumers and peak shaving for the utilities. In chapter 4, a hybrid machine learning model using GLM with other methods was designed to forecast wind generation data. Finally, in chapter 5, multiple formulations for Alternating Current Optimal Power Flow (ACOPF) were designed for large scale grids in a high-performance computing environment. The ACOPF formulations, namely, power balance polar, power balance Cartesian, and current balance Cartesian, are tested on bus systems ranging from a 9-bus to 25,000. The current balance Cartesian formulation had an average of 23% faster computational time than two other formulations on a 25,000 bus system

Observações em redes neuronais

The many advances that machine learning, and especially its workhorse, deep learning, has provided to our society are undeniable. However, there is an increasing feeling that the field has become little understood, with researchers going as far as to make the analogy that it has developed into a form of alchemy. There is the need for a deeper understanding of the tools being used since, otherwise, one is only making progress in the dark, frequently relying on trial and error. In this thesis, we experiment with feedforward neural networks, trying to deconstruct the phenomenons we observe, and finding their root cause. We start by experimenting with a synthetic dataset. Using this toy problem, we find that the weights of trained networks show correlations that can be well-understood by the structure of the data samples themselves. This insight may be useful in areas such as Explainable Artificial Intelligence, to explain why a model behaves the way it does. We also find that the mere change of the activation function used in a layer may cause the nodes of the network to assume fundamentally different roles. This understanding may help to draw firm conclusions regarding the conditions in which Transfer Learning may be applied successfully. While testing with this problem, we also found that the initial configuration of weights of a network may, in some situations, ultimately determine the quality of the minimum (i.e., loss/accuracy) to which the networks converge, more so than what could be initially suspected. This observation motivated the remainder of our experiments. We continued our tests with the real-world datasets MNIST and HASYv2. We devised an initialization strategy, which we call the Dense sliced initialization, that works by combining the merits of a sparse initialization with those of a typical random initialization. Afterward, we found that the initial configuration of weights of a network “sticks” throughout training, suggesting that training does not imply substantial updates — instead, it is, to some extent, a fine-tuning process. We saw this by training networks marked with letters, and observing that those marks last throughout hundreds of epochs. Moreover, our results suggest that the small scale of the deviations caused by the training process is a fingerprint (i.e., a necessary condition) of training — as long as the training is successful, the marks remain visible. Based on these observations and our intuition for the reasons behind them, we developed what we call the Filter initialization strategy. It showed improvements in the training of the networks tested, but at the same time, it worsened their generalization. Understanding the root cause for these observations may prove to be valuable to devise new initialization methods that generalize better.É impossível ignorar os muitos avanços que aprendizagem automática, e em particular o seu método de eleição, aprendizagem profunda, têm proporcionado à nossa sociedade. No entanto, existe um sentimento crescente de que ao longo dos anos a área se tem vindo a tornar confusa e pouco clara, com alguns investigadores inclusive afirmando que aprendizagem automática se tornou na alquimia dos nossos tempos. Existe uma necessidade crescente de (voltar a) compreender em profundidade as ferramentas usadas, já que de outra forma o progresso acontece às escuras e, frequentemente, por tentativa e erro. Nesta dissertação conduzimos testes com redes neuronais artificiais dirigidas, com o objetivo de compreender os fenómenos subjacentes e encontrar as suas causas. Começamos por testar com um conjunto de dados sintético. Usando um problema amostra, descobrimos que a configuração dos pesos de redes treinadas evolui de forma a mostrar correlações que podem ser compreendidas atendendo à estrutura das amostras do próprio conjunto de dados. Esta observação poderá revelar-se útil em áreas como Inteligência Artificial Explicável, de forma a clarificar porque é que um dado modelo funciona de certa forma. Descobrimos também que a mera alteração da função de ativação de uma camada pode causar alterações organizacionais numa rede, a nível do papel que os nós nela desempenham. Este conhecimento poderá ser usado em áreas como Aprendizagem por Transferência, de forma a desenvolver critérios precisos sobre os limites/condições de aplicabilidade destas técnicas. Enquanto experimentávamos com este problema, descobrimos também que a configuração inicial dos pesos de uma rede pode condicionar totalmente a qualidade do mínimo para que ela converge, mais do que poderia ser esperado. Esta observação motiva os nossos restantes resultados. Continuamos testes com conjuntos de dados do mundo real, em particular com o MNIST e HASYv2. Desenvolvemos uma estratégia de inicialização, à qual chamamos de inicialização densa por fatias, que funciona combinado os méritos de uma inicialização esparsa com os de uma inicialização típica (densa). Descobrimos também que a configuração inicial dos pesos de uma rede persiste ao longo do seu treino, sugerindo que o processo de treino não causa atualizações bruscas dos pesos. Ao invés, é maioritariamente um processo de afinação. Visualizamos este efeito ao marcar as camadas de uma rede com letras do abecedário e observar que as marcas se mantêm por centenas de épocas de treino. Mais do que isso, a escala reduzida das atualizações dos pesos aparenta ser uma impressão digital (isto é, uma condição necessária) de treino com sucesso — enquanto o treino é bem sucedido, as marcas permanecem. Baseados neste conhecimento propusemos uma estratégia de inicialização inspirada em filtros. A estratégia mostrou bons resultados durante o treino das redes testadas, mas simultaneamente piorou a sua generalização. Perceber as razões por detrás deste fenómeno pode permitir desenvolver novas estratégias de inicialização que generalizem melhor que as atuais.Mestrado em Engenharia de Computadores e Telemátic