A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks

Spiking neural networks (SNNs) have demonstrated excellent capabilities in various intelligent scenarios. Most existing methods for training SNNs are based on the concept of synaptic plasticity; however, learning in the realistic brain also utilizes intrinsic non-synaptic mechanisms of neurons. The spike threshold of biological neurons is a critical intrinsic neuronal feature that exhibits rich dynamics on a millisecond timescale and has been proposed as an underlying mechanism that facilitates neural information processing. In this study, we develop a novel synergistic learning approach that simultaneously trains synaptic weights and spike thresholds in SNNs. SNNs trained with synapse-threshold synergistic learning (STL-SNNs) achieve significantly higher accuracies on various static and neuromorphic datasets than SNNs trained with two single-learning models of the synaptic learning (SL) and the threshold learning (TL). During training, the synergistic learning approach optimizes neural thresholds, providing the network with stable signal transmission via appropriate firing rates. Further analysis indicates that STL-SNNs are robust to noisy data and exhibit low energy consumption for deep network structures. Additionally, the performance of STL-SNN can be further improved by introducing a generalized joint decision framework (JDF). Overall, our findings indicate that biologically plausible synergies between synaptic and intrinsic non-synaptic mechanisms may provide a promising approach for developing highly efficient SNN learning methods.Comment: 13 pages, 9 figures, submitted for publicatio

An Adaptive Locally Connected Neuron Model: Focusing Neuron

This paper presents a new artificial neuron model capable of learning its receptive field in the topological domain of inputs. The model provides adaptive and differentiable local connectivity (plasticity) applicable to any domain. It requires no other tool than the backpropagation algorithm to learn its parameters which control the receptive field locations and apertures. This research explores whether this ability makes the neuron focus on informative inputs and yields any advantage over fully connected neurons. The experiments include tests of focusing neuron networks of one or two hidden layers on synthetic and well-known image recognition data sets. The results demonstrated that the focusing neurons can move their receptive fields towards more informative inputs. In the simple two-hidden layer networks, the focusing layers outperformed the dense layers in the classification of the 2D spatial data sets. Moreover, the focusing networks performed better than the dense networks even when 70$\%$ of the weights were pruned. The tests on convolutional networks revealed that using focusing layers instead of dense layers for the classification of convolutional features may work better in some data sets.Comment: 45 pages, a national patent filed, submitted to Turkish Patent Office, No: -2017/17601, Date: 09.11.201

A Survey on Reservoir Computing and its Interdisciplinary Applications Beyond Traditional Machine Learning

Reservoir computing (RC), first applied to temporal signal processing, is a recurrent neural network in which neurons are randomly connected. Once initialized, the connection strengths remain unchanged. Such a simple structure turns RC into a non-linear dynamical system that maps low-dimensional inputs into a high-dimensional space. The model's rich dynamics, linear separability, and memory capacity then enable a simple linear readout to generate adequate responses for various applications. RC spans areas far beyond machine learning, since it has been shown that the complex dynamics can be realized in various physical hardware implementations and biological devices. This yields greater flexibility and shorter computation time. Moreover, the neuronal responses triggered by the model's dynamics shed light on understanding brain mechanisms that also exploit similar dynamical processes. While the literature on RC is vast and fragmented, here we conduct a unified review of RC's recent developments from machine learning to physics, biology, and neuroscience. We first review the early RC models, and then survey the state-of-the-art models and their applications. We further introduce studies on modeling the brain's mechanisms by RC. Finally, we offer new perspectives on RC development, including reservoir design, coding frameworks unification, physical RC implementations, and interaction between RC, cognitive neuroscience and evolution.Comment: 51 pages, 19 figures, IEEE Acces

Beyond spiking networks: the computational advantages of dendritic amplification and input segregation

The brain can efficiently learn a wide range of tasks, motivating the search for biologically inspired learning rules for improving current artificial intelligence technology. Most biological models are composed of point neurons, and cannot achieve the state-of-the-art performances in machine learning. Recent works have proposed that segregation of dendritic input (neurons receive sensory information and higher-order feedback in segregated compartments) and generation of high-frequency bursts of spikes would support error backpropagation in biological neurons. However, these approaches require propagating errors with a fine spatio-temporal structure to the neurons, which is unlikely to be feasible in a biological network. To relax this assumption, we suggest that bursts and dendritic input segregation provide a natural support for biologically plausible target-based learning, which does not require error propagation. We propose a pyramidal neuron model composed of three separated compartments. A coincidence mechanism between the basal and the apical compartments allows for generating high-frequency bursts of spikes. This architecture allows for a burst-dependent learning rule, based on the comparison between the target bursting activity triggered by the teaching signal and the one caused by the recurrent connections, providing the support for target-based learning. We show that this framework can be used to efficiently solve spatio-temporal tasks, such as the store and recall of 3D trajectories. Finally, we suggest that this neuronal architecture naturally allows for orchestrating ``hierarchical imitation learning'', enabling the decomposition of challenging long-horizon decision-making tasks into simpler subtasks. This can be implemented in a two-level network, where the high-network acts as a ``manager'' and produces the contextual signal for the low-network, the ``worker''.Comment: arXiv admin note: substantial text overlap with arXiv:2201.1171

Bidirectional long short-term memory network for proto-object representation

Researchers have developed many visual saliency models in order to advance the technology in computer vision. Neural networks, Convolution Neural Networks (CNNs) in particular, have successfully differentiate objects in images through feature extraction. Meanwhile, Cummings et al. has proposed a proto-object image saliency (POIS) model that shows perceptual objects or shapes can be modelled through the bottom-up saliency algorithm. Inspired from their work, this research is aimed to explore the imbedding features in the proto-object representations and utilizing artificial neural networks (ANN) to capture and predict the saliency output of POIS. A combination of CNN and a bi-directional long short-term memory (BLSTM) neural network is proposed for this saliency model as a machine learning alternative to the border ownership and grouping mechanism in POIS. As ANNs become more efficient in performing visual saliency tasks, the result of this work would extend their application in computer vision through successful implementation for proto-object based saliency

Enhancing Neuromorphic Computing with Advanced Spiking Neural Network Architectures

This dissertation proposes ways to address current limitations of neuromorphic computing to create energy-efficient and adaptable systems for AI applications. It does so by designing novel spiking neural networks architectures that improve their performance. Specifically, the two proposed architectures address the issues of training complexity, hyperparameter selection, computational flexibility, and scarcity of neuromorphic training data. The first architecture uses auxiliary learning to improve training performance and data usage, while the second architecture leverages neuromodulation capability of spiking neurons to improve multitasking classification performance. The proposed architectures are tested on Intel\u27s Loihi2 neuromorphic chip using several neuromorphic datasets, such as NMIST, DVSCIFAR10, and DVS128-Gesture. The presented results demonstrate potential of the proposed architectures but also reveal some of their limitations which are proposed as future research

Mejora de computación neuromórfica con arquitecturas avanzadas de redes neuronales por impulsos

La computación neuromórfica (NC, del inglés neuromorphic computing) pretende revolucionar el campo de la inteligencia artificial. Implica diseñar e implementar sistemas electrónicos que simulen el comportamiento de las neuronas biológicas utilizando hardware especializado, como matrices de puertas programables en campo (FPGA, del ingl´es field-programmable gate array) o chips neuromórficos dedicados [1, 2]. NC está diseñado para ser altamente eficiente, optimizado para bajo consumo de energía y alto paralelismo [3]. Estos sistemas son adaptables a entornos cambiantes y pueden aprender durante la operación, lo que los hace muy adecuados para resolver problemas dinámicos e impredecibles [4]. Sin embargo, el uso de NC para resolver problemas de la vida real actualmente está limitado porque el rendimiento de las redes neuronales por impulsos (SNN), las redes neuronales empleadas en NC, no es tan alta como el de los sistemas de computación tradicionales, como los alcanzados en dispositivos de aprendizaje profundo especializado, en términos de precisión y velocidad de aprendizaje [5, 6]. Varias razones contribuyen a la brecha de rendimiento: los SNN son más difíciles de entrenar debido a que necesitan algoritmos de entrenamiento especializados [7, 8]; son más sensibles a hiperparámetros, ya que son sistemas dinámicos con interacciones complejas [9], requieren conjuntos de datos especializados (datos neuromórficos) que actualmente son escasos y de tamaño limitado [10], y el rango de funciones que los SNN pueden aproximar es más limitado en comparación con las redes neuronales artificiales (ANN) tradicionales [11]. Antes de que NC pueda tener un impacto más significativo en la IA y la tecnología informática, es necesario abordar estos desafíos relacionados con los SNN.This dissertation addresses current limitations of neuromorphic computing to create energy-efficient and adaptable artificial intelligence systems. It focuses on increasing utilization of neuromorphic computing by designing novel architectures that improve the performance of the spiking neural networks. Specifically, the architectures address the issues of training complexity, hyperparameter selection, computational flexibility, and scarcity of training data. The first proposed architecture utilizes auxiliary learning to improve training performance and data usage, while the second architecture leverages neuromodulation capability of spiking neurons to improve multitasking classification performance. The proposed architectures are tested on the Intel’s Loihi2 neuromorphic computer using several neuromorphic data sets, such as NMIST, DVSCIFAR10, and DVS128-Gesture. Results presented in this dissertation demonstrate the potential of the proposed architectures, but also reveal some limitations that are proposed as future work

AI of Brain and Cognitive Sciences: From the Perspective of First Principles

Nowadays, we have witnessed the great success of AI in various applications, including image classification, game playing, protein structure analysis, language translation, and content generation. Despite these powerful applications, there are still many tasks in our daily life that are rather simple to humans but pose great challenges to AI. These include image and language understanding, few-shot learning, abstract concepts, and low-energy cost computing. Thus, learning from the brain is still a promising way that can shed light on the development of next-generation AI. The brain is arguably the only known intelligent machine in the universe, which is the product of evolution for animals surviving in the natural environment. At the behavior level, psychology and cognitive sciences have demonstrated that human and animal brains can execute very intelligent high-level cognitive functions. At the structure level, cognitive and computational neurosciences have unveiled that the brain has extremely complicated but elegant network forms to support its functions. Over years, people are gathering knowledge about the structure and functions of the brain, and this process is accelerating recently along with the initiation of giant brain projects worldwide. Here, we argue that the general principles of brain functions are the most valuable things to inspire the development of AI. These general principles are the standard rules of the brain extracting, representing, manipulating, and retrieving information, and here we call them the first principles of the brain. This paper collects six such first principles. They are attractor network, criticality, random network, sparse coding, relational memory, and perceptual learning. On each topic, we review its biological background, fundamental property, potential application to AI, and future development.Comment: 59 pages, 5 figures, review articl

Feed-Forward Optimization With Delayed Feedback for Neural Networks

Backpropagation has long been criticized for being biologically implausible, relying on concepts that are not viable in natural learning processes. This paper proposes an alternative approach to solve two core issues, i.e., weight transport and update locking, for biological plausibility and computational efficiency. We introduce Feed-Forward with delayed Feedback (F$^3$), which improves upon prior work by utilizing delayed error information as a sample-wise scaling factor to approximate gradients more accurately. We find that F$^3$ reduces the gap in predictive performance between biologically plausible training algorithms and backpropagation by up to 96%. This demonstrates the applicability of biologically plausible training and opens up promising new avenues for low-energy training and parallelization