The effect of heterogeneity on decorrelation mechanisms in spiking neural networks: a neuromorphic-hardware study

High-level brain function such as memory, classification or reasoning can be realized by means of recurrent networks of simplified model neurons. Analog neuromorphic hardware constitutes a fast and energy efficient substrate for the implementation of such neural computing architectures in technical applications and neuroscientific research. The functional performance of neural networks is often critically dependent on the level of correlations in the neural activity. In finite networks, correlations are typically inevitable due to shared presynaptic input. Recent theoretical studies have shown that inhibitory feedback, abundant in biological neural networks, can actively suppress these shared-input correlations and thereby enable neurons to fire nearly independently. For networks of spiking neurons, the decorrelating effect of inhibitory feedback has so far been explicitly demonstrated only for homogeneous networks of neurons with linear sub-threshold dynamics. Theory, however, suggests that the effect is a general phenomenon, present in any system with sufficient inhibitory feedback, irrespective of the details of the network structure or the neuronal and synaptic properties. Here, we investigate the effect of network heterogeneity on correlations in sparse, random networks of inhibitory neurons with non-linear, conductance-based synapses. Emulations of these networks on the analog neuromorphic hardware system Spikey allow us to test the efficiency of decorrelation by inhibitory feedback in the presence of hardware-specific heterogeneities. The configurability of the hardware substrate enables us to modulate the extent of heterogeneity in a systematic manner. We selectively study the effects of shared input and recurrent connections on correlations in membrane potentials and spike trains. Our results confirm ...Comment: 20 pages, 10 figures, supplement

A Unifying review of linear gaussian models

Factor analysis, principal component analysis, mixtures of gaussian clusters, vector quantization, Kalman filter models, and hidden Markov models can all be unified as variations of unsupervised learning under a single basic generative model. This is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity. Through the use of other nonlinearities, we show how independent component analysis is also a variation of the same basic generative model.We show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term. We introduce a new model for static data, known as sensible principal component analysis, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models

単層カーボンナノチューブ/ポルフィリン-ポリ酸ランダムネットワークを用いたマテリアルリザバー演算素子 —次世代機械知能への新規アプローチ

In a layman’s term, computation is defined as the execution of a given instruction through a programmable algorithm. History has it that starting from the simplest calculator to the sophisticated von Neumann machine, the above definition has been followed without a flaw. Logical operations for which a human takes a minute long to solve, is a matter of fraction of seconds for these gadgets. But contrastingly, when it comes to critical and analytical thinking that requires learning through observation like the human brain, these powerful machines falter and lag behind. Thus, inspired from the brain’s neural circuit, software models of neural networks (NN) integrated with high-speed supercomputers were developed as an alternative tool to implement machine intelligent tasks of function optimization, pattern, and voice recognition. But as device downscaling and transistor performance approaches the constant regime of Moore’s law due to high CMOS fabrication cost and large tunneling energy loss, training these algorithms over multiple hidden layers is turning out to be a grave concern for future applications. As a result, the interplay between faster performance and low computational power requirement for complex tasks deems highly disproportional. Therefore, alternative in terms of both NN models and conventional Neumann architecture needs to be addressed in today’s age for next-generation machine intelligence systems. Fortunately, through extensive research and studies, unconventional computing using a reservoir based neural network platform, called in-materio reservoir computing (RC) has come to the rescue. In-maerio RC uses physical, biological, chemical, cellular automata and other inanimate dynamical systems as a source of non-linear high dimensional spatio-temporal information processing unit to construct a specific target task. RC not only has a three-layer simplified neural architectural layer, but also imposes a cheap, fast, and simplified optimization of only the readout weights with machine intelligent regression algorithm to construct the supervised objective target via a weighted linear combination of the readouts. Thus, utilizing this idea, herein in this work we report such an in-materio RC with a dynamical random network of single walled carbon nanotube/porphyrin-polyoxometalate (SWNT/Por-POM) device. We begin with Chapter 1, which deals with the introduction covering the literature of ANN evolution and the shortcomings of von Neumann architecture and training models of these ANN, which leads us to adopt the in-materio RC architecture. We design the problem statement focused on extending the theoretical RC model of previously suggested SWNT/POM network to an experimental one and present the objective of fabricating a random network based on nanomaterials as they closely resemble the network structure of the brain. Finally, we conclude by stating the scope of this research work aiming towards validating the non-linear high dimensional reservoir property SWNT/Por-POM holds for it to explicitly demonstrate the RC benchmark tasks of optimization and classification. Chapter 2 describes the methodology including the chemical repository required for the facile synthesis of the material. The synthesis part is divided broadly into SWNT purification and then its dispersion with Por-POM to form the desired complex. It is then followed up with the microelectrode array fabrication and the consequent wet-transfer thin film deposition to give the ultimate reservoir architecture of input-output control read pads with SWNT/Por-POM reservoir. Finally we give a briefing of AFM, UV-Vis spectroscopy, FE-SEM characterization techniques of SWNT/Por-POM complex along with the electrical set-up interfaced with software algorithm to demonstrate the RC approach of in-materio machine intelligence. In Chapter 3, we study the current dynamics as a function of voltage and time and validate the non-linear information processing ability intrinsic to the device. The study reveals that the negative differential resistance (NDR) arising from redox nature of Por-POM results in oscillating random noise outputs giving rise to 1/f brain-like spatio-temporal information. We compute the memory capacity (MC) and prove that the device exhibits echo state property of fading memory, but remembers very little of the past information. The low MC and high non-linearity allowed us to choose mostly non-linear tasks of waveform generation, Boolean logic optimization and one-hot vector binary object classification as the RC benchmark. The Chapter 4 relates to the waveform generation task. Utilizing the high dimensional voltage readouts of varying amplitude, phase and higher harmonic frequencies, relative to input sine wave, a regression optimization was performed towards constructing cosine, triangular, square and sawtooth waves resulting in a high accuracy of around 95%. The task complexity of function optimization was further enhanced in Chapter 5 where two inputs were used to construct Boolean logic functions of OR, AND, XOR, NOR, NAND and XNOR. Similar to the waveform, accuracy over 95% could be achieved due to the presence of NDR nonlinearity. Furthermore, the device was also tested for classification problem in Chapter 6. Here we showed an off-line binary classification of four object toys; hedgehog, dog, block and bus, using the grasped tactile information of these objects as inputs obtained from the Toyota Human Support Robot. A one-ridge regression analysis to fit the hot vector supervised target was used to optimize the output weights for predicting the correct outcome. All the objects were successfully classified owing to the 1/f information processing factor. Lastly, we conclude the section in Chapter 7 with the future scope of extending the idea to fabricate a 3-D model of the same material as it opens up opportunity for higher memory capacity fruitful for future benchmark tasks of time-series prediction. Overall, our research marks a step stone in utilizing SWNT/Por-POM as the in-materio RC for the very first time thereby making it a desirable candidate for next-generation machine intelligence.九州工業大学博士学位論文 学位記番号：生工博甲第425号 学位授与年月日：令和3年12月27日1 Introduction and Literature review|2 Methodology|3 Reservoir dynamics emerging from an incidental structure of single-walled carbon nanotube/porphyrin-polyoxometalate complex|4 Fourier transform waveforms via in-materio reservoir computing from single-walled carbon nanotube/porphyrin-polyoxometalate complex|5 Room temperature demonstration of in-materio reservoir computing for optimizing Boolean function with single-walled carbon nanotube/porphyrin-polyoxometalate composite|6 Binary object classification with tactile sensory input information of via single-walled carbon nanotube/porphyrin-polyoxometalate network as in-materio reservoir computing|7 Future scope and Conclusion九州工業大学令和3年

Study, Design and Fabrication of an Analogue VLSI Ormia-Ochracea-Inspired Delay Magnification System

This Thesis entails the development of a low-power delay magnification system inspired by the mechanical structure of the ear of the parasitoid fly Ormia Ochracea (O2). The proposed system is suitable as a preprocessing unit for binaural sound localization processors equipped with miniature acoustic sensors. The core of the Thesis involves the study of a delay magnification system based on the O2 sound localization mechanism and the design and testing of a low-power analog integrated circuit based on a proposed, novel delay magnification system inspired by Ormia Ochracea. The study of the delay magnification system based on the O2 sound localization mechanism is divided into two main parts. The first part studies in detail the delay magnification mechanism of the O2 ears. This study sheds light and tries to comprehend what mechanical parameters of the O2 ears are involved in the delay magnification process and how these parameters contribute to the magnification of the delay. The study presents the signal-flow-graph of the O2 system which can be used as a generic delay magnification model for the O2 ears. We also explore the effects of the tuning of the O2 system parameters on the output interaural time difference (ITD). Inspired by the study of the O2 system, in the second part of our study, we modify the O2 system using simpler building blocks and structure which can provide a delay magnification comparable to the original O2 system. We present a new binaural sound localization system suitable for small ITDs which utilizes the new modified O2 system, cochlea filter banks, cross-correlograms and our re-mapping algorithm and show that it can be used to encode very small input delay values that could not be resolved by means of a conventional binaural processor based on the Jeffress’s coincidence detection model. We evaluate the sound localization performance of our new binaural sound localization system for a single sound source and a sound source in the presence of a competing sound source scenario through detailed simulation. The performance of the proposed system is also explored in the presence of filter bandwidth variation and cochlea filter mismatch. After the study of the O2 delay magnification system, we present an analog VLSI chip which morphs the O2 delay magnification system. To determine what topology is the best morphing platform for the O2 system, we present the design and comparative performance of the O2 system when log-domain and gm-C second order weak-inversion filters are employed. The design of the proposed low-power modified O2 system circuit based on translinear loops is detailed. Its performance is evaluated through detailed simulation. Subsequently the Thesis proceeds with the design, fabrication and testing of the new chip based on the modified O2 circuit. The synthesis and testing of the proposed circuit using 0.35μm AMS CMOS process technology parameters is discussed. Detailed measured results confirm the delay magnification ability of the modified O2 circuit and its compliance with theoretical analysis explained earlier in the Thesis. The fabricated system is tuned to operate in the 100Hz to 1kHz frequency range, is able to achieve a delay gain of approximately 3.5 to 9.5 when the input (physical) delay ranges from 0μs to 20μs, and consumes 13.1μW with a 2 V power supply

Bio-inspired electronics for micropower vision processing

Vision processing is a topic traditionally associated with neurobiology; known to encode, process and interpret visual data most effectively. For example, the human retina; an exquisite sheet of neurobiological wetware, is amongst the most powerful and efficient vision processors known to mankind. With improving integrated technologies, this has generated considerable research interest in the microelectronics community in a quest to develop effective, efficient and robust vision processing hardware with real-time capability. This thesis describes the design of a novel biologically-inspired hybrid analogue/digital vision chip ORASIS1 for centroiding, sizing and counting of enclosed objects. This chip is the first two-dimensional silicon retina capable of centroiding and sizing multiple objects2 in true parallel fashion. Based on a novel distributed architecture, this system achieves ultra-fast and ultra-low power operation in comparison to conventional techniques. Although specifically applied to centroid detection, the generalised architecture in fact presents a new biologically-inspired processing paradigm entitled: distributed asynchronous mixed-signal logic processing. This is applicable to vision and sensory processing applications in general that require processing of large numbers of parallel inputs, normally presenting a computational bottleneck. Apart from the distributed architecture, the specific centroiding algorithm and vision chip other original contributions include: an ultra-low power tunable edge-detection circuit, an adjustable threshold local/global smoothing network and an ON/OFF-adaptive spiking photoreceptor circuit. Finally, a concise yet comprehensive overview of photodiode design methodology is provided for standard CMOS technologies. This aims to form a basic reference from an engineering perspective, bridging together theory with measured results. Furthermore, an approximate photodiode expression is presented, aiming to provide vision chip designers with a basic tool for pre-fabrication calculations