A lightweight data-driven spiking neuronal network model of Drosophila olfactory nervous system with dedicated hardware support

Data-driven spiking neuronal network (SNN) models enable in-silico analysis of the nervous system at the cellular and synaptic level. Therefore, they are a key tool for elucidating the information processing principles of the brain. While extensive research has focused on developing data-driven SNN models for mammalian brains, their complexity poses challenges in achieving precision. Network topology often relies on statistical inference, and the functions of specific brain regions and supporting neuronal activities remain unclear. Additionally, these models demand huge computing facilities and their simulation speed is considerably slower than real-time. Here, we propose a lightweight data-driven SNN model that strikes a balance between simplicity and reproducibility. The model is built using a qualitative modeling approach that can reproduce key dynamics of neuronal activity. We target the Drosophila olfactory nervous system, extracting its network topology from connectome data. The model was successfully implemented on a small entry-level field-programmable gate array and simulated the activity of a network in real-time. In addition, the model reproduced olfactory associative learning, the primary function of the olfactory system, and characteristic spiking activities of different neuron types. In sum, this paper propose a method for building data-driven SNN models from biological data. Our approach reproduces the function and neuronal activities of the nervous system and is lightweight, acceleratable with dedicated hardware, making it scalable to large-scale networks. Therefore, our approach is expected to play an important role in elucidating the brain's information processing at the cellular and synaptic level through an analysis-by-construction approach. In addition, it may be applicable to edge artificial intelligence systems in the future

Dynamic state and parameter estimation applied to neuromorphic systems

Neuroscientists often propose detailed computational models to probe the properties of the neural systems they study. With the advent of neuromorphic engineering, there is an increasing number of hardware electronic analogs of biological neural systems being proposed as well. However, for both biological and hardware systems, it is often difficult to estimate the parameters of the model so that they are meaningful to the experimental system under study, especially when these models involve a large number of states and parameters that cannot be simultaneously measured. We have developed a procedure to solve this problem in the context of interacting neural populations using a recently developed dynamic state and parameter estimation (DSPE) technique. This technique uses synchronization as a tool for dynamically coupling experimentally measured data to its corresponding model to determine its parameters and internal state variables. Typically experimental data are obtained from the biological neural system and the model is simulated in software; here we show that this technique is also efficient in validating proposed network models for neuromorphic spike-based very large-scale integration (VLSI) chips and that it is able to systematically extract network parameters such as synaptic weights, time constants, and other variables that are not accessible by direct observation. Our results suggest that this method can become a very useful tool for model-based identification and configuration of neuromorphic multichip VLSI systems

Hybrid Deep Learning Techniques for Securing Bioluminescent Interfaces in Internet of Bio Nano Things

The Internet of bio-nano things (IoBNT) is an emerging paradigm employing nanoscale (~1–100 nm) biological transceivers to collect in vivo signaling information from the human body and communicate it to healthcare providers over the Internet. Bio-nano-things (BNT) offer external actuation of in-body molecular communication (MC) for targeted drug delivery to otherwise inaccessible parts of the human tissue. BNTs are inter-connected using chemical diffusion channels, forming an in vivo bio-nano network, connected to an external ex vivo environment such as the Internet using bio-cyber interfaces. Bio-luminescent bio-cyber interfacing (BBI) has proven to be promising in realizing IoBNT systems due to their non-obtrusive and low-cost implementation. BBI security, however, is a key concern during practical implementation since Internet connectivity exposes the interfaces to external threat vectors, and accurate classification of anomalous BBI traffic patterns is required to offer mitigation. However, parameter complexity and underlying intricate correlations among BBI traffic characteristics limit the use of existing machine-learning (ML) based anomaly detection methods typically requiring hand-crafted feature designing. To this end, the present work investigates the employment of deep learning (DL) algorithms allowing dynamic and scalable feature engineering to discriminate between normal and anomalous BBI traffic. During extensive validation using singular and multi-dimensional models on the generated dataset, our hybrid convolutional and recurrent ensemble (CNN + LSTM) reported an accuracy of approximately ~93.51% over other deep and shallow structures. Furthermore, employing a hybrid DL network allowed automated extraction of normal as well as temporal features in BBI data, eliminating manual selection and crafting of input features for accurate prediction. Finally, we recommend deployment primitives of the extracted optimal classifier in conventional intrusion detection systems as well as evolving non-Von Neumann architectures for real-time anomaly detection

Harnessing Evolution in-Materio as an Unconventional Computing Resource

This thesis illustrates the use and development of physical conductive analogue systems for unconventional computing using the Evolution in-Materio (EiM) paradigm. EiM uses an Evolutionary Algorithm to configure and exploit a physical material (or medium) for computation. While EiM processors show promise, fundamental questions and scaling issues remain. Additionally, their development is hindered by slow manufacturing and physical experimentation. This work addressed these issues by implementing simulated models to speed up research efforts, followed by investigations of physically implemented novel in-materio devices. Initial work leveraged simulated conductive networks as single substrate ‘monolithic’ EiM processors, performing classification by formulating the system as an optimisation problem, solved using Differential Evolution. Different material properties and algorithm parameters were isolated and investigated; which explained the capabilities of configurable parameters and showed ideal nanomaterial choice depended upon problem complexity. Subsequently, drawing from concepts in the wider Machine Learning field, several enhancements to monolithic EiM processors were proposed and investigated. These ensured more efficient use of training data, better classification decision boundary placement, an independently optimised readout layer, and a smoother search space. Finally, scalability and performance issues were addressed by constructing in-Materio Neural Networks (iM-NNs), where several EiM processors were stacked in parallel and operated as physical realisations of Hidden Layer neurons. Greater flexibility in system implementation was achieved by re-using a single physical substrate recursively as several virtual neurons, but this sacrificed faster parallelised execution. These novel iM-NNs were first implemented using Simulated in-Materio neurons, and trained for classification as Extreme Learning Machines, which were found to outperform artificial networks of a similar size. Physical iM-NN were then implemented using a Raspberry Pi, custom Hardware Interface and Lambda Diode based Physical in-Materio neurons, which were trained successfully with neuroevolution. A more complex AutoEncoder structure was then proposed and implemented physically to perform dimensionality reduction on a handwritten digits dataset, outperforming both Principal Component Analysis and artificial AutoEncoders. This work presents an approach to exploit systems with interesting physical dynamics, and leverage them as a computational resource. Such systems could become low power, high speed, unconventional computing assets in the future

Efficient Learning Machines

Computer scienc

Temporal Data Analysis Using Reservoir Computing and Dynamic Memristors

Temporal data analysis including classification and forecasting is essential in a range of fields from finance to engineering. While static data are largely independent of each other, temporal data have a considerable correlation between the samples, which is important for temporal data analysis. Neural networks thus offer a more general and flexible approach since they do not depend on parameters of specific tasks but are driven only by the data. In particular, recurrent neural networks have gathered much attention since the temporal information captured by the recurrent connections improves the prediction performance. Recently, reservoir computing (RC), which evolves from recurrent neural networks, has been extensively studied for temporal data analysis as it can offer efficient temporal processing of recurrent neural networks with a low training cost. This dissertation presents a hardware implementation of the RC system using an emerging device - memristor, followed by a theoretical study on hierarchical architectures of the RC system. A RC hardware system based on dynamic tungsten oxide (WOx) memristors is first demonstrated. The internal short-term memory effects of the WOx memristors allow the memristor-based reservoir to nonlinearly map temporal inputs into reservoir states, where the projected features can be readily processed by a simple linear readout function. We use the system to experimentally demonstrate two standard benchmarking tasks: isolated spoken digit recognition with partial inputs and chaotic system forecasting. High classification accuracy of 99.2% is obtained for spoken digit recognition and autonomous chaotic time series forecasting has been demonstrated over the long term. We then investigate the influence of the hierarchical reservoir structure on the properties of the reservoir and the performance of the RC system. Analogous to deep neural networks, stacking sub-reservoirs in series is an efficient way to enhance the nonlinearity of data transformation to high-dimensional space and expand the diversity of temporal information captured by the reservoir. These deep reservoir systems offer better performance when compared to simply increasing the size of the reservoir or the number of sub-reservoirs. Low-frequency components are mainly captured by the sub-reservoirs in the later stages of the deep reservoir structure, similar to observations that more abstract information can be extracted by layers in the late stage of deep neural networks. When the total size of the reservoir is fixed, the tradeoff between the number of sub-reservoirs and the size of each sub-reservoir needs to be carefully considered, due to the degraded ability of the individual sub-reservoirs at small sizes. Improved performance of the deep reservoir structure alleviates the difficulty of implementing the RC system on hardware systems. Beyond temporal data classification and prediction, one of the interesting applications of temporal data analysis is inferring the neural connectivity patterns from the high-dimensional neural activity recording data. By computing the temporal correlation between the neural spikes, connections between the neurons can be inferred using statistics-based techniques, but it becomes increasingly computationally expensive for large scale neural systems. We propose a second-order memristor-based hardware system using the natively implemented spike-timing-dependent plasticity learning rule for neural connectivity inference. By incorporating biological features such as transmission delay to the neural networks, the proposed concept not only correctly infers the direct connections but also distinguishes direct connections from indirect connections. Effects of additional biophysical properties not considered in the simulation and challenges of experimental memristor implementation will be also discussed.PHDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/167995/1/moonjohn_1.pd

Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics

Three recent breakthroughs due to AI in arts and science serve as motivation: An award winning digital image, protein folding, fast matrix multiplication. Many recent developments in artificial neural networks, particularly deep learning (DL), applied and relevant to computational mechanics (solid, fluids, finite-element technology) are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. Hybrid methods combine traditional PDE discretizations with ML methods either (1) to help model complex nonlinear constitutive relations, (2) to nonlinearly reduce the model order for efficient simulation (turbulence), or (3) to accelerate the simulation by predicting certain components in the traditional integration methods. Here, methods (1) and (2) relied on Long-Short-Term Memory (LSTM) architecture, with method (3) relying on convolutional neural networks. Pure ML methods to solve (nonlinear) PDEs are represented by Physics-Informed Neural network (PINN) methods, which could be combined with attention mechanism to address discontinuous solutions. Both LSTM and attention architectures, together with modern and generalized classic optimizers to include stochasticity for DL networks, are extensively reviewed. Kernel machines, including Gaussian processes, are provided to sufficient depth for more advanced works such as shallow networks with infinite width. Not only addressing experts, readers are assumed familiar with computational mechanics, but not with DL, whose concepts and applications are built up from the basics, aiming at bringing first-time learners quickly to the forefront of research. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics, even in well-known references. Positioning and pointing control of a large-deformable beam is given as an example.Comment: 275 pages, 158 figures. Appeared online on 2023.03.01 at CMES-Computer Modeling in Engineering & Science

Algorithms and Software for Biological MP Modeling by Statistical and Optimization Techniques

I sistemi biologici sono gruppi di entit\ue0 biologiche (es. molecole ed organismi), che interagiscono producendo specifiche dinamiche. Questi sistemi sono solitamente caratterizzati da una elevata complessit\ue0 perch\ue8 coinvolgono un elevato numero di componenti con molte interconnessioni. La comprensione dei meccanismi che governano i sistemi biologici e la previsione dei loro comportamenti in condizioni normali e patologiche \ue8 una sfida cruciale della biologia dei sistemi (in inglese detta systems biology), un'area di ricerca al confine tra biologia, medicina, matematica ed informatica. In questa tesi i P sistemi metabolici, detti brevemente sistemi MP, sono stati utilizzati come modello discreto per l'analisi di dinamiche biologiche. Essi sono una classe deterministica dei P sistemi classici, che utilizzano regole di riscrittura per rappresentare le reazioni chimiche e "funzioni di regolazioni di flusso" per regolare la reattivit\ue0 di ciascuna reazione rispetto alla quantita' di sostanze presenti istantaneamente nel sistema. Dopo un excursus sulla letteratura relativa ad alcuni modelli convenzionali (come le equazioni differenziali ed i modelli stocastici proposti da Gillespie) e non-convenzionali (come i P sistemi ed i P sistemi metabolici), saranno presentati i risultati della mia ricerca. Essi riguardano tre argomenti principali: i) l'equivalenza tra sistemi MP e reti di Petri ibride funzionali, ii) le prospettive statistiche e di ottimizzazione nella generazione di sistemi MP a partire da dati sperimentali, iii) lo sviluppo di un laboratorio virtuale chiamato MetaPlab, un software Java basato sui sistemi MP. L'equivalenza tra i sistemi MP e le reti di Petri ibride funzionali \ue8 stata dimostrata per mezzo di due teoremi ed alcuni esperimenti al computer per il caso di studio del meccanismo regolativo del gene operone lac nella pathway glicolitica. Il secondo argomento di ricerca concerne nuovi approcci per la sintesi delle funzioni di regolazione di flusso. La regressione stepwise e le reti neurali sono state impiegate come approssimatori di funzioni, mentre algoritmi di ottimizzazione classici ed evolutivi (es. backpropagation, algoritmi genetici, particle swarm optimization ed algoritmi memetici) sono stati impiegati per l'addestramento dei modelli. Una completo workflow per l'analisi dei dati sperimentali \ue8 stato presentato. Esso gestisce ed indirizza l'intero processo di sintesi delle funzioni di regolazione, dalla preparazione dei dati alla selezione delle variabili, fino alla generazione dei modelli ed alla loro validazione. Le metodologie proposte sono state testate con successo tramite esperimenti al computer sui casi di studio dell'oscillatore mitotico negli embrioni anfibi e del non photochemical quenching (NPQ). L'ultimo tema di ricerca \ue8 infine piu' applicativo e riguarda la progettazione e lo sviluppo di una architettura Java basata su plugin e di una serie di plugin che consentono di automatizzare varie fasi del processo di modellazione con sistemi MP, come la simulazione di dinamiche, la determinazione dei flussi e la generazione delle funzioni di regolazione.Biological systems are groups of biological entities, (e.g., molecules and organisms), that interact together producing specific dynamics. These systems are usually characterized by a high complexity, since they involve a large number of components having many interconnections. Understanding biological system mechanisms, and predicting their behaviors in normal and pathological conditions is a crucial challenge in systems biology, which is a central research area on the border among biology, medicine, mathematics and computer science. In this thesis metabolic P systems, also called MP systems, have been employed as discrete modeling framework for the analysis of biological system dynamics. They are a deterministic class of P systems employing rewriting rules to represent chemical reactions and "flux regulation functions" to tune reactions reactivity according to the amount of substances present in the system. After an excursus on the literature about some conventional (i.e., differential equations, Gillespie's models) and unconventional (i.e., P systems and metabolic P systems) modeling frameworks, the results of my research are presented. They concern three research topics: i) equivalences between MP systems and hybrid functional Petri nets, ii) statistical and optimization perspectives in the generation of MP models from experimental data, iii) development of the virtual laboratory MetaPlab, a Java software based on MP systems. The equivalence between MP systems and hybrid functional Petri nets is proved by two theorems and some in silico experiments for the case study of the lac operon gene regulatory mechanism and glycolytic pathway. The second topic concerns new approaches to the synthesis of flux regulation functions. Stepwise linear regression and neural networks are employed as function approximators, and classical/evolutionary optimization algorithms (e.g., backpropagation, genetic algorithms, particle swarm optimization, memetic algorithms) as learning techniques. A complete pipeline for data analysis is also presented, which addresses the entire process of flux regulation function synthesis, from data preparation to feature selection, model generation and statistical validation. The proposed methodologies have been successfully tested by means of in silico experiments on the mitotic oscillator in early amphibian embryos and the non photochemical quenching (NPQ). The last research topic is more applicative, and pertains the design and development of a Java plugin architecture and several plugins which enable to automatize many tasks related to MP modeling, such as, dynamics computation, flux discovery, and regulation function synthesis

Using MapReduce Streaming for Distributed Life Simulation on the Cloud

Distributed software simulations are indispensable in the study of large-scale life models but often require the use of technically complex lower-level distributed computing frameworks, such as MPI. We propose to overcome the complexity challenge by applying the emerging MapReduce (MR) model to distributed life simulations and by running such simulations on the cloud. Technically, we design optimized MR streaming algorithms for discrete and continuous versions of Conway’s life according to a general MR streaming pattern. We chose life because it is simple enough as a testbed for MR’s applicability to a-life simulations and general enough to make our results applicable to various lattice-based a-life models. We implement and empirically evaluate our algorithms’ performance on Amazon’s Elastic MR cloud. Our experiments demonstrate that a single MR optimization technique called strip partitioning can reduce the execution time of continuous life simulations by 64%. To the best of our knowledge, we are the first to propose and evaluate MR streaming algorithms for lattice-based simulations. Our algorithms can serve as prototypes in the development of novel MR simulation algorithms for large-scale lattice-based a-life models.https://digitalcommons.chapman.edu/scs_books/1014/thumbnail.jp

Electronic Nanodevices

The start of high-volume production of field-effect transistors with a feature size below 100 nm at the end of the 20th century signaled the transition from microelectronics to nanoelectronics. Since then, downscaling in the semiconductor industry has continued until the recent development of sub-10 nm technologies. The new phenomena and issues as well as the technological challenges of the fabrication and manipulation at the nanoscale have spurred an intense theoretical and experimental research activity. New device structures, operating principles, materials, and measurement techniques have emerged, and new approaches to electronic transport and device modeling have become necessary. Examples are the introduction of vertical MOSFETs in addition to the planar ones to enable the multi-gate approach as well as the development of new tunneling, high-electron mobility, and single-electron devices. The search for new materials such as nanowires, nanotubes, and 2D materials for the transistor channel, dielectrics, and interconnects has been part of the process. New electronic devices, often consisting of nanoscale heterojunctions, have been developed for light emission, transmission, and detection in optoelectronic and photonic systems, as well for new chemical, biological, and environmental sensors. This Special Issue focuses on the design, fabrication, modeling, and demonstration of nanodevices for electronic, optoelectronic, and sensing applications