Adaptive Synaptogenesis Constructs Neural Codes That Benefit Discrimination

Intelligent organisms face a variety of tasks requiring the acquisition of expertise within a specific domain, including the ability to discriminate between a large number of similar patterns. From an energy-efficiency perspective, effective discrimination requires a prudent allocation of neural resources with more frequent patterns and their variants being represented with greater precision. In this work, we demonstrate a biologically plausible means of constructing a single-layer neural network that adaptively (i.e., without supervision) meets this criterion. Specifically, the adaptive algorithm includes synaptogenesis, synaptic shedding, and bi-directional synaptic weight modification to produce a network with outputs (i.e. neural codes) that represent input patterns proportional to the frequency of related patterns. In addition to pattern frequency, the correlational structure of the input environment also affects allocation of neural resources. The combined synaptic modification mechanisms provide an explanation of neuron allocation in the case of self-taught experts

Identifying Network Correlates of Memory Consolidation

Neuronal spiking activity carries information about our experiences in the waking world but exactly how the brain can quickly and efficiently encode sensory information into a useful neural code and then subsequently consolidate that information into memory remains a mystery. While neuronal networks are known to play a vital role in these processes, detangling the properties of network activity from the complex spiking dynamics observed is a formidable challenge, requiring collaborations across scientific disciplines. In this work, I outline my contributions in computational modeling and data analysis toward understanding how network dynamics facilitate memory consolidation. For experimental perspective, I investigate hippocampal recordings of mice that are subjected to contextual fear conditioning and subsequently undergo sleep-dependent fear memory consolidation. First, I outline the development of a functional connectivity algorithm which rapidly and robustly assesses network structure based on neuronal spike timing. I show that the relative stability of these functional networks can be used to identify global network dynamics, revealing that an increase in functional network stability correlates with successful fear memory consolidation in vivo. Using an attractor-based model to simulate memory encoding and consolidation, I go on to show that dynamics associated with a second-order phase transition, at a critical point in phase-space, are necessary for recruiting additional neurons into network dynamics associated with memory consolidation. I show that successful consolidation subsequently shifts dynamics away from a critical point and towards sub-critical dynamics. Investigations of in vivo spiking dynamics likewise revealed that hippocampal dynamics during non-rapid-eye-movement (NREM) sleep show features of being near a critical point and that fear memory consolidation leads to a shift in dynamics. Finally, I investigate the role of NREM sleep in facilitating memory consolidation using a conductance-based model of neuronal activity that can easily switch between modes of activity loosely representing waking and NREM sleep. Analysis of model simulations revealed that oscillations associated with NREM sleep promote a phase-based coding of information; neurons with high firing rates during periods of wake lead spiking activity during NREM oscillations. I show that when phase-coding is active in both simulations and in vivo, synaptic plasticity selectively strengthens the input to neurons firing late in the oscillation while simultaneously reducing input to neurons firing early in the oscillation. The effect is a net homogenization of firing rates observed in multiple other studies, and subsequently leads to recruitment of new neurons into a memory engram and information transfer from fast firing neurons to slow firing neurons. Taken together, my work outlines important, newly-discovered features of neuronal network dynamics related to memory encoding and consolidation: networks near criticality promote recruitment of additional neurons into stable firing patterns through NREM-associated oscillations and subsequently consolidates information into memories through phase-based coding.PHDBiophysicsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/162991/1/qmskill_1.pd

Wind Power Forecasting Based on Echo State Networks and Long Short-Term Memory

Wind power generation has presented an important development around the world. However, its integration into electrical systems presents numerous challenges due to the variable nature of the wind. Therefore, to maintain an economical and reliable electricity supply, it is necessary to accurately predict wind generation. The Wind Power Prediction Tool (WPPT) has been proposed to solve this task using the power curve associated with a wind farm. Recurrent Neural Networks (RNNs) model complex non-linear relationships without requiring explicit mathematical expressions that relate the variables involved. In particular, two types of RNN, Long Short-Term Memory (LSTM) and Echo State Network (ESN), have shown good results in time series forecasting. In this work, we present an LSTM+ESN architecture that combines the characteristics of both networks. An architecture similar to an ESN is proposed, but using LSTM blocks as units in the hidden layer. The training process of this network has two key stages: (i) the hidden layer is trained with a descending gradient method online using one epoch; (ii) the output layer is adjusted with a regularized regression. In particular, the case is proposed where Step (i) is used as a target for the input signal, in order to extract characteristics automatically as the autoencoder approach; and in the second stage (ii), a quantile regression is used in order to obtain a robust estimate of the expected target. The experimental results show that LSTM+ESN using the autoencoder and quantile regression outperforms the WPPT model in all global metrics used

Analysing and forecasting tourism demand in Vietnam with artificial neural networks

Mestrado APNORVietnam has experienced a tourism boom over the last decade with more than 18 million international tourists in 2019, compared to 1.5 million twenty-five years ago. Tourist spending has translated into rising employment and income for the tourism sector, making it the key driver to the socio-economic development of the country. Facing the COVID-19 pandemic, Vietnam´s tourism has suffered extreme economic losses. However, the number of international tourists is expected to reach the pre-pandemic levels in the next few years after the COVID-19 pandemic subsides. Forecasting tourism demand plays an essential role in predicting future economic development. Accurate predictions of tourism volume would facilitate decision-makers and managers to optimize resource allocation as well as to balance environmental and economic aspects. Various methods to predict tourism demand have been introduced over the years. One of the most prominent approaches is Artificial Neural Network (ANN) thanks to its capability to handle highly volatile and non-linear data. Given the significance of tourism to the economy, a precise forecast of tourism demand would help to foresee the potential economic growth of Vietnam. First, the research aims to analyse Vietnam´s tourism sector with a special focus on international tourists. Next, several ANN architectures are experimented with the datasets from 2008 to 2020, to predict the monthly number of international tourists traveling to Vietnam including COVID-19 lockdown periods. The results showed that with the correct selection of ANN architectures and data from the previous 12 months, the best ANN models can forecast the number of international tourists for next month with a MAPE between 7.9% and 9.2%. As the method proves its forecasting accuracy, it would serve as a valuable tool for Vietnam´s policymakers and firm managers to make better investment and strategic decisions to promote tourism after the COVID-19 situation.O Vietname conheceu um boom turístico na última década com mais de 18 milhões de turistas internacionais em 2019, em comparação com 1,5 milhões há vinte e cinco anos. As despesas turísticas traduziram-se num aumento do emprego e de receitas no sector do turismo, tornando-o no principal motor do desenvolvimento socioeconómico do país. Perante a pandemia da COVID-19, o turismo no Vietname sofreu perdas económicas extremas. Porém, espera-se que o número de turistas internacionais, pós pandemia da COVID-19, atinja os níveis pré-pandémicos nos próximos anos. A previsão da procura turística desempenha um papel essencial na previsão do desenvolvimento económico futuro. Previsões precisas facilitariam os decisores e gestores a otimizar a afetação de recursos, bem como o equilíbrio entre os aspetos ambientais e económicos. Vários métodos para prever a procura turística têm sido introduzidos ao longo dos anos. Uma das abordagens mais proeminentes assenta na metodologia das Redes Neuronais Artificiais (ANN) dada a sua capacidade de lidar com dados voláteis e não lineares. Dada a importância do turismo para a economia, uma previsão precisa da procura turística ajudaria a prever o crescimento económico potencial do Vietname. Em primeiro lugar, a investigação tem por objetivo analisar o sector turístico do Vietname com especial incidência nos turistas internacionais. Em seguida, várias arquiteturas de ANN são experimentadas com um conjunto de dados de 2008 a 2020, para prever o número mensal de turistas internacionais que se deslocam ao Vietname, incluindo os períodos de confinamento relacionados com a COVID-19. Os resultados mostraram, com a correta seleção de arquiteturas ANN e dados dos 12 meses anteriores, os melhores modelos ANN podem prever o número de turistas internacionais para o próximo mês com uma MAPE entre 7,9% e 9,2%. Como o método evidenciou a sua precisão de previsão, o mesmo pode servir como uma ferramenta valiosa para os decisores políticos e gestores de empresas do Vietname, pois irá permitir fazer melhores investimentos e tomarem decisões estratégicas para promover o turismo pós situação da COVID-19

The evolutionary emergence of neural organisation in computational models of primitive organisms

Over the decades, the question why did neural organisation emerge in the way that it did? has proved to be massively elusive. Whilst much of the literature paints a picture of common ancestry the idea that a species at the root of the tree of nervous system evolution spawned numerous descendants the actual evolutionary forces responsible for such changes, major transitions or otherwise, have been less clear. The view presented in this thesis is that via interactions with the environment, neural organisation has emerged in concert with the constraints enforced by body plan morphology and a need to process information eciently and robustly. Whilst these factors are two smaller parts of a much greater whole, their impact during the evolutionary process cannot be ignored, for they are fundamentally signicant. Thus computer simulations have been developed to provide insight into how neural organisation of an articial agent should emerge given the constraints of its body morphology, its symmetry, feedback from the environment, and a loss of energy. The first major finding is that much of the computational process of the nervous system can be ooaded to the body morphology, which has a commensurate bearing on neural architecture, neural dynamics and motor symmetry. The second major finding is that sensory feedback strengthens the dynamic coupling between the neural system and the body plan morphology, resulting in minimal neural circuitry yet more ecient agent behaviour. The third major finding is that under the constraint of energy loss, neural circuitry again emerges to be minimalistic. Throughout, an emphasis is placed on the coupling between the nervous system and body plan morphology which are known in the literature to be tightly integrated; accordingly, both are considered on equal footings

Feedforward deep architectures for classification and synthesis

Cette thèse par article présente plusieurs contributions au domaine de l'apprentissage de représentations profondes, avec des applications aux problèmes de classification et de synthèse d'images naturelles. Plus spécifiquement, cette thèse présente plusieurs nouvelles techniques pour la construction et l'entraînment de réseaux neuronaux profonds, ainsi q'une étude empirique de la technique de «dropout», une des approches de régularisation les plus populaires des dernières années. Le premier article présente une nouvelle fonction d'activation linéaire par morceau, appellée «maxout», qui permet à chaque unité cachée d'un réseau de neurones d'apprendre sa propre fonction d'activation convexe. Nous démontrons une performance améliorée sur plusieurs tâches d'évaluation du domaine de reconnaissance d'objets, et nous examinons empiriquement les sources de cette amélioration, y compris une meilleure synergie avec la méthode de régularisation «dropout» récemment proposée. Le second article poursuit l'examen de la technique «dropout». Nous nous concentrons sur les réseaux avec fonctions d'activation rectifiées linéaires (ReLU) et répondons empiriquement à plusieurs questions concernant l'efficacité remarquable de «dropout» en tant que régularisateur, incluant les questions portant sur la méthode rapide de rééchelonnement au temps de l´évaluation et la moyenne géometrique que cette méthode approxime, l'interprétation d'ensemble comparée aux ensembles traditionnels, et l'importance d'employer des critères similaires au «bagging» pour l'optimisation. Le troisième article s'intéresse à un problème pratique de l'application à l'échelle industrielle de réseaux neuronaux profonds au problème de reconnaissance d'objets avec plusieurs etiquettes, nommément l'amélioration de la capacité d'un modèle à discriminer entre des étiquettes fréquemment confondues. Nous résolvons le problème en employant la prédiction du réseau des sous-composantes dédiées à chaque sous-ensemble de la partition. Finalement, le quatrième article s'attaque au problème de l'entraînment de modèles génératifs adversariaux (GAN) récemment proposé. Nous présentons une procédure d'entraînment améliorée employant un auto-encodeur débruitant, entraîné dans un espace caractéristiques abstrait appris par le discriminateur, pour guider le générateur à apprendre un encodage qui s'aligne de plus près aux données. Nous évaluons le modèle avec le score «Inception» récemment proposé.This thesis by articles makes several contributions to the field of deep learning, with applications to both classification and synthesis of natural images. Specifically, we introduce several new techniques for the construction and training of deep feedforward networks, and present an empirical investigation into dropout, one of the most popular regularization strategies of the last several years. In the first article, we present a novel piece-wise linear parameterization of neural networks, maxout, which allows each hidden unit of a neural network to effectively learn its own convex activation function. We demonstrate improvements on several object recognition benchmarks, and empirically investigate the source of these improvements, including an improved synergy with the recently proposed dropout regularization method. In the second article, we further interrogate the dropout algorithm in particular. Focusing on networks of the popular rectified linear units (ReLU), we empirically examine several questions regarding dropout’s remarkable effectiveness as a regularizer, including questions surrounding the fast test-time rescaling trick and the geometric mean it approximates, interpretations as an ensemble as compared with traditional ensembles, and the importance of using a bagging-like criterion for optimization. In the third article, we address a practical problem in industrial-scale application of deep networks for multi-label object recognition, namely improving an existing model’s ability to discriminate between frequently confused classes. We accomplish this by using the network’s own predictions to inform a partitioning of the label space, and augment the network with dedicated discriminative capacity addressing each of the partitions. Finally, in the fourth article, we tackle the problem of fitting implicit generative models of open domain collections of natural images using the recently introduced Generative Adversarial Networks (GAN) paradigm. We introduce an augmented training procedure which employs a denoising autoencoder, trained in a high-level feature space learned by the discriminator, to guide the generator towards feature encodings which more closely resemble the data. We quantitatively evaluate our findings using the recently proposed Inception score

Biologically inspired computational structures and processes for autonomous agents and robots

Recent years have seen a proliferation of intelligent agent applications: from robots for space exploration to software agents for information filtering and electronic commerce on the Internet. Although the scope of these agent applications have blossomed tremendously since the advent of compact, affordable computing (and the recent emergence of the World Wide Web), the design of such agents for specific applications remains a daunting engineering problem;Rather than approach the design of artificial agents from a purely engineering standpoint, this dissertation views animals as biological agents, and considers artificial analogs of biological structures and processes in the design of effective agent behaviors. In particular, it explores behaviors generated by artificial neural structures appropriately shaped by the processes of evolution and spatial learning;The first part of this dissertation deals with the evolution of artificial neural controllers for a box-pushing robot task. We show that evolution discovers high fitness structures using little domain-specific knowledge, even in feedback-impoverished environments. Through a careful analysis of the evolved designs we also show how evolution exploits the environmental constraints and properties to produce designs of superior adaptive value. By modifying the task constraints in controlled ways, we also show the ability of evolution to quickly adapt to these changes and exploit them to obtain significant performance gains. We also use evolution to design the sensory systems of the box-pushing robots, particularly the number, placement, and ranges of their sensors. We find that evolution automatically discards unnecessary sensors retaining only the ones that appear to significantly affect the performance of the robot. This optimization of design across multiple dimensions (performance, number of sensors, size of neural controller, etc.) is implicitly achieved by the evolutionary algorithm without any external pressure (e.g., penalty on the use of more sensors or neurocontroller units). When used in the design of robots with limited battery capacities , evolution produces energy-efficient robot designs that use minimal numbers of components and yet perform reasonably well. The performance as well as the complexity of robot designs increase when the robots have access to a spatial learning mechanism that allows them to learn, remember, and navigate to power sources in the environment;The second part of this dissertation develops a computational characterization of the hippocampal formation which is known to play a significant role in animal spatial learning. The model is based on neuroscientific and behavioral data, and learns place maps based on interactions of sensory and dead-reckoning information streams. Using an estimation mechanism known as Kalman filtering, the model explicitly deals with uncertainties in the two information streams, allowing the robot to effectively learn and localize even in the presence sensing and motion errors. Additionally, the model has mechanisms to handle perceptual aliasing problems (where multiple places in the environment appear sensorily identical), incrementally learn and integrate local place maps, and learn and remember multiple goal locations in the environment. We show a number of properties of this spatial learning model including computational replication of several behavioral experiments performed with rodents. Not only does this model make significant contributions to robot localization, but also offers a number of predictions and suggestions that can be validated (or refuted) through systematic neurobiological and behavioral experiments with animals

Hybrid Deep Learning Techniques for Securing Bioluminescent Interfaces in Internet of Bio Nano Things

The Internet of bio-nano things (IoBNT) is an emerging paradigm employing nanoscale (~1–100 nm) biological transceivers to collect in vivo signaling information from the human body and communicate it to healthcare providers over the Internet. Bio-nano-things (BNT) offer external actuation of in-body molecular communication (MC) for targeted drug delivery to otherwise inaccessible parts of the human tissue. BNTs are inter-connected using chemical diffusion channels, forming an in vivo bio-nano network, connected to an external ex vivo environment such as the Internet using bio-cyber interfaces. Bio-luminescent bio-cyber interfacing (BBI) has proven to be promising in realizing IoBNT systems due to their non-obtrusive and low-cost implementation. BBI security, however, is a key concern during practical implementation since Internet connectivity exposes the interfaces to external threat vectors, and accurate classification of anomalous BBI traffic patterns is required to offer mitigation. However, parameter complexity and underlying intricate correlations among BBI traffic characteristics limit the use of existing machine-learning (ML) based anomaly detection methods typically requiring hand-crafted feature designing. To this end, the present work investigates the employment of deep learning (DL) algorithms allowing dynamic and scalable feature engineering to discriminate between normal and anomalous BBI traffic. During extensive validation using singular and multi-dimensional models on the generated dataset, our hybrid convolutional and recurrent ensemble (CNN + LSTM) reported an accuracy of approximately ~93.51% over other deep and shallow structures. Furthermore, employing a hybrid DL network allowed automated extraction of normal as well as temporal features in BBI data, eliminating manual selection and crafting of input features for accurate prediction. Finally, we recommend deployment primitives of the extracted optimal classifier in conventional intrusion detection systems as well as evolving non-Von Neumann architectures for real-time anomaly detection