Fast and robust learning by reinforcement signals: explorations in the insect brain

We propose a model for pattern recognition in the insect brain. Departing from a well-known body of knowledge about the insect brain, we investigate which of the potentially present features may be useful to learn input patterns rapidly and in a stable manner. The plasticity underlying pattern recognition is situated in the insect mushroom bodies and requires an error signal to associate the stimulus with a proper response. As a proof of concept, we used our model insect brain to classify the well-known MNIST database of handwritten digits, a popular benchmark for classifiers. We show that the structural organization of the insect brain appears to be suitable for both fast learning of new stimuli and reasonable performance in stationary conditions. Furthermore, it is extremely robust to damage to the brain structures involved in sensory processing. Finally, we suggest that spatiotemporal dynamics can improve the level of confidence in a classification decision. The proposed approach allows testing the effect of hypothesized mechanisms rather than speculating on their benefit for system performance or confidence in its responses

Optimal anticipatory control as a theory of motor preparation

Supported by a decade of primate electrophysiological experiments, the prevailing theory of neural motor control holds that movement generation is accomplished by a preparatory process that progressively steers the state of the motor cortex into a movement-specific optimal subspace prior to movement onset. The state of the cortex then evolves from these optimal subspaces, producing patterns of neural activity that serve as control inputs to the musculature. This theory, however, does not address the following questions: what characterizes the optimal subspace and what are the neural mechanisms that underlie the preparatory process? We address these questions with a circuit model of movement preparation and control. Specifically, we propose that preparation can be achieved by optimal feedback control (OFC) of the cortical state via a thalamo-cortical loop. Under OFC, the state of the cortex is selectively controlled along state-space directions that have future motor consequences, and not in other inconsequential ones. We show that OFC enables fast movement preparation and explains the observed orthogonality between preparatory and movement-related monkey motor cortex activity. This illustrates the importance of constraining new theories of neural function with experimental data. However, as recording technologies continue to improve, a key challenge is to extract meaningful insights from increasingly large-scale neural recordings. Latent variable models (LVMs) are powerful tools for addressing this challenge due to their ability to identify the low-dimensional latent variables that best explain these large data sets. One shortcoming of most LVMs, however, is that they assume a Euclidean latent space, while many kinematic variables, such as head rotations and the configuration of an arm, are naturally described by variables that live on non-Euclidean latent spaces (e.g., SO(3) and tori). To address this shortcoming, we propose the Manifold Gaussian Process Latent Variable Model, a method for simultaneously inferring nonparametric tuning curves and latent variables on non-Euclidean latent spaces. We show that our method is able to correctly infer the latent ring topology of the fly and mouse head direction circuits.This work was supported by a Trinity-Henry Barlow scholarship and a scholarship from the Ministry of Education, ROC Taiwan

Real-Time Progressive Learning: Mutually Reinforcing Learning and Control with Neural-Network-Based Selective Memory

Memory, as the basis of learning, determines the storage, update and forgetting of the knowledge and further determines the efficiency of learning. Featured with a mechanism of memory, a radial basis function neural network (RBFNN) based learning control scheme named real-time progressive learning (RTPL) is proposed to learn the unknown dynamics of the system with guaranteed stability and closed-loop performance. Instead of the stochastic gradient descent (SGD) update law in adaptive neural control (ANC), RTPL adopts the selective memory recursive least squares (SMRLS) algorithm to update the weights of the RBFNN. Through SMRLS, the approximation capabilities of the RBFNN are uniformly distributed over the feature space and thus the passive knowledge forgetting phenomenon of SGD method is suppressed. Subsequently, RTPL achieves the following merits over the classical ANC: 1) guaranteed learning capability under low-level persistent excitation (PE), 2) improved learning performance (learning speed, accuracy and generalization capability), and 3) low gain requirement ensuring robustness of RTPL in practical applications. Moreover, the RTPL based learning and control will gradually reinforce each other during the task execution, making it appropriate for long-term learning control tasks. As an example, RTPL is used to address the tracking control problem of a class of nonlinear systems with RBFNN being an adaptive feedforward controller. Corresponding theoretical analysis and simulation studies demonstrate the effectiveness of RTPL.Comment: 16 pages, 15 figure

The neural marketplace

The `retroaxonal hypothesis' (Harris, 2008) posits a role for slow retrograde signalling in learning. It is based on the intuition that cells with strong output synapses tend to be those that encode useful information; and that cells which encode useful information should not modify their input synapses too readily. The hypothesis has two parts: rst, that the stronger a cell's output synapses, the less likely it is to change its input synapses; and second, that a cell is more likely to revert changes to its input synapses when the changes are followed by weakening of its output synapses. It is motivated in part by analogy between a neural network and a market economy, viewing neurons as `entrepreneurs' who `sell' spike trains to each other. In this view, the slow retrograde signals which tell a neuron that it has strong output synapses are `money' and imply that what it produces is useful. This thesis constructs a mathematical model of learning, which validates the intuition of the retroaxonal hypothesis. In this model, we show that neurons can estimate their usefulness, or `worth', from the magnitude of their output weights. We also show that by making each cell's input synapses more or less plastic according to its worth, the performance of a network can be improved.Open Acces

Computational Principles of Multiple-Task Learning in Humans and Artificial Neural Networks

While humans can learn to perform many specific and highly specialized behaviors,perhaps what is most unique about human cognitive capabilities is their capacity to generalize, to share information across contexts and adapt to the myriad problems that can arise in complex environments. While it is possible to imagine agents who learn to deal with each challenge they experience separately, humans instead integrate new situations into the framework of the tasks they have experienced in their life, allowing them to reuse insight and strategies across them. Yet the precise forms of shared representations across tasks, as well as computational principles for how sharing of insight over learning multiple tasks may impact behavior, remain uncertain. The significant complexity in the problem of cognition capable of generalizing across tasks has been both an inspiration and a significant impediment to building useful and insightful models. The increasing utilization of artificial neural networks (ANN) as a model for cortical computation provides a potent opportunity to identify mechanisms and principles underlying multiple-task learning and performance in the brain. In this work we use ANNs in conjunction with human behavior to explore how a single agent may utilize information across multiple tasks to create high performing and general representations. First, we present a flexible framework to facilitate training recurrent neural networks (RNN), increasing the ease of training models on tasks of interest. Second, we explore how an ANN model can build shared representations to facilitate performance on a wide variety of delay task problems, as well as how such a joint representation can explain observed phenomena identified in the firing rates of prefrontal cortical neurons. Third, we analyze human multiple-task learning in two tasks and use ANNs to provide insight into how the structure of representations can give rise to the specific learning patterns and generalization strategies observed in humans. Overall, we provide computational insight into mechanisms of multiple-task learning and generalization as well as use those findings in conjunction with observed human behavior to constrain possible computational mechanisms employed in cortical circuits