Training Input-Output Recurrent Neural Networks through Spectral Methods

We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks. We propose a novel spectral approach for learning the network parameters. It is based on decomposition of the cross-moment tensor between the output and a non-linear transformation of the input, based on score functions. We guarantee consistent learning with polynomial sample and computational complexity under transparent conditions such as non-degeneracy of model parameters, polynomial activations for the neurons, and a Markovian evolution of the input sequence. We also extend our results to Bidirectional RNN which uses both previous and future information to output the label at each time point, and is employed in many NLP tasks such as POS tagging

Learning Hybrid System Models for Supervisory Decoding of Discrete State, with applications to the Parietal Reach Region

Based on Gibbs sampling, a novel method to identify mathematical models of neural activity in response to temporal changes of behavioral or cognitive state is presented. This work is motivated by the developing field of neural prosthetics, where a supervisory controller is required to classify activity of a brain region into suitable discrete modes. Here, neural activity in each discrete mode is modeled with nonstationary point processes, and transitions between modes are modeled as hidden Markov models. The effectiveness of this framework is first demonstrated on a simulated example. The identification algorithm is then applied to extracellular neural activity recorded from multi-electrode arrays in the parietal reach region of a rhesus monkey, and the results demonstrate the ability to decode discrete changes even from small data sets