Compressing Recurrent Neural Network with Tensor Train

Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.Comment: Accepted at IJCNN 201

Examination of the relationship between essential genes in PPI network and hub proteins in reverse nearest neighbor topology

Abstract Background In many protein-protein interaction (PPI) networks, densely connected hub proteins are more likely to be essential proteins. This is referred to as the "centrality-lethality rule", which indicates that the topological placement of a protein in PPI network is connected with its biological essentiality. Though such connections are observed in many PPI networks, the underlying topological properties for these connections are not yet clearly understood. Some suggested putative connections are the involvement of essential proteins in the maintenance of overall network connections, or that they play a role in essential protein clusters. In this work, we have attempted to examine the placement of essential proteins and the network topology from a different perspective by determining the correlation of protein essentiality and reverse nearest neighbor topology (RNN). Results The RNN topology is a weighted directed graph derived from PPI network, and it is a natural representation of the topological dependences between proteins within the PPI network. Similar to the original PPI network, we have observed that essential proteins tend to be hub proteins in RNN topology. Additionally, essential genes are enriched in clusters containing many hub proteins in RNN topology (RNN protein clusters). Based on these two properties of essential genes in RNN topology, we have proposed a new measure; the RNN cluster centrality. Results from a variety of PPI networks demonstrate that RNN cluster centrality outperforms other centrality measures with regard to the proportion of selected proteins that are essential proteins. We also investigated the biological importance of RNN clusters. Conclusions This study reveals that RNN cluster centrality provides the best correlation of protein essentiality and placement of proteins in PPI network. Additionally, merged RNN clusters were found to be topologically important in that essential proteins are significantly enriched in RNN clusters, and biologically important because they play an important role in many Gene Ontology (GO) processes.http://deepblue.lib.umich.edu/bitstream/2027.42/78257/1/1471-2105-11-505.xmlhttp://deepblue.lib.umich.edu/bitstream/2027.42/78257/2/1471-2105-11-505-S1.DOChttp://deepblue.lib.umich.edu/bitstream/2027.42/78257/3/1471-2105-11-505.pdfPeer Reviewe

A multi-stage recurrent neural network better describes decision-related activity in dorsal premotor cortex

We studied how a network of recurrently connected artificial units solve a visual perceptual decision-making task. The goal of this task is to discriminate the dominant color of a central static checkerboard and report the decision with an arm movement. This task has been used to study neural activity in the dorsal premotor (PMd) cortex. When a single recurrent neural network (RNN) was trained to perform the task, the activity of artificial units in the RNN differed from neural recordings in PMd, suggesting that inputs to PMd differed from inputs to the RNN. We expanded our architecture and examined how a multi-stage RNN performed the task. In the multi-stage RNN, the last stage exhibited similarities with PMd by representing direction information but not color information. We then investigated how the representation of color and direction information evolve across RNN stages. Together, our results are a demonstration of the importance of incorporating architectural constraints into RNN models. These constraints can improve the ability of RNNs to model neural activity in association areas.https://doi.org/10.32470/CCN.2019.1123-0Accepted manuscrip

Economic MPC of Nonlinear Processes via Recurrent Neural Networks Using Structural Process Knowledge

This work discusses three methods that incorporate a priori process knowledge into recurrent neural network (RNN) modeling of nonlinear processes to get increased prediction accuracy and provide information on how the neural network models are structured. The first method proposes a hybrid model that integrates first-principles models and RNN models together. The second method proposes a partially-connected RNN model which its structure is based on a priori structural process knowledge. The third method proposes a weight-constrained RNN model that integrates weight constraints into the training of the RNN model. The proposed RNN models are used in an economic model predictive control system and then applied to a chemical process example to validate the improved approximation performance compared to a fully-connected RNN model that is treated as a black box model