11,498 research outputs found
Convolutional Neural Networks with Recurrent Neural Filters
We introduce a class of convolutional neural networks (CNNs) that utilize
recurrent neural networks (RNNs) as convolution filters. A convolution filter
is typically implemented as a linear affine transformation followed by a
non-linear function, which fails to account for language compositionality. As a
result, it limits the use of high-order filters that are often warranted for
natural language processing tasks. In this work, we model convolution filters
with RNNs that naturally capture compositionality and long-term dependencies in
language. We show that simple CNN architectures equipped with recurrent neural
filters (RNFs) achieve results that are on par with the best published ones on
the Stanford Sentiment Treebank and two answer sentence selection datasets.Comment: Accepted by EMNLP 2018 as a short pape
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review
The paper characterizes classes of functions for which deep learning can be
exponentially better than shallow learning. Deep convolutional networks are a
special case of these conditions, though weight sharing is not the main reason
for their exponential advantage
Dynamic Compositional Neural Networks over Tree Structure
Tree-structured neural networks have proven to be effective in learning
semantic representations by exploiting syntactic information. In spite of their
success, most existing models suffer from the underfitting problem: they
recursively use the same shared compositional function throughout the whole
compositional process and lack expressive power due to inability to capture the
richness of compositionality. In this paper, we address this issue by
introducing the dynamic compositional neural networks over tree structure
(DC-TreeNN), in which the compositional function is dynamically generated by a
meta network. The role of meta-network is to capture the metaknowledge across
the different compositional rules and formulate them. Experimental results on
two typical tasks show the effectiveness of the proposed models.Comment: Accepted by IJCAI 201
Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown
distinct advantages, e.g., solving memory-dependent tasks and meta-learning.
However, little effort has been spent on improving RNN architectures and on
understanding the underlying neural mechanisms for performance gain. In this
paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical
results show that the network can autonomously learn to abstract sub-goals and
can self-develop an action hierarchy using internal dynamics in a challenging
continuous control task. Furthermore, we show that the self-developed
compositionality of the network enhances faster re-learning when adapting to a
new task that is a re-composition of previously learned sub-goals, than when
starting from scratch. We also found that improved performance can be achieved
when neural activities are subject to stochastic rather than deterministic
dynamics
- …