2,920 research outputs found
On the Computational Complexity and Formal Hierarchy of Second Order Recurrent Neural Networks
Artificial neural networks (ANNs) with recurrence and self-attention have
been shown to be Turing-complete (TC). However, existing work has shown that
these ANNs require multiple turns or unbounded computation time, even with
unbounded precision in weights, in order to recognize TC grammars. However,
under constraints such as fixed or bounded precision neurons and time, ANNs
without memory are shown to struggle to recognize even context-free languages.
In this work, we extend the theoretical foundation for the -order
recurrent network ( RNN) and prove there exists a class of a
RNN that is Turing-complete with bounded time. This model is capable of
directly encoding a transition table into its recurrent weights, enabling
bounded time computation and is interpretable by design. We also demonstrate
that nd order RNNs, without memory, under bounded weights and time
constraints, outperform modern-day models such as vanilla RNNs and gated
recurrent units in recognizing regular grammars. We provide an upper bound and
a stability analysis on the maximum number of neurons required by nd order
RNNs to recognize any class of regular grammar. Extensive experiments on the
Tomita grammars support our findings, demonstrating the importance of tensor
connections in crafting computationally efficient RNNs. Finally, we show
order RNNs are also interpretable by extraction and can extract state
machines with higher success rates as compared to first-order RNNs. Our results
extend the theoretical foundations of RNNs and offer promising avenues for
future explainable AI research.Comment: 12 pages, 5 tables, 1 figur
Understanding Hidden Memories of Recurrent Neural Networks
Recurrent neural networks (RNNs) have been successfully applied to various
natural language processing (NLP) tasks and achieved better results than
conventional methods. However, the lack of understanding of the mechanisms
behind their effectiveness limits further improvements on their architectures.
In this paper, we present a visual analytics method for understanding and
comparing RNN models for NLP tasks. We propose a technique to explain the
function of individual hidden state units based on their expected response to
input texts. We then co-cluster hidden state units and words based on the
expected response and visualize co-clustering results as memory chips and word
clouds to provide more structured knowledge on RNNs' hidden states. We also
propose a glyph-based sequence visualization based on aggregate information to
analyze the behavior of an RNN's hidden state at the sentence-level. The
usability and effectiveness of our method are demonstrated through case studies
and reviews from domain experts.Comment: Published at IEEE Conference on Visual Analytics Science and
Technology (IEEE VAST 2017
Memristors for the Curious Outsiders
We present both an overview and a perspective of recent experimental advances
and proposed new approaches to performing computation using memristors. A
memristor is a 2-terminal passive component with a dynamic resistance depending
on an internal parameter. We provide an brief historical introduction, as well
as an overview over the physical mechanism that lead to memristive behavior.
This review is meant to guide nonpractitioners in the field of memristive
circuits and their connection to machine learning and neural computation.Comment: Perpective paper for MDPI Technologies; 43 page
On the Tensor Representation and Algebraic Homomorphism of the Neural State Turing Machine
Recurrent neural networks (RNNs) and transformers have been shown to be
Turing-complete, but this result assumes infinite precision in their hidden
representations, positional encodings for transformers, and unbounded
computation time in general. In practical applications, however, it is crucial
to have real-time models that can recognize Turing complete grammars in a
single pass. To address this issue and to better understand the true
computational power of artificial neural networks (ANNs), we introduce a new
class of recurrent models called the neural state Turing machine (NSTM). The
NSTM has bounded weights and finite-precision connections and can simulate any
Turing Machine in real-time. In contrast to prior work that assumes unbounded
time and precision in weights, to demonstrate equivalence with TMs, we prove
that a -neuron bounded tensor RNN, coupled with third-order synapses, can
model any TM class in real-time. Furthermore, under the Markov assumption, we
provide a new theoretical bound for a non-recurrent network augmented with
memory, showing that a tensor feedforward network with th-order finite
precision weights is equivalent to a universal TM.Comment: 14 pages, 7 table
- …