304 research outputs found
SARDSRN: A NEURAL NETWORK SHIFT-REDUCE PARSER
Simple Recurrent Networks (SRNs) have been widely used in natural language tasks. SARDSRN extends the SRN by
explicitly representing the input sequence in a SARDNET self-organizing map. The distributed SRN component leads to good generalization and robust cognitive properties, whereas the SARDNET map provides exact representations of the sentence constituents. This combination allows SARDSRN to learn to parse sentences with more complicated structure than can the SRN alone, and suggests that the approach could scale up to realistic natural language
Scaling connectionist compositional representations
The Recursive Auto-Associative Memory (RAAM) has come to dominate connectionist investigations into representing compositional structure. Although an adequate model when dealing with limited data, the capacity of RAAM to scale-up to real-world tasks has been frequently questioned. RAAM networks are difficult to train (due to the moving target effect) and as such training times can be lengthy. Investigations into RAAM have produced many variants in an attempt to overcome such limitations. We outline how one such model ((S)RAAM) is able to quickly produce context-sensitive representations that may be used to aid a deterministic parsing process. By substituting a symbolic stack in an existing hybrid parser, we show that (S)RAAM is more than capable of encoding the real-world data sets employed. We conclude by suggesting that models such as (S)RAAM offer valuable insights into the features of connectionist compositional representations.<br /
The Unstoppable Rise of Computational Linguistics in Deep Learning
In this paper, we trace the history of neural networks applied to natural
language understanding tasks, and identify key contributions which the nature
of language has made to the development of neural network architectures. We
focus on the importance of variable binding and its instantiation in
attention-based models, and argue that Transformer is not a sequence model but
an induced-structure model. This perspective leads to predictions of the
challenges facing research in deep learning architectures for natural language
understanding.Comment: 13 pages. Accepted for publication at ACL 2020, in the theme trac
Integer Sparse Distributed Memory and Modular Composite Representation
Challenging AI applications, such as cognitive architectures, natural language understanding, and visual object recognition share some basic operations including pattern recognition, sequence learning, clustering, and association of related data. Both the representations used and the structure of a system significantly influence which tasks and problems are most readily supported. A memory model and a representation that facilitate these basic tasks would greatly improve the performance of these challenging AI applications.Sparse Distributed Memory (SDM), based on large binary vectors, has several desirable properties: auto-associativity, content addressability, distributed storage, robustness over noisy inputs that would facilitate the implementation of challenging AI applications. Here I introduce two variations on the original SDM, the Extended SDM and the Integer SDM, that significantly improve these desirable properties, as well as a new form of reduced description representation named MCR.Extended SDM, which uses word vectors of larger size than address vectors, enhances its hetero-associativity, improving the storage of sequences of vectors, as well as of other data structures. A novel sequence learning mechanism is introduced, and several experiments demonstrate the capacity and sequence learning capability of this memory.Integer SDM uses modular integer vectors rather than binary vectors, improving the representation capabilities of the memory and its noise robustness. Several experiments show its capacity and noise robustness. Theoretical analyses of its capacity and fidelity are also presented.A reduced description represents a whole hierarchy using a single high-dimensional vector, which can recover individual items and directly be used for complex calculations and procedures, such as making analogies. Furthermore, the hierarchy can be reconstructed from the single vector. Modular Composite Representation (MCR), a new reduced description model for the representation used in challenging AI applications, provides an attractive tradeoff between expressiveness and simplicity of operations. A theoretical analysis of its noise robustness, several experiments, and comparisons with similar models are presented.My implementations of these memories include an object oriented version using a RAM cache, a version for distributed and multi-threading execution, and a GPU version for fast vector processing
A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics
Inspired by humans' remarkable ability to master arithmetic and generalize to
unseen problems, we present a new dataset, HINT, to study machines' capability
of learning generalizable concepts at three different levels: perception,
syntax, and semantics. In particular, concepts in HINT, including both digits
and operators, are required to learn in a weakly-supervised fashion: Only the
final results of handwriting expressions are provided as supervision. Learning
agents need to reckon how concepts are perceived from raw signals such as
images (i.e., perception), how multiple concepts are structurally combined to
form a valid expression (i.e., syntax), and how concepts are realized to afford
various reasoning tasks (i.e., semantics). With a focus on systematic
generalization, we carefully design a five-fold test set to evaluate both the
interpolation and the extrapolation of learned concepts. To tackle this
challenging problem, we propose a neural-symbolic system by integrating neural
networks with grammar parsing and program synthesis, learned by a novel
deduction--abduction strategy. In experiments, the proposed neural-symbolic
system demonstrates strong generalization capability and significantly
outperforms end-to-end neural methods like RNN and Transformer. The results
also indicate the significance of recursive priors for extrapolation on syntax
and semantics.Comment: Preliminary wor
- …