Sublinear Parallel Time Recognition of Tree Adjoining Language

A parallel algorithm is presented for recognizing the class of languages generated by tree adjoining grammars, a tree rewriting system which has applications in computational Linguistics. This class of languages is known to properly include all context-free languages; for example, the non-context-free sets {anbncn} and {ww) are in this class. It is shown that the recognition problem for tree adjoining languages can be solved by a concurrent-read, exclusive-write parallel random-access machine (CREW PRAM) in 0 (log2(n)) time using polynomially many processors. This extends a previous result for context-free languages

Sublinearly space bounded iterative arrays

Iterative arrays (IAs) are a, parallel computational model with a sequential processing of the input. They are one-dimensional arrays of interacting identical deterministic finite automata. In this note, realtime-lAs with sublinear space bounds are used to accept formal languages. The existence of a proper hierarchy of space complexity classes between logarithmic anel linear space bounds is proved. Furthermore, an optimal spacc lower bound for non-regular language recognition is shown. Key words: Iterative arrays, cellular automata, space bounded computations, decidability questions, formal languages, theory of computatio

CLiFF Notes: Research In Natural Language Processing at the University of Pennsylvania

CLIFF is the Computational Linguists\u27 Feedback Forum. We are a group of students and faculty who gather once a week to hear a presentation and discuss work currently in progress. The \u27feedback\u27 in the group\u27s name is important: we are interested in sharing ideas, in discussing ongoing research, and in bringing together work done by the students and faculty in Computer Science and other departments. However, there are only so many presentations which we can have in a year. We felt that it would be beneficial to have a report which would have, in one place, short descriptions of the work in Natural Language Processing at the University of Pennsylvania. This report then, is a collection of abstracts from both faculty and graduate students, in Computer Science, Psychology and Linguistics. We want to stress the close ties between these groups, as one of the things that we pride ourselves on here at Penn is the communication among different departments and the inter-departmental work. Rather than try to summarize the varied work currently underway at Penn, we suggest reading the abstracts to see how the students and faculty themselves describe their work. The report illustrates the diversity of interests among the researchers here, as well as explaining the areas of common interest. In addition, since it was our intent to put together a document that would be useful both inside and outside of the university, we hope that this report will explain to everyone some of what we are about

Upper Bounds on Recognition of a Hierarchy of Non-Context-Free Languages

Control grammars, a generalization of context-free grammars recently introduced for use in natural language recognition, are investigated. In particular, it is shown that a hierarchy of non-context-free languages, called the Control Language Hierarchy (CLH), generated by control grammars can be recognized in polynomial time. Previously, the best known upper bound was exponential time. It is also shown that CLH is in NC(2) the class of languages recognizable by uniform boolean circuits of polynomial size and O(log2 n) depth

In Memoriam, Solomon Marcus

This book commemorates Solomon Marcus’s fifth death anniversary with a selection of articles in mathematics, theoretical computer science, and physics written by authors who work in Marcus’s research fields, some of whom have been influenced by his results and/or have collaborated with him

Learning and time : on using memory and curricula for language understanding

Cette thèse présente quelques-unes des étapes entreprises pour pouvoir un jour résoudre le problème de la compréhension du langage naturel et d’apprentissage de dépendances à long terme, dans le but de développer de meilleurs algorithmes d’intelligence artificielle. Cette thèse est écrite comme une thèse par articles, et contient cinq publications scientifiques. Chacun de ces articles propose un nouveau modèle ou algorithme et démontre leur efficacité sur des problèmes qui impliquent des dépendances à long terme ou la compréhension du langage naturel. Malgré le fait que quelque uns de ces modèles n’ont été testés que sur une seule tâche (comme la traduction automatique neuronale), les méthodes proposées sont généralement applicables dans d’autres domaines et sur d’autres tâches. Dans l’introduction de la thèse, nous expliquons quelques concepts fondamentaux de l'entraînement de réseaux de neurones appliqués sur des données séquentielles. Tout d'abord, nous présentons succinctement les réseaux de neurones, puis, de façon plus détaillé, certains algorithmes et méthodes utilisés à travers cette thèse. Dans notre premier article, nous proposons une nouvelle méthode permettant d'utiliser la grande quantité de données monolingue disponible afin d'entraîner des modèles de traduction. Nous avons accompli cela en entraînant d’abord un modèle Long short-term memory (LSTM) sur un large corpus monolingue. Nous lions ensuite la sortie de la couche cachée du modèle avec celle d’un décodeur d’un modèle de traduction automatique. Ce dernier utilise un mécanisme d’attention et est entièrement entraîné par descente de gradient. Nous avons montré que la méthode proposée peut augmenter la performance des modèles de traduction automatique neuronale de façon significative sur les tâches où peu de données multilingues sont disponibles. Notre approche augmente également l’efficacité de l’utilisation des données dans les systèmes de traduction automatique. Nous montrons aussi des améliorations sur les paires de langues suivantes: turc-anglais, allemand-anglais, chinois-anglais et tchèque-anglais. Dans notre deuxième article, nous proposons une approche pour aborder le problème des mots rares dans plusieurs tâches du traitement des langages. Notre approche modifie l’architecture habituelle des modèles encodeur-décodeur avec attention, en remplaçant la couche softmax du décodeur par notre couche pointer-softmax. Celle-ci permet au décodeur de pointer à différents endroits dans la phrase d’origine. Notre modèle apprend à alterner entre copier un mot de la phrase d’origine et prédire un mot provenant d’une courte liste de mots prédéfinie, de manière probabiliste. L’approche que nous avons proposée est entièrement entraînable par descente de gradient et n’utilise qu’un objectif de maximum de vraisemblance sur les tâches de traduction. Nous avons aussi montré que le pointer-softmax aide de manière significative aux tâches de traduction et de synthèse de documents. Dans notre article "Plan, Attend, Generate: Planning for Sequence-to-Sequence Models", nous proposons deux approches pour apprendre l’alignement dans les modèles entraînés sur des séquences. Lorsque la longueur de l’entrée et celle de la sortie sont trop grandes, apprendre les alignements peut être très difficile. La raison est que lorsque le décodeur est trop puissant, il a tendance à ignorer l’alignement des mots pour ne se concentrer que sur le dernier mot de la séquence d’entrée. Nous avons proposé une nouvelle approche, inspirée d’un algorithme d’apprentissage par renforcement, en ajoutant explicitement un mécanisme de planification au décodeur. Ce nouveau mécanisme planifie à l’avance l’alignement pour les k prochaines prédictions. Notre modèle apprend également un plan de correction pour déterminer lorsqu’il est nécessaire de recalculer les alignements. Notre approche peut apprendre de haut niveaux d’abstraction au point de vue temporel et nous montrons que les alignements sont généralement de meilleure qualité. Nous obtenons également des gains de performance significatifs comparativement à notre modèle de référence, malgré le fait que nos modèles ont moins de paramètres et qu’ils aient été entraînés moins longtemps. Dans notre article "Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes", nous proposons une nouvelle approche pour ajouter de manière explicite un mécanisme de mémoire aux réseaux de neurones. Contrairement aux RNNs conventionnels, la mémoire n’est pas seulement représentée au niveau des activations du réseau, mais également dans une mémoire externe. Notre modèle, D-NTM, utilise un mécanisme d’adressage plus simple que les Neural Turing Machine (NTM) en utilisant des paires clé-valeur. Nous montrons que les modèles disposant de ce nouveau mécanisme peuvent plus efficacement apprendre les dépendances à long terme, en plus de mieux généraliser. Nous obtenons des améliorations sur plusieurs tâches incluant entre autres la réponse aux questions sur bAbI, le raisonnement avec implication, MNIST permuté, ainsi que des tâches synthétiques. Dans notre article "Noisy Activation Functions", nous proposons une nouvelle fonction d’activation, qui rend les activations stochastiques en leur ajoutant du bruit. Notre motivation dans cet article est d’aborder les problèmes d’optimisation qui surviennent lorsque nous utilisons des fonctions d’activation qui saturent, comme celles généralement utilisées dans les RNNs. Notre approche permet d’utiliser des fonctions d’activation linéaires par morceaux sur les RNNs à porte. Nous montrons des améliorations pour un grand nombre de tâches sans effectuer de recherche d'hyper paramètres intensive. Nous montrons également que supprimer le bruit dans les fonctions d’activation a un profond impact sur l’optimisation.The goal of this thesis is to present some of the small steps taken on the path towards solving natural language understanding and learning long-term dependencies to develop artificial intelligence algorithms that can reason with language. This thesis is written as a thesis by articles and contains five articles. Each article in this thesis proposes a new model or algorithm and demonstrates the efficiency of the proposed approach to solve problems that involve long-term dependencies or require natural language understanding. Although some of the models are tested on a particular task (such as neural machine translation), the proposed methods in this thesis are generally applicable to other domains and tasks (and have been used in the literature). In the introduction of this thesis, we introduce some of the fundamental concepts behind training sequence models using neural networks. We first provide a brief introduction to neural networks and then dive into details of the some of approaches and algorithms that are used throughout this thesis. In our first article, we propose a novel method to utilize the abundant amount of available monolingual data for training neural machine translation models. We have accomplished this goal by first training a long short-term memory (LSTM) language model on a large monolingual corpus and then fusing the outputs or the hidden states of the LSTM language model with the decoder of the neural machine translation model. Our neural machine translation model is trained end to end with an attention mechanism. We have shown that our proposed approaches can improve the performance of the neural machine translation models significantly on the rare resource translation tasks and our approach improved the data-efficiency of the end to end neural machine translation systems. We report improvements on Turkish-English (Tr-En), German-English (De-En), Chinese-English (Zh-En) and Czech-English (Cz-En) translation tasks. In our second paper, we propose an approach to address the problem of rare words for natural language processing tasks. Our approach augments the encoder-decoder architecture with attention model by replacing the final softmax layer with our proposed pointer-softmax layer that creates pointers to the source sentences as the decoder translates. In the case of pointer-softmax, our model learns to switch between copying a word from the source and predicting a word from a shortlist vocabulary in a probabilistic manner. Our proposed approach is end-to-end trainable with a single maximum likelihood objective of the NMT model. We have also shown that it improves the performance of summarization and the neural machine translation model. We report significant improvements in machine translation and summarization tasks. In our "Plan, Attend, Generate: Planning for Sequence-to-Sequence Models" paper, we propose two new approaches to learn alignments in a sequence to sequence model. If the input and the source context is very long, learning the alignments for a sequence to sequence model can be difficult. In particular, because when the decoder is a large network, it can learn to ignore the alignments and attend more on the last token of the input sequence. We propose a new approach which is inspired by a hierarchical reinforcement learning algorithm and extend our model with an explicit planning mechanism. The proposed alignment mechanism plans and computes the alignments for the next $k$ tokens in the decoder. Our model also learns a commitment plan to decide when to recompute the alignment matrix. Our proposed approach can learn high-level temporal abstractions, and we show that it qualitatively learns better alignments. We also achieve significant improvements over our baseline despite using smaller models and with less training. In "Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes," we propose a new approach for augmenting neural networks with an explicit memory mechanism. As opposed to conventional RNNs, the memory is not only represented in the activations of the neural network but in an external memory that can be accessed via the neural network controller. Our model, D-NTM uses a more straightforward memory addressing mechanism than NTM which is achieved by using key-value pairs for each memory cell. We find out that the models augmented with an external memory mechanism can learn tasks that involve long-term dependencies more efficiently and achieve better generalization. We achieve improvements on many tasks including but not limited to episodic question answering on bAbI, reasoning with entailment, permuted MNIST task and synthetic tasks. In our "Noisy Activation Functions" paper, we propose a novel activation function that makes the activations stochastic by injecting a particular form of noise to them. Our motivation in this paper is to address the optimization problem of commonly used saturating activation functions that are used with the recurrent neural networks. Our approach enables us to use piece-wise linear activation functions on the gated recurrent neural network models. We show improvements in a wide range of tasks without doing any extensive hyperparameter search by a drop-in replacement. We also show that annealing the noise of the activation function can have a profound continuation-like effect on the optimization of the network

Small-Space Algorithms for the Online Language Distance Problem for Palindromes and Squares

We study the online variant of the language distance problem for two classical formal languages, the language of palindromes and the language of squares, and for the two most fundamental distances, the Hamming distance and the edit (Levenshtein) distance. In this problem, defined for a fixed formal language $L$, we are given a string $T$ of length $n$, and the task is to compute the minimal distance to $L$ from every prefix of $T$. We focus on the low-distance regime, where one must compute only the distances smaller than a given threshold $k$. In this work, our contribution is twofold: - First, we show streaming algorithms, which access the input string $T$ only through a single left-to-right scan. Both for palindromes and squares, our algorithms use $O(k \cdot\mathrm{poly}~\log n)$ space and time per character in the Hamming-distance case and $O(k^2 \cdot\mathrm{poly}~\log n)$ space and time per character in the edit-distance case. These algorithms are randomised by necessity, and they err with probability inverse-polynomial in $n$. - Second, we show deterministic read-only online algorithms, which are also provided with read-only random access to the already processed characters of $T$. Both for palindromes and squares, our algorithms use $O(k \cdot\mathrm{poly}~\log n)$ space and time per character in the Hamming-distance case and $O(k^4 \cdot\mathrm{poly}~\log n)$ space and amortised time per character in the edit-distance case.Comment: Accepted to ISAAC'2

Proof nets for linguistic analysis

This book investigates the possible linguistic applications of proof nets, redundancy free representations of proofs, which were introduced by Girard for linear logic. We will adapt the notion of proof net to allow the formulation of a proof net calculus which is soundand complete for the multimodal Lambek calculus. Finally, we will investigate the computational and complexity theoretic consequences of this calculus and give an introduction to a practical grammar development tool based on proof nets

28th Annual Symposium on Combinatorial Pattern Matching : CPM 2017, July 4-6, 2017, Warsaw, Poland

Peer reviewe