Weighting Finite-State Transductions With Neural Context

How should one apply deep learning to tasks such as morphological reinflection, which stochastically edit one string to get another? A recent approach to such sequence-to-sequence tasks is to compress the input string into a vector that is then used to generate the out- put string, using recurrent neural networks. In contrast, we propose to keep the traditional architecture, which uses a finite-state trans- ducer to score all possible output strings, but to augment the scoring function with the help of recurrent networks. A stack of bidirec- tional LSTMs reads the input string from left- to-right and right-to-left, in order to summa- rize the input context in which a transducer arc is applied. We combine these learned fea- tures with the transducer to define a probabil- ity distribution over aligned output strings, in the form of a weighted finite-state automaton. This reduces hand-engineering of features, al- lows learned features to examine unbounded context in the input string, and still permits ex- act inference through dynamic programming. We illustrate our method on the tasks of mor- phological reinflection and lemmatization

Paradigm Completion for Derivational Morphology

The generation of complex derived word forms has been an overlooked problem in NLP; we fill this gap by applying neural sequence-to-sequence models to the task. We overview the theoretical motivation for a paradigmatic treatment of derivational morphology, and introduce the task of derivational paradigm completion as a parallel to inflectional paradigm completion. State-of-the-art neural models, adapted from the inflection task, are able to learn a range of derivation patterns, and outperform a non-neural baseline by 16.4%. However, due to semantic, historical, and lexical considerations involved in derivational morphology, future work will be needed to achieve performance parity with inflection-generating systems.Comment: EMNLP 201

Unsupervised Neural Hidden Markov Models

In this work, we present the first results for neuralizing an Unsupervised Hidden Markov Model. We evaluate our approach on tag in- duction. Our approach outperforms existing generative models and is competitive with the state-of-the-art though with a simpler model easily extended to include additional context.Comment: accepted at EMNLP 2016, Workshop on Structured Prediction for NLP. Oral presentatio