18,053 research outputs found
Quantifying Uncertainties in Natural Language Processing Tasks
Reliable uncertainty quantification is a first step towards building
explainable, transparent, and accountable artificial intelligent systems.
Recent progress in Bayesian deep learning has made such quantification
realizable. In this paper, we propose novel methods to study the benefits of
characterizing model and data uncertainties for natural language processing
(NLP) tasks. With empirical experiments on sentiment analysis, named entity
recognition, and language modeling using convolutional and recurrent neural
network models, we show that explicitly modeling uncertainties is not only
necessary to measure output confidence levels, but also useful at enhancing
model performances in various NLP tasks.Comment: To appear at AAAI 201
Using the Output Embedding to Improve Language Models
We study the topmost weight matrix of neural network language models. We show
that this matrix constitutes a valid word embedding. When training language
models, we recommend tying the input embedding and this output embedding. We
analyze the resulting update rules and show that the tied embedding evolves in
a more similar way to the output embedding than to the input embedding in the
untied model. We also offer a new method of regularizing the output embedding.
Our methods lead to a significant reduction in perplexity, as we are able to
show on a variety of neural network language models. Finally, we show that
weight tying can reduce the size of neural translation models to less than half
of their original size without harming their performance.Comment: To appear in EACL 201
- …