21 research outputs found
An Interpretable Knowledge Transfer Model for Knowledge Base Completion
Knowledge bases are important resources for a variety of natural language
processing tasks but suffer from incompleteness. We propose a novel embedding
model, \emph{ITransF}, to perform knowledge base completion. Equipped with a
sparse attention mechanism, ITransF discovers hidden concepts of relations and
transfer statistical strength through the sharing of concepts. Moreover, the
learned associations between relations and concepts, which are represented by
sparse attention vectors, can be interpreted easily. We evaluate ITransF on two
benchmark datasets---WN18 and FB15k for knowledge base completion and obtains
improvements on both the mean rank and Hits@10 metrics, over all baselines that
do not use additional information.Comment: Accepted by ACL 2017. Minor updat
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
Mixture of Softmaxes (MoS) has been shown to be effective at addressing the
expressiveness limitation of Softmax-based models. Despite the known advantage,
MoS is practically sealed by its large consumption of memory and computational
time due to the need of computing multiple Softmaxes. In this work, we set out
to unleash the power of MoS in practical applications by investigating improved
word coding schemes, which could effectively reduce the vocabulary size and
hence relieve the memory and computation burden. We show both BPE and our
proposed Hybrid-LightRNN lead to improved encoding mechanisms that can halve
the time and memory consumption of MoS without performance losses. With MoS, we
achieve an improvement of 1.5 BLEU scores on IWSLT 2014 German-to-English
corpus and an improvement of 0.76 CIDEr score on image captioning. Moreover, on
the larger WMT 2014 machine translation dataset, our MoS-boosted Transformer
yields 29.5 BLEU score for English-to-German and 42.1 BLEU score for
English-to-French, outperforming the single-Softmax Transformer by 0.8 and 0.4
BLEU scores respectively and achieving the state-of-the-art result on WMT 2014
English-to-German task
RACE: Large-scale ReAding Comprehension Dataset From Examinations
We present RACE, a new dataset for benchmark evaluation of methods in the
reading comprehension task. Collected from the English exams for middle and
high school Chinese students in the age range between 12 to 18, RACE consists
of near 28,000 passages and near 100,000 questions generated by human experts
(English instructors), and covers a variety of topics which are carefully
designed for evaluating the students' ability in understanding and reasoning.
In particular, the proportion of questions that requires reasoning is much
larger in RACE than that in other benchmark datasets for reading comprehension,
and there is a significant gap between the performance of the state-of-the-art
models (43%) and the ceiling human performance (95%). We hope this new dataset
can serve as a valuable resource for research and evaluation in machine
comprehension. The dataset is freely available at
http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at
https://github.com/qizhex/RACE_AR_baselines.Comment: EMNLP 201
Recurrent Polynomial Network for Dialogue State Tracking
Dialogue state tracking (DST) is a process to estimate the distribution of the dialogue states as a dialogue progresses. Recent studies on constrained Markov Bayesian polynomial (CMBP) framework take the first step towards bridging the gap between rule-based and statistical approaches for DST. In this paper, the gap is further bridged by a novel framework -- recurrent polynomial network (RPN). RPN's unique structure enables the framework to have all the advantages of CMBP including efficiency, portability and interpretability. Additionally, RPN achieves more properties of statistical approaches than CMBP. RPN was evaluated on the data corpora of the second and the third Dialog State Tracking Challenge (DSTC-2/3). Experiments showed that RPN can significantly outperform both traditional rule-based approaches and statistical approaches with similar feature set. Compared with the state-of-the-art statistical DST approaches with a lot richer features, RPN is also competitive
Automatic Model Selection with Large Language Models for Reasoning
Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two
distinct reasoning methods, each with its own strengths. CoT employs natural
language, offering flexibility and interpretability, while PAL utilizes
programming language, yielding more structured and rigorous logic. We introduce
a model selection method to combine the best of both worlds by employing a
large language model (LLM) to dynamically select between them. Our theoretical
analysis underscores the feasibility of this method, which is further
corroborated by empirical results. Our proposed method demonstrates significant
performance improvements across eight reasoning datasets with Codex, ChatGPT,
and GPT-4. Additionally, our method is complementary to self-consistency; when
integrated, it can further enhance performance while significantly reducing
computation costs. Moreover, we achieve new state-of-the-art results on GSM8K
and SVAMP, with respective accuracies of 96.8% and 93.7%. Our code, data and
prompts are available at https://github.com/XuZhao0/Model-Selection-ReasoningComment: EMNLP 2023 Finding