135 research outputs found
A Dependency-Based Neural Network for Relation Classification
Previous research on relation classification has verified the effectiveness
of using dependency shortest paths or subtrees. In this paper, we further
explore how to make full use of the combination of these dependency
information. We first propose a new structure, termed augmented dependency path
(ADP), which is composed of the shortest dependency path between two entities
and the subtrees attached to the shortest path. To exploit the semantic
representation behind the ADP structure, we develop dependency-based neural
networks (DepNN): a recursive neural network designed to model the subtrees,
and a convolutional neural network to capture the most important features on
the shortest path. Experiments on the SemEval-2010 dataset show that our
proposed method achieves state-of-art results.Comment: This preprint is the full version of a short paper accepted in the
annual meeting of the Association for Computational Linguistics (ACL) 2015
(Beijing, China
Learning to Retrieve In-Context Examples for Large Language Models
Large language models (LLMs) have demonstrated their ability to learn
in-context, allowing them to perform various tasks based on a few input-output
examples. However, the effectiveness of in-context learning is heavily reliant
on the quality of the selected examples. In this paper, we propose a novel
framework to iteratively train dense retrievers that can identify high-quality
in-context examples for LLMs. Our framework initially trains a reward model
based on LLM feedback to evaluate the quality of candidate examples, followed
by knowledge distillation to train a bi-encoder based dense retriever. Our
experiments on a suite of 30 tasks demonstrate that our framework significantly
enhances in-context learning performance. Furthermore, we show the
generalization ability of our framework to unseen tasks during training. An
in-depth analysis reveals that our model improves performance by retrieving
examples with similar patterns, and the gains are consistent across LLMs of
varying sizes.Comment: 16 page
Query2doc: Query Expansion with Large Language Models
This paper introduces a simple yet effective query expansion approach,
denoted as query2doc, to improve both sparse and dense retrieval systems. The
proposed method first generates pseudo-documents by few-shot prompting large
language models (LLMs), and then expands the query with generated
pseudo-documents. LLMs are trained on web-scale text corpora and are adept at
knowledge memorization. The pseudo-documents from LLMs often contain highly
relevant information that can aid in query disambiguation and guide the
retrievers. Experimental results demonstrate that query2doc boosts the
performance of BM25 by 3% to 15% on ad-hoc IR datasets, such as MS-MARCO and
TREC DL, without any model fine-tuning. Furthermore, our method also benefits
state-of-the-art dense retrievers in terms of both in-domain and out-of-domain
results.Comment: 9 page
Learning to Rank in Generative Retrieval
Generative retrieval is a promising new paradigm in text retrieval that
generates identifier strings of relevant passages as the retrieval target. This
paradigm leverages powerful generation models and represents a new paradigm
distinct from traditional learning-to-rank methods. However, despite its rapid
development, current generative retrieval methods are still limited. They
typically rely on a heuristic function to transform predicted identifiers into
a passage rank list, which creates a gap between the learning objective of
generative retrieval and the desired passage ranking target. Moreover, the
inherent exposure bias problem of text generation also persists in generative
retrieval. To address these issues, we propose a novel framework, called LTRGR,
that combines generative retrieval with the classical learning-to-rank
paradigm. Our approach involves training an autoregressive model using a
passage rank loss, which directly optimizes the autoregressive model toward the
optimal passage ranking. This framework only requires an additional training
step to enhance current generative retrieval systems and does not add any
burden to the inference stage. We conducted experiments on three public
datasets, and our results demonstrate that LTRGR achieves state-of-the-art
performance among generative retrieval methods, indicating its effectiveness
and robustness
In-context Autoencoder for Context Compression in a Large Language Model
We propose the In-context Autoencoder (ICAE) for context compression in a
large language model (LLM). The ICAE has two modules: a learnable encoder
adapted with LoRA from an LLM for compressing a long context into a limited
number of memory slots, and a fixed decoder which is the target LLM that can
condition on the memory slots for various purposes. We first pretrain the ICAE
using both autoencoding and language modeling objectives on massive text data,
enabling it to generate memory slots that accurately and comprehensively
represent the original context. Then, we fine-tune the pretrained ICAE on a
small amount of instruct data to enhance its interaction with various prompts
for producing desirable responses. Our experimental results demonstrate that
the ICAE learned with our proposed pretraining and fine-tuning paradigm can
effectively produce memory slots with context compression, which can
be well conditioned on by the target LLM to respond to various prompts. The
promising results demonstrate significant implications of the ICAE for its
novel approach to the long context problem and its potential to reduce
computation and memory overheads for LLM inference in practice, suggesting
further research effort in context management for an LLM. Our code and data
will be released shortly.Comment: Work in progres
- …