27 research outputs found
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Despite their remarkable capabilities, large language models (LLMs) often
produce responses containing factual inaccuracies due to their sole reliance on
the parametric knowledge they encapsulate. Retrieval-Augmented Generation
(RAG), an ad hoc approach that augments LMs with retrieval of relevant
knowledge, decreases such issues. However, indiscriminately retrieving and
incorporating a fixed number of retrieved passages, regardless of whether
retrieval is necessary, or passages are relevant, diminishes LM versatility or
can lead to unhelpful response generation. We introduce a new framework called
Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's
quality and factuality through retrieval and self-reflection. Our framework
trains a single arbitrary LM that adaptively retrieves passages on-demand, and
generates and reflects on retrieved passages and its own generations using
special tokens, called reflection tokens. Generating reflection tokens makes
the LM controllable during the inference phase, enabling it to tailor its
behavior to diverse task requirements. Experiments show that Self-RAG (7B and
13B parameters) significantly outperforms state-of-the-art LLMs and
retrieval-augmented models on a diverse set of tasks. Specifically, Self-RAG
outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA,
reasoning and fact verification tasks, and it shows significant gains in
improving factuality and citation accuracy for long-form generations relative
to these models.Comment: 30 pages, 2 figures, 12 table
Learning Cross-Lingual IR from an English Retriever
We present DR.DECR (Dense Retrieval with Distillation-Enhanced Cross-Lingual
Representation), a new cross-lingual information retrieval (CLIR) system
trained using multi-stage knowledge distillation (KD). The teacher of DR.DECR
relies on a highly effective but computationally expensive two-stage inference
process consisting of query translation and monolingual IR, while the student,
DR.DECR, executes a single CLIR step. We teach DR.DECR powerful multilingual
representations as well as CLIR by optimizing two corresponding KD objectives.
Learning useful representations of non-English text from an English-only
retriever is accomplished through a cross-lingual token alignment algorithm
that relies on the representation capabilities of the underlying multilingual
encoders. In both in-domain and zero-shot out-of-domain evaluation, DR.DECR
demonstrates far superior accuracy over direct fine-tuning with labeled CLIR
data. It is also the best single-model retriever on the XOR-TyDi benchmark at
the time of this writing.Comment: 9 page