11 research outputs found
KALA: Knowledge-Augmented Language Model Adaptation
Pre-trained language models (PLMs) have achieved remarkable success on
various natural language understanding tasks. Simple fine-tuning of PLMs, on
the other hand, might be suboptimal for domain-specific tasks because they
cannot possibly cover knowledge from all domains. While adaptive pre-training
of PLMs can help them obtain domain-specific knowledge, it requires a large
training cost. Moreover, adaptive pre-training can harm the PLM's performance
on the downstream task by causing catastrophic forgetting of its general
knowledge. To overcome such limitations of adaptive pre-training for PLM
adaption, we propose a novel domain adaption framework for PLMs coined as
Knowledge-Augmented Language model Adaptation (KALA), which modulates the
intermediate hidden representations of PLMs with domain knowledge, consisting
of entities and their relational facts. We validate the performance of our KALA
on question answering and named entity recognition tasks on multiple datasets
across various domains. The results show that, despite being computationally
efficient, our KALA largely outperforms adaptive pre-training. Code is
available at: https://github.com/Nardien/KALA/.Comment: NAACL 202
Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation
Language models have achieved impressive performances on dialogue generation
tasks. However, when generating responses for a conversation that requires
factual knowledge, they are far from perfect, due to an absence of mechanisms
to retrieve, encode, and reflect the knowledge in the generated responses. Some
knowledge-grounded dialogue generation methods tackle this problem by
leveraging facts from Knowledge Graphs (KGs); however, they do not guarantee
that the model utilizes a relevant piece of knowledge from the KG. To overcome
this limitation, we propose SUbgraph Retrieval-augmented GEneration (SURGE), a
framework for generating context-relevant and knowledge-grounded dialogues with
the KG. Specifically, our SURGE framework first retrieves the relevant subgraph
from the KG, and then enforces consistency across facts by perturbing their
word embeddings conditioned by the retrieved subgraph. Then, we utilize
contrastive learning to ensure that the generated texts have high similarity to
the retrieved subgraphs. We validate our SURGE framework on OpendialKG and
KOMODIS datasets, showing that it generates high-quality dialogues that
faithfully reflect the knowledge from KG.Comment: Preprint. Under revie
Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion
Large Language Models (LLMs) excel at tackling various natural language
tasks. However, due to the significant costs involved in re-training or
fine-tuning them, they remain largely static and difficult to personalize.
Nevertheless, a variety of applications could benefit from generations that are
tailored to users' preferences, goals, and knowledge. Among them is web search,
where knowing what a user is trying to accomplish, what they care about, and
what they know can lead to improved search experiences. In this work, we
propose a novel and general approach that augments an LLM with relevant context
from users' interaction histories with a search engine in order to personalize
its outputs. Specifically, we construct an entity-centric knowledge store for
each user based on their search and browsing activities on the web, which is
then leveraged to provide contextually relevant LLM prompt augmentations. This
knowledge store is light-weight, since it only produces user-specific aggregate
projections of interests and knowledge onto public knowledge graphs, and
leverages existing search log infrastructure, thereby mitigating the privacy,
compliance, and scalability concerns associated with building deep user
profiles for personalization. We then validate our approach on the task of
contextual query suggestion, which requires understanding not only the user's
current search context but also what they historically know and care about.
Through a number of experiments based on human evaluation, we show that our
approach is significantly better than several other LLM-powered baselines,
generating query suggestions that are contextually more relevant, personalized,
and useful
Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Large Language Models (LLMs) have shown promising performance in
knowledge-intensive reasoning tasks that require a compound understanding of
knowledge. However, deployment of the LLMs in real-world applications can be
challenging due to their high computational requirements and concerns on data
privacy. Previous studies have focused on building task-specific small language
models (LMs) by fine-tuning them with labeled data or distilling LLMs. However,
these approaches are ill-suited for knowledge-intensive reasoning tasks due to
the limited capacity of small LMs in memorizing the knowledge required.
Motivated by our theoretical analysis on memorization, we propose
Knowledge-Augmented Reasoning Distillation (KARD), a novel method that
fine-tunes small LMs to generate rationales with augmented knowledge retrieved
from an external knowledge base. Moreover, we further propose a neural reranker
to obtain documents relevant to rationale generation. We empirically show that
KARD significantly improves the performance of small T5 and Flan-T5 models on
the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE and
StrategyQA. Notably, our method makes the 250M models achieve superior
performance against the fine-tuned 3B models, having 12 times larger
parameters, on both MedQA-USMLE and StrategyQA benchmarks.Comment: Preprint. Under revie
Knowledge-Augmented Language Model Verification
Recent Language Models (LMs) have shown impressive capabilities in generating
texts with the knowledge internalized in parameters. Yet, LMs often generate
the factually incorrect responses to the given queries, since their knowledge
may be inaccurate, incomplete, and outdated. To address this problem, previous
works propose to augment LMs with the knowledge retrieved from an external
knowledge source. However, such approaches often show suboptimal text
generation performance due to two reasons: 1) the model may fail to retrieve
the knowledge relevant to the given query, or 2) the model may not faithfully
reflect the retrieved knowledge in the generated text. To overcome these, we
propose to verify the output and the knowledge of the knowledge-augmented LMs
with a separate verifier, which is a small LM that is trained to detect those
two types of errors through instruction-finetuning. Then, when the verifier
recognizes an error, we can rectify it by either retrieving new knowledge or
generating new text. Further, we use an ensemble of the outputs from different
instructions with a single verifier to enhance the reliability of the
verification processes. We validate the effectiveness of the proposed
verification steps on multiple question answering benchmarks, whose results
show that the proposed verifier effectively identifies retrieval and generation
errors, allowing LMs to provide more factually correct outputs. Our code is
available at https://github.com/JinheonBaek/KALMV.Comment: EMNLP 202
Graph Self-supervised Learning with Accurate Discrepancy Learning
Self-supervised learning of graph neural networks (GNNs) aims to learn an
accurate representation of the graphs in an unsupervised manner, to obtain
transferable representations of them for diverse downstream tasks. Predictive
learning and contrastive learning are the two most prevalent approaches for
graph self-supervised learning. However, they have their own drawbacks. While
the predictive learning methods can learn the contextual relationships between
neighboring nodes and edges, they cannot learn global graph-level similarities.
Contrastive learning, while it can learn global graph-level similarities, its
objective to maximize the similarity between two differently perturbed graphs
may result in representations that cannot discriminate two similar graphs with
different properties. To tackle such limitations, we propose a framework that
aims to learn the exact discrepancy between the original and the perturbed
graphs, coined as Discrepancy-based Self-supervised LeArning (D-SLA).
Specifically, we create multiple perturbations of the given graph with varying
degrees of similarity, and train the model to predict whether each graph is the
original graph or the perturbed one. Moreover, we further aim to accurately
capture the amount of discrepancy for each perturbed graph using the graph edit
distance. We validate our D-SLA on various graph-related downstream tasks,
including molecular property prediction, protein function prediction, and link
prediction tasks, on which ours largely outperforms relevant baselines.Comment: 9 page