KALA: Knowledge-Augmented Language Model Adaptation

Pre-trained language models (PLMs) have achieved remarkable success on various natural language understanding tasks. Simple fine-tuning of PLMs, on the other hand, might be suboptimal for domain-specific tasks because they cannot possibly cover knowledge from all domains. While adaptive pre-training of PLMs can help them obtain domain-specific knowledge, it requires a large training cost. Moreover, adaptive pre-training can harm the PLM's performance on the downstream task by causing catastrophic forgetting of its general knowledge. To overcome such limitations of adaptive pre-training for PLM adaption, we propose a novel domain adaption framework for PLMs coined as Knowledge-Augmented Language model Adaptation (KALA), which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts. We validate the performance of our KALA on question answering and named entity recognition tasks on multiple datasets across various domains. The results show that, despite being computationally efficient, our KALA largely outperforms adaptive pre-training. Code is available at: https://github.com/Nardien/KALA/.Comment: NAACL 202

Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation

Language models have achieved impressive performances on dialogue generation tasks. However, when generating responses for a conversation that requires factual knowledge, they are far from perfect, due to an absence of mechanisms to retrieve, encode, and reflect the knowledge in the generated responses. Some knowledge-grounded dialogue generation methods tackle this problem by leveraging facts from Knowledge Graphs (KGs); however, they do not guarantee that the model utilizes a relevant piece of knowledge from the KG. To overcome this limitation, we propose SUbgraph Retrieval-augmented GEneration (SURGE), a framework for generating context-relevant and knowledge-grounded dialogues with the KG. Specifically, our SURGE framework first retrieves the relevant subgraph from the KG, and then enforces consistency across facts by perturbing their word embeddings conditioned by the retrieved subgraph. Then, we utilize contrastive learning to ensure that the generated texts have high similarity to the retrieved subgraphs. We validate our SURGE framework on OpendialKG and KOMODIS datasets, showing that it generates high-quality dialogues that faithfully reflect the knowledge from KG.Comment: Preprint. Under revie

Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks

Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge. However, deployment of the LLMs in real-world applications can be challenging due to their high computational requirements and concerns on data privacy. Previous studies have focused on building task-specific small language models (LMs) by fine-tuning them with labeled data or distilling LLMs. However, these approaches are ill-suited for knowledge-intensive reasoning tasks due to the limited capacity of small LMs in memorizing the knowledge required. Motivated by our theoretical analysis on memorization, we propose Knowledge-Augmented Reasoning Distillation (KARD), a novel method that fine-tunes small LMs to generate rationales with augmented knowledge retrieved from an external knowledge base. Moreover, we further propose a neural reranker to obtain documents relevant to rationale generation. We empirically show that KARD significantly improves the performance of small T5 and Flan-T5 models on the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE and StrategyQA. Notably, our method makes the 250M models achieve superior performance against the fine-tuned 3B models, having 12 times larger parameters, on both MedQA-USMLE and StrategyQA benchmarks.Comment: Preprint. Under revie

Knowledge-Augmented Language Model Verification

Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning. Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs. Our code is available at https://github.com/JinheonBaek/KALMV.Comment: EMNLP 202

Urodynamic and Histological Changes in a Sterile Rabbit Vesicoureteral Reflux Model

This study aimed to investigate pressure changes of renal pelvis and histological change of kidneys in a surgically induced sterile rabbit vesicoureteral reflux (VUR) model. Five rabbits served as a control group, 7 as the sham-operated group, and 8 served as the VUR group. Three weeks later, urodynamic studies were performed, and histological examinations evaluated degree of inflammation, fibrosis, and tubular damage in the kidneys. At a low infusion rate, renal pelvic pressure in the VUR group was stable until late filling phase and then increased slightly. At a high infusion rate, the renal pelvic pressures of the sham-operated and control groups were stable until late filling phase and then increased slightly, whereas the renal pelvic pressure in the VUR group steadily increased from mid filling phase. Focal thinning of the tubular epithelium and interstitial widening were observed in certain cortical areas of refluxing kidneys, without inflammatory cell infiltration. Obvious changes in the mean diameters of distal tubules and extracellular matrix volume fractions were observed in two highly refluxing kidneys. High pressure reflux with bladder instability may result in renal cortical changes

Lessons learned over a decade of pediatric robotic ureteral reimplantation

The da Vinci robotic system has improved surgeon dexterity, ergonomics, and visualization to allow for a minimally invasive option for complex reconstructive procedures in children. Over the past decade, robot-assisted laparoscopic ureteral reimplantation (RALUR) has become a viable minimally invasive surgical option for pediatric vesicoureteral reflux (VUR). However, higher-thanexpected complication rates and suboptimal reflux resolution rates at some centers have also been reported. The heterogeneity of surgical outcomes may arise from the inherent and underestimated complexity of the RALUR procedure that may justify its reclassification as a complex reconstructive procedure and especially for robotic surgeons early in their learning curve. Currently, no consensus exists on the role of RALUR for the surgical management of VUR. High success rates and low major complication rates are the expected norm for the current gold standard surgical option of open ureteral reimplantation. Similar to how robot-assisted laparoscopic surgery has gradually replaced open surgery as the most utilized option for prostatectomy in prostate cancer patients, RALUR may become a higher utilized surgical option in children with VUR if the adoption of standardized surgical techniques that have been associated with optimal outcomes can be adopted during the second decade of RALUR. A future standard of RALUR for children with VUR whose parents seek a minimally invasive surgical option can arise if widespread achievement of high success rates and low major complication rates can be obtained, similar to the replacement of open surgery with robot-assisted laparoscopic radical prostectomy as the new strandard for men with prostate cancer

Simple but Effective Way To Enhance Photoelectrochemical Solar-Water-Splitting Performance of ZnO Nanorod Arrays: Charge-Trapping Zn(OH)2 Annihilation and Oxygen Vacancy Generation by Vacuum Annealing

This study presents an effective and the simplest method to substantially improve the photoelectrochemical water-splitting ability of hydrothermally grown ZnO nanorod arrays (NRAs). In the hydrothermal growth of ZnO NRAs, unwanted Zn(OH)(2) species are formed, which act as trapping sites of photoexcited charges. We found that those inherent charge-trapping sites could be annihilated by the desorption of the hydroxyl groups upon vacuum annealing above 200 degrees C, which resulted in an enhancement of the charge-separation efficiency and photocurrent density. Another drastic increase in the photocurrent density occurred when ZnO NRAs were treated with annealing at higher temperature (700 degrees C), which can be attributed to,the introduced oxygen vacancies acting, as shallow donors in the ZnO crystal lattice. The removal of the charge-trapping Zn(OH)(2) and the generation of oxygen vacancies were confirmed by photoluminescence (PL) and XPS analyses. The ZnO NRAs treated by this simple method yield a photocurrent density of 600 mu A/cm(2) at 1.23 V-RHE under 1 sun illumination, which is 20 times higher than that obtained from as-grown ZnO NRAs. This, study presents a highly efficient way of increasing the bulk electric conductivity and photoelectrochemical activity of metal oxide nanorods, without requiring the introduction of any extrinsic dopants.1122sciescopu