Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates

Calibration strengthens the trustworthiness of black-box models by producing better accurate confidence estimates on given examples. However, little is known about if model explanations can help confidence calibration. Intuitively, humans look at important features attributions and decide whether the model is trustworthy. Similarly, the explanations can tell us when the model may or may not know. Inspired by this, we propose a method named CME that leverages model explanations to make the model less confident with non-inductive attributions. The idea is that when the model is not highly confident, it is difficult to identify strong indications of any class, and the tokens accordingly do not have high attribution scores for any class and vice versa. We conduct extensive experiments on six datasets with two popular pre-trained language models in the in-domain and out-of-domain settings. The results show that CME improves calibration performance in all settings. The expected calibration errors are further reduced when combined with temperature scaling. Our findings highlight that model explanations can help calibrate posterior estimates.Comment: EMNLP 202

Prompt-based Text Entailment for Low-Resource Named Entity Recognition

Pre-trained Language Models (PLMs) have been applied in NLP tasks and achieve promising results. Nevertheless, the fine-tuning procedure needs labeled data of the target domain, making it difficult to learn in low-resource and non-trivial labeled scenarios. To address these challenges, we propose Prompt-based Text Entailment (PTE) for low-resource named entity recognition, which better leverages knowledge in the PLMs. We first reformulate named entity recognition as the text entailment task. The original sentence with entity type-specific prompts is fed into PLMs to get entailment scores for each candidate. The entity type with the top score is then selected as final label. Then, we inject tagging labels into prompts and treat words as basic units instead of n-gram spans to reduce time complexity in generating candidates by n-grams enumeration. Experimental results demonstrate that the proposed method PTE achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD dataset in low-resource settings.Comment: COLING 202

Data-driven method to learn the most probable transition pathway and stochastic differential equations

Transition phenomena between metastable states play an important role in complex systems due to noisy fluctuations. In this paper, the physics informed neural networks (PINNs) are presented to compute the most probable transition pathway. It is shown that the expected loss is bounded by the empirical loss. And the convergence result for the empirical loss is obtained. Then, a sampling method of rare events is presented to simulate the transition path by the Markovian bridge process. And we investigate the inverse problem to extract the stochastic differential equation from the most probable transition pathway data and the Markovian bridge process data, respectively. Finally, several numerical experiments are presented to verify the effectiveness of our methods

Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation

Large language models (LLMs) have been widely used in various applications but are known to suffer from issues related to untruthfulness and toxicity. While parameter-efficient modules (PEMs) have demonstrated their effectiveness in equipping models with new skills, leveraging PEMs for deficiency unlearning remains underexplored. In this work, we propose a PEMs operation approach, namely Extraction-before-Subtraction (Ext-Sub), to enhance the truthfulness and detoxification of LLMs through the integration of ``expert'' PEM and ``anti-expert'' PEM. Remarkably, even anti-expert PEM possess valuable capabilities due to their proficiency in generating fabricated content, which necessitates language modeling and logical narrative competence. Rather than merely negating the parameters, our approach involves extracting and eliminating solely the deficiency capability within anti-expert PEM while preserving the general capabilities. To evaluate the effectiveness of our approach in terms of truthfulness and detoxification, we conduct extensive experiments on LLMs, encompassing additional abilities such as language modeling and mathematical reasoning. Our empirical results demonstrate that our approach effectively improves truthfulness and detoxification, while largely preserving the fundamental abilities of LLMs

Temporal Knowledge Question Answering via Abstract Reasoning Induction

In this paper, we tackle the significant challenge of temporal knowledge reasoning in Large Language Models (LLMs), an area where such models frequently encounter difficulties. These difficulties often result in the generation of misleading or incorrect information, primarily due to their limited capacity to process evolving factual knowledge and complex temporal logic. In response, we propose a novel, constructivism-based approach that advocates for a paradigm shift in LLM learning towards an active, ongoing process of knowledge synthesis and customization. At the heart of our proposal is the Abstract Reasoning Induction ARI framework, which divides temporal reasoning into two distinct phases: Knowledge-agnostic and Knowledge-based. This division aims to reduce instances of hallucinations and improve LLMs' capacity for integrating abstract methodologies derived from historical data. Our approach achieves remarkable improvements, with relative gains of 29.7\% and 9.27\% on two temporal QA datasets, underscoring its efficacy in advancing temporal reasoning in LLMs. The code will be released at https://github.com/czy1999/ARI.Comment: 17 pages, 10 figure

ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination

As ChatGPT and GPT-4 spearhead the development of Large Language Models (LLMs), more researchers are investigating their performance across various tasks. But more research needs to be done on the interpretability capabilities of LLMs, that is, the ability to generate reasons after an answer has been given. Existing explanation datasets are mostly English-language general knowledge questions, which leads to insufficient thematic and linguistic diversity. To address the language bias and lack of medical resources in generating rationales QA datasets, we present ExplainCPE (over 7k instances), a challenging medical benchmark in Simplified Chinese. We analyzed the errors of ChatGPT and GPT-4, pointing out the limitations of current LLMs in understanding text and computational reasoning. During the experiment, we also found that different LLMs have different preferences for in-context learning. ExplainCPE presents a significant challenge, but its potential for further investigation is promising, and it can be used to evaluate the ability of a model to generate explanations. AI safety and trustworthiness need more attention, and this work makes the first step to explore the medical interpretability of LLMs.The dataset is available at https://github.com/HITsz-TMG/ExplainCPE.Comment: EMNLP 2023 Finding

Pregnane X receptor is required for interleukin-6-mediated down-regulation of cytochrome P450 3A4 in human hepatocytes

Cytochrome P450 3A4 (CYP3A4) is the most abundant cytochrome P450 enzyme in human liver and metabolizes more than 60% of prescribed drugs in human body. Patients with liver conditions such as cirrhosis show increased secretion of cytokines (e.g., interleukin-6) and decreased capacity of oxidation of many drugs. In this study, we provided molecular evidence that cytokine secretion directly contributed to the decreased capacity of oxidative biotransformation in human liver. After human hepatocytes were treated with IL-6, the expression of CYP3A4 decreased at both mRNA and protein levels, so did the CYP3A4 enzymatic activity. Meanwhile, the repression of CYP3A4 by IL-6 occurred after the decrease of pregnane X receptor (PXR) in human hepatocytes. The PXR-overexpressed cells (transfected with human PXR) increased the CYP3A4 mRNA level, and the repression of CYP3A4 by IL-6 was greater in the PXR-overexpressed cells than in the control cells. Further, PXR knockdown (transfected with siPXR construct) decreased the CYP3A4 mRNA level with less repression by IL-6 than in the control cells transfected with corresponding vector. Collectively, our study suggests that PXR is necessary for IL-6-mediated repression of the CYP3A4 expression in human hepatocytes

Serum from patients with ankylosing spondylitis can increase PPARD, fra-1, MMP7, OPG and RANKL expression in MG63 cells

OBJECTIVES: To explore the effects of serum from patients with ankylosing spondylitis on the canonical Wnt/β-catenin pathway and to assess whether the serum has an osteogenic effect in MG63 cells. METHODS: MG63 cells were cultured with serum from 45 ankylosing spondylitis patients, 30 healthy controls, or 45 rheumatoid arthritis patients. The relative PPARD, fra-1, MMP7, OPG and RANKL mRNA levels were measured using quantitative real-time polymerase chain reaction. Associations between gene expression and patient demographics and clinical assessments were then analyzed. RESULTS: MG63 cells treated with serum from ankylosing spondylitis patients had higher PPARD, fra-1, MMP7 and OPG gene expression than did cells treated with serum from controls or rheumatoid arthritis patients (all

Molecular characteristics and function of elliptical Kiwifruit

Article describes study analyzing the chemical components of elliptical kiwifruit (Actinidia chinensis Planch) using Fourier transform infrared spectroscopy (FT-IR) and gas chromatography mass spectrometry (GC–MS) technologies

Long-term outcomes of infantile spasms in children treated with ketogenic diet therapy in combination with anti-seizure medications in a resource-limited region

ObjectiveDespite numerous guidelines, the overall outcome of infantile spasms is poor, with only a small number of patients being able to attend school. The purpose of this study was to investigate long-term outcomes. Patients had poor access to the recommended first-line anti-seizure medications (ASMs), such as hormones (corticotropin or prednisolone/prednisone) and vigabatrin, and their alternative treatment was other ASMs and a ketogenic diet.MethodsPatients suffering from infantile spasms who had at least 2 years of medical records in the electronic medical record system between January 2014 and August 2022 were included in this study. Patient information was retrospectively reviewed. All patients had received ketogenic diet therapy (mainly classical ketogenic diet therapy). The ketogenic diet therapy was combined with ASMs not used as first-line therapies. The primary endpoint outcome measure was the number of patients with seizure freedom. The secondary measures included the duration of ketogenic diet therapy, choice of ASMs, and patient development at the last visit.ResultsA total of 177 patients with infantile spasms were included, and 152 (86%) of them had seizure freedom. The median duration from the first to the last hospital visit was 53.27 months, and the number of visits was 47.00. The median age at the initial hospital visit was 8.00 months, and the median age at initiation of the ketogenic diet was 17.73 months. At the last visit, the proportions of patients with neurodevelopmental delay, developmental epileptic encephalopathy, drug-resistant epilepsy, and generalized seizures increased significantly. The frequently used ASMs were topiramate, valproic acid, levetiracetam, nitrazepam, and vitamin B6 injection, while the recommended first-line drugs corticotropin and vigabatrin were rarely selected. The study duration of 9.5 years was divided into three periods but the prescription of ASMs did not change significantly between these periods.ConclusionsAlthough the seizure freedom rate was high with ketogenic diet therapy combined with non-standard ASMs, the patients had a significant neurodevelopmental delay at the last visit, which was, however, similar to that of standard treatment. To improve the outcomes of infantile spasms, multicenter clinical trials of the ketogenic diet as a first-line treatment in combination with non-standard ASMs are needed