Teaching Large Language Models to Self-Debug

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any feedback on the code correctness or error messages, the model is able to identify its mistakes by explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest label by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs

Large Language Models Can Be Easily Distracted by Irrelevant Context

Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information

Understanding Haemophilus parasuis infection in porcine spleen through a transcriptomics approach

<p>Abstract</p> <p>Background</p> <p><it>Haemophilus parasuis </it>(HPS) is an important swine pathogen that causes Glässer's disease, which is characterized by fibrinous polyserositis, meningitis and arthritis. The molecular mechanisms that underlie the pathogenesis of the disease remain poorly understood, particularly the resistance of porcine immune system to HPS invasion. In this study, we investigated the global changes in gene expression in the spleen following HPS infection using the Affymetrix Porcine Genechip™.</p> <p>Results</p> <p>A total of 931 differentially expressed (DE) transcripts were identified in the porcine spleen 7 days after HPS infection; of these, 92 unique genes showed differential expression patterns based on analysis using BLASTX and Gene Ontology. The DE genes involved in the immune response included genes for inflammasomes (<it>RETN</it>, <it>S100A8</it>, <it>S100A9</it>, <it>S100A12</it>), adhesion molecules (<it>CLDN3</it>, <it>CSPG2</it>, <it>CD44</it>, <it>LGALS8</it>), transcription factors (<it>ZBTB16</it>, <it>SLC39A14</it>, <it>CEBPD</it>, <it>CEBPB</it>), acute-phase proteins and complement (<it>SAA1</it>, <it>LTF</it>, <it>HP</it>, <it>C3</it>), differentiation genes for epithelial cells and keratinocytes (<it>TGM1</it>, <it>MS4A8B</it>, <it>CSTA</it>), and genes related to antigen processing and presentation (<it>HLA-B</it>, <it>HLA-DRB1</it>). Further immunostimulation analyses indicated that mRNA levels of <it>S100A8</it>, <it>S100A9</it>, and <it>S100A12 </it>in porcine PK-15 cells increased within 48 h and were sustained after administration of lipopolysaccharide (LPS) and Poly(I:C) respectively. In addition, mapping of DE genes to porcine health traits QTL regions showed that 70 genes were distributed in 7 different known porcine QTL regions. Finally, 10 DE genes were validated by quantitative PCR.</p> <p>Conclusion</p> <p>Our findings demonstrate previously unrecognized changes in gene transcription that are associated with HPS infection <it>in vivo</it>, and many potential cascades identified in the study clearly merit further investigation. Our data provide new clues to the nature of the immune response in mammals, and we have identified candidate genes that are related to resistance to HPS.</p

Compositional Semantic Parsing with Large Language Models

Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable large language models (LLMs) to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.Comment: Fixed metadata. No other change

Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models

We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide the reasoning steps, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L models and observe substantial performance gains on a wide range of challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU Physics and Chemistry by 7% and 11%, TimeQA by 27%, and MuSiQue by 7%

Universal Self-Consistency for Large Language Model Generation

Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs). However, self-consistency relies on the answer extraction process to aggregate multiple solutions, which is not applicable to free-form answers. In this work, we propose Universal Self-Consistency (USC), which leverages LLMs themselves to select the most consistent answer among multiple candidates. We evaluate USC on a variety of benchmarks, including mathematical reasoning, code generation, long-context summarization, and open-ended question answering. On open-ended generation tasks where the original self-consistency method is not applicable, USC effectively utilizes multiple samples and improves the performance. For mathematical reasoning, USC matches the standard self-consistency performance without requiring the answer formats to be similar. Finally, without access to execution results, USC also matches the execution-based voting performance on code generation

The Emerging Roles of the RNA Binding Protein QKI in Cardiovascular Development and Function

RNA binding proteins (RBPs) have a broad biological and physiological function and are critical in regulating pre-mRNA posttranscriptional processing, intracellular migration, and mRNA stability. QKI, also known as Quaking, is a member of the signal transduction and activation of RNA (STAR) family, which also belongs to the heterogeneous nuclear ribonucleoprotein K- (hnRNP K-) homology domain protein family. There are three major alternatively spliced isoforms, QKI-5, QKI-6, and QKI-7, differing in carboxy-terminal domains. They share a common RNA binding property, but each isoform can regulate pre-mRNA splicing, transportation or stability differently in a unique cell type-specific manner. Previously, QKI has been known for its important role in contributing to neurological disorders. A series of recent work has further demonstrated that QKI has important roles in much broader biological systems, such as cardiovascular development, monocyte to macrophage differentiation, bone metabolism, and cancer progression. In this mini-review, we will focus on discussing the emerging roles of QKI in regulating cardiac and vascular development and function and its potential link to cardiovascular pathophysiology

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.Comment: Preprin

Towards interpreting recurrent neural networks through probabilistic abstraction

National Research Foundation (NRF) Singapore under its AI Singapore Programm