Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Mathematical reasoning is a challenging task for large language models (LLMs), while the scaling relationship of it with respect to LLM capacity is under-explored. In this paper, we investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that pre-training loss is a better indicator of the model's performance than the model's parameter count. We apply supervised fine-tuning (SFT) with different amounts of supervised data and empirically find a log-linear relation between data amount and model performance, and we find better models improve less with enlarged supervised datasets. To augment more data samples for improving model performances without any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT uses supervised models to generate and collect correct reasoning paths as augmented fine-tuning datasets. We find with augmented samples containing more distinct reasoning paths, RFT improves mathematical reasoning performance more for LLMs. We also find RFT brings more improvement for less performant LLMs. Furthermore, we combine rejection samples from multiple models which push LLaMA-7B to an accuracy of 49.3\% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9\% significantly.Comment: Working in Progres

Prognostic and therapeutic significance of microbial cell-free DNA in plasma of people with acutely decompensated cirrhosis

BACKGROUND AND AIMS: Although the effect of bacterial infection on cirrhosis has been well-described, the effect of non-hepatotropic virus (NHV) infection is unknown. This study evaluated the genome fragments of circulating microorganisms using metagenomic next-generation sequencing (mNGS) in cirrhosis patients with acute decompensation (AD), focusing on NHVs and related the findings to clinical outcomes. METHODS: Plasma mNGS was performed in 129 cirrhosis patients with AD in study cohort. Ten healthy volunteers and 20, 39, and 81 patients with stable cirrhosis, severe sepsis and hematological malignancies, respectively, were enrolled as controls. Validation assays for human cytomegalovirus (CMV) reactivation in a validation cohort (n = 58) were performed and exploratory treatment instituted. RESULTS: In study cohort, 188 microorganisms were detected in 74.4% (96/129) patients, including viruses (58.0%), bacteria (34.1%), fungi (7.4%) and chlamydia (0.5%). Patients with AD had an NHV signature, and CMV was the most frequent NHV, which correlated with the clinical effect of empirical antibiotic treatment, progression to acute-on-chronic liver failure (ACLF), and 90-day mortality. The NHV signature in ACLF patients was similar to patients with sepsis and hematological malignancies. The treatable NHV, CMV was detected in 24.1% (14/58) patients in the validation cohort. Of the 14 cases with detectable CMV by mNGS, 9 were further validated by DNA RT-PCR or pp65 antigenemia testing. Three patients with CMV reactivation received ganciclovir therapy in exploratory manner with clinical resolutions. CONCLUSIONS: The results of this study suggests that NHVs may have a pathogenic role in complicating the course of AD. Further validation is needed to define whether this should be incorporated in the routine management of AD patients. IMPACT AND IMPLICATIONS: ●Cirrhosis patients with acute decompensation have a non-hepatotropic virus (NHV) signature, which is similar to that in sepsis and hematological malignancies patients. ●The detected viral signature had clinical correlates, including clinical efficacy of empirical antibiotic treatment, progression to acute-on-chronic liver failure and short-term mortality. ●The treatable NHV, CMV reactivation may be involved in the clinical outcomes of decompensated cirrhosis. ●Routine screening for NHVs, especially CMV, may be useful for the management of patients with acutely decompensated cirrhosis

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5

Comprehensive Analysis of the Structure and Function of Peptide:N-Glycanase 1 and Relationship with Congenital Disorder of Deglycosylation

The cytosolic PNGase (peptide:N-glycanase), also known as peptide-N4-(N-acetyl-β-glucosaminyl)-asparagine amidase, is a well-conserved deglycosylation enzyme (EC 3.5.1.52) which catalyzes the non-lysosomal hydrolysis of an N(4)-(acetyl-β-d-glucosaminyl) asparagine residue (Asn, N) into a N-acetyl-β-d-glucosaminyl-amine and a peptide containing an aspartate residue (Asp, D). This enzyme (NGLY1) plays an essential role in the clearance of misfolded or unassembled glycoproteins through a process named ER-associated degradation (ERAD). Accumulating evidence also points out that NGLY1 deficiency can cause an autosomal recessive (AR) human genetic disorder associated with abnormal development and congenital disorder of deglycosylation. In addition, the loss of NGLY1 can affect multiple cellular pathways, including but not limited to NFE2L1 pathway, Creb1/Atf1-AQP pathway, BMP pathway, AMPK pathway, and SLC12A2 ion transporter, which might be the underlying reasons for a constellation of clinical phenotypes of NGLY1 deficiency. The current comprehensive review uncovers the NGLY1’ssdetailed structure and its important roles for participation in ERAD, involvement in CDDG and potential treatment for NGLY1 deficiency

GAAD: A Gene and Autoimmiune Disease Association Database

Autoimmune diseases (ADs) arise from an abnormal immune response of the body against substances and tissues normally present in the body. More than a hundred of ADs have been described in the literature so far. Although their etiology remains largely unclear, various types of ADs tend to share more associated genes with other types of ADs than with non-AD types. Here we present GAAD, a gene and AD association database. In GAAD, we collected 44,762 associations between 49 ADs and 4249 genes from public databases and MEDLINE documents. We manually verified the associations to ensure the quality and credibility. We reconstructed and recapitulated the relationships among ADs using their shared genes, which further validated the quality of our data. We also provided a list of significantly co-occurring gene pairs among ADs; with embedded tools, users can query gene co-occurrences and construct customized co-occurrence network with genes of interest. To make GAAD more straightforward to experimental biologists and medical scientists, we extracted additional information describing the associations through text mining, including the putative diagnostic value of the associations, type and position of gene polymorphisms, expression changes of implicated genes, as well as the phenotypical consequences, and grouped the associations accordingly. GAAD is freely available at http://gaad.medgenius.info. Keywords: Autoimmune diseases, Disease–gene association, Database, Text minin

Visual and optical quality outcomes of SMILE and FS-LASIK for myopia in the very early phase after surgery

Abstract Background Small incision lenticule extraction (SMILE) and femtosecond laser-assisted in situ keratomileusis (FS-LASIK) are frequently used to treat myopia. However, little is known about the impact on recovery of these approaches in the very early postsurgical phase (within 24 h). Methods To compare the efficacy of these two procedures for the treatment of myopia in the early phase after surgery, differences in visual acuity, OSI (objective scattering index), cutoff for modulation transfer function (MTF), and SR (Strehl ratio) between SMILE and FS-LASIK were evaluated at 0, 2, 4 and 24 h postoperatively using two-way analysis of variance (ANOVA). Results No significant differences between SMILE and FS-LASIK in the MTF cutoff and SR were found (p > 0.05). However, at 2 h and 4 h after surgery, OSI values in the SMILE group were significantly higher than those in the FS-LASIK group, and visual acuity scores in the SMILE group were significantly poorer than those in the FS-LASIK group (p < 0.05). Regarding subjective symptoms, the number of patients complaining of eye dryness, blurred vision, foreign body sensation and eye soreness in the SMILE group were lower than the number in the FS-LASIK group. Conclusions In conclusion, visual and optical quality outcomes of FS-LASIK for myopia were better than those of SMILE in the very early phase after surgery, a difference that is attributable to the formation of interface haze. Trial registration ChiCTR1900021451

Selection for energy efficiency drives strand-biased gene distribution in prokaryotes

Lagging-strand genes accumulate more deleterious mutations. Genes are thus preferably located on the leading strand, an observation known as strand-biased gene distribution (SGD). Despite of this mechanistic understanding, a satisfactory quantitative model is still lacking. Replication-transcription-collisions induce stalling of the replication machinery, expose DNA to various attacks, and are followed by error-prone repairs. We found that mutational biases in non-transcribed regions can explain similar to 71% of the variations in SGDs in 1,552 genomes, supporting the mutagenesis origin of SGD. Mutational biases introduce energetically cheaper nucleotides on the lagging strand, and result in more expensive protein products; consistently, the cost difference between the two strands explains similar to 50% of the variance in SGDs. Protein costs decrease with increasing gene expression. At similar expression levels, protein products of leading-strand genes are generally cheaper than lagging-strand genes; however, highly-expressed lagging genes are still cheaper than lowly-expressed leading genes. Selection for energy efficiency thus drives some genes to the leading strand, especially those highly expressed and essential, but certainly not all genes. Stronger mutational biases are often associated with low-GC genomes; as low-GC genes encode expensive proteins, low-GC genomes thus tend to have stronger SGDs to alleviate the stronger pressure on efficient energy usage

OGEE v2: an update of the online gene essentiality database with special focus on differentially essential genes in human cancer cell lines

OGEE is an Online GEne Essentiality database. To enhance our understanding of the essentiality of genes, in OGEE we collected experimentally tested essential and non-essential genes, as well as associated gene properties known to contribute to gene essentiality. We focus on large-scale experiments, and complement our data with text-mining results. We organized tested genes into data sets according to their sources, and tagged those with variable essentiality statuses across data sets as conditionally essential genes, intending to highlight the complex interplay between gene functions and environments/experimental perturbations. Developments since the last public release include increased numbers of species and gene essentiality data sets, inclusion of non-coding essential sequences and genes with intermediate essentiality statuses. In addition, we included 16 essentiality data sets from cancer cell lines, corresponding to 9 human cancers; with OGEE, users can easily explore the shared and differentially essential genes within and between cancer types. These genes, especially those derived from cell lines that are similar to tumor samples, could reveal the oncogenic drivers, paralogous gene expression pattern and chromosomal structure of the corresponding cancer types, and can be further screened to identify targets for cancer therapy and/or new drug development. OGEE is freely available at http://ogee.medgenius.info