ModuLoRA: Finetuning 3-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers

We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 3-bit or 4-bit precision on as little as one 48GB GPU. Our method, modular low-rank adaptation (ModuLoRA), integrates any user-specified weight quantizer with finetuning via low-rank adapters (LoRAs). Our approach relies on a simple quantization-agnostic backward pass that adaptively materializes low-precision LLM weights from a custom black-box quantization module. This approach enables finetuning 3-bit LLMs for the first time--leveraging state-of-the-art 3-bit OPTQ quantization often outperforms finetuning that relies on less sophisticated 4-bit and 8-bit methods. In our experiments, ModuLoRA attains competitive performance on text classification, natural language infernece, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-the-art ROUGE score on a popular summarization task. We release ModuLoRA together with a series of low-precision models--including the first family of 3-bit instruction following Alpaca LLMs--as part of LLMTOOLS, a user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs

From Static to Dynamic Structures: Improving Binding Affinity Prediction with a Graph-Based Deep Learning Model

Accurate prediction of the protein-ligand binding affinities is an essential challenge in the structure-based drug design. Despite recent advance in data-driven methods in affinity prediction, their accuracy is still limited, partially because they only take advantage of static crystal structures while the actual binding affinities are generally depicted by the thermodynamic ensembles between proteins and ligands. One effective way to approximate such a thermodynamic ensemble is to use molecular dynamics (MD) simulation. Here, we curated an MD dataset containing 3,218 different protein-ligand complexes, and further developed Dynaformer, which is a graph-based deep learning model. Dynaformer was able to accurately predict the binding affinities by learning the geometric characteristics of the protein-ligand interactions from the MD trajectories. In silico experiments demonstrated that our model exhibits state-of-the-art scoring and ranking power on the CASF-2016 benchmark dataset, outperforming the methods hitherto reported. Moreover, we performed a virtual screening on the heat shock protein 90 (HSP90) using Dynaformer that identified 20 candidates and further experimentally validated their binding affinities. We demonstrated that our approach is more efficient, which can identify 12 hit compounds (two were in the submicromolar range), including several newly discovered scaffolds. We anticipate this new synergy between large-scale MD datasets and deep learning models will provide a new route toward accelerating the early drug discovery process.Comment: totally reorganize the texts and figure

M2^2Hub: Unlocking the Potential of Machine Learning for Materials Discovery

We introduce M$^2$Hub, a toolkit for advancing machine learning in materials discovery. Machine learning has achieved remarkable progress in modeling molecular structures, especially biomolecules for drug discovery. However, the development of machine learning approaches for modeling materials structures lag behind, which is partly due to the lack of an integrated platform that enables access to diverse tasks for materials discovery. To bridge this gap, M$^2$Hub will enable easy access to materials discovery tasks, datasets, machine learning methods, evaluations, and benchmark results that cover the entire workflow. Specifically, the first release of M$^2$Hub focuses on three key stages in materials discovery: virtual screening, inverse design, and molecular simulation, including 9 datasets that covers 6 types of materials with 56 tasks across 8 types of material properties. We further provide 2 synthetic datasets for the purpose of generative tasks on materials. In addition to random data splits, we also provide 3 additional data partitions to reflect the real-world materials discovery scenarios. State-of-the-art machine learning methods (including those are suitable for materials structures but never compared in the literature) are benchmarked on representative tasks. Our codes and library are publicly available at https://github.com/yuanqidu/M2Hub

RNA-Seq Analyses of the Role of miR-21 in Acute Pancreatitis

Background/Aims: Our previous study demonstrated that a deficiency of microRNA 21 (miR-21) protects mice from acute pancreatitis, yet the underlying molecular networks associated with miR-21 in pancreatitis and pancreatitis-associated lung injury remain unexplored. Methods: We used next generation sequencing to analyze gene expression profiles of pancreatic tissues from wild-type (WT) and miR-21 knockout (KO) mice treated with caerulein by using a 1-day treatment protocol. The Database for Annotation, Visualization, and Integrated Discovery gene annotation tool and Ingenuity Pathway Analysis were used to analyze the molecular pathways, while quantitative real-time PCR, western blotting, and immunohistochemistry were used to explore the molecular mechanisms. Results: We identified 152 differentially expressed genes (DEGs) in pancreata between WT and KO mice treated with caerulein. Cellular biogenesis and metabolism were the major pathways affected between WT and KO mice, whereas cell death and inflammatory response discriminated between WT and KO mice under acute pancreatitis. We validated 16 DEGs, consisting of 6 upregulated genes and 10 downregulated genes, involved in pancreatic injury. In particular, the upregulation of Pias3 and downregulation of Hmgb1 in KO pancreata coincided with a reduced severity of pancreatitis. In addition, we found Hmgb1 stimulation resulted in the overexpression of miR-21 in peripheral blood mononuclear cells, and deletion of miR-21 led to a reduction of caerulein-induced acute pancreatitis-associated lung injury by repressing Hmgb1 expression. Conclusion: Our data support the hypothesis that miR-21 modulates the inflammatory response during acute pancreatitis through the upregulation of Pias3 and downregulation of Hmgb1. Our findings further underscore a role for miR-21 in the promotion of acute pancreatitis

Thyroid Hormone Promotes Remodeling of Coronary Resistance Vessels

Low thyroid hormone (TH) function has been linked to impaired coronary blood flow, reduced density of small arterioles, and heart failure. Nonetheless, little is known about the mechanisms by which THs regulate coronary microvascular remodeling. The current study examined the initial cellular events associated with coronary remodeling induced by triiodothyronine (T3) in hypothyroid rats. Rats with established hypothyroidism, eight weeks after surgical thyroidectomy (TX), were treated with T3 for 36 or 72 hours. The early effects of T3 treatment on coronary microvasculature were examined morphometrically. Gene expression changes in the heart were assessed by quantitative PCR Array. Hypothyroidism resulted in arteriolar atrophy in the left ventricle. T3 treatment rapidly induced small arteriolar muscularization and, within 72 hours, restored arteriolar density to control levels. Total length of the capillary network was not affected by TX or T3 treatment. T3 treatment resulted in the coordinate regulation of Angiopoietin 1 and 2 expression. The response of Angiopoietins was consistent with vessel enlargement. In addition to the well known effects of THs on vasoreactivity, these results suggest that THs may affect function of small resistance arteries by phenotypic remodeling of vascular smooth muscle cells (VSMC)

an consumers' attitudes towards the safety of milk powder after the melamine scandal in 2008 and the factors influencing the attitudes

Purpose – This paper aims to analyze the factors that influence urban consumers' attitudes towards food safety after the melamine scandal. Design/methodology/approach – Based on the research about the attitudes of urban consumers in Nanjing towards the safety of milk powder after the melamine scandal in 2008, this paper adopts the ordered logit model to test which factors significantly influence consumers' attitudes. Findings – The findings suggest that: first, there is a common concern among consumers about the safety of milk powder after the melamine scandal; second, according to the research, the concern is in inverse relation to the level of educational attainment, consumers' awareness of food safety incidents and their opinion of governments' action after the incident. Moreover, those who always have a concern about the safety of the alternatives to milk powder are more easily affected. Originality/value – Different from other researches, the paper focuses on consumers' attitudes towards food safety by studying a specific case, namely the melamine scandal.China, Consumer behaviour, Food safety, Public opinion

Healthy diet and food system transformation in China

The Chinese food system has expanded its focus from aiming to solve food problems to tackling current health and environmental issues. The Chinese diet has increased in quantity and improved in safety, but there is still room for improvement in terms of health and sustainability. This study used Chinese dietary data provided by the Global Diet Database to analyze the changes in China’s dietary structure from 1990 to 2018 and highlight differences in urban and rural areas and across education levels. Findings show that the intake of food and beverage, macronutrients, and micronutrients in urban areas is higher than in rural areas. The difference in food and beverage intake between urban and rural areas is significant. The dietary gap between urban and rural areas has gradually widened. The difference in food and beverage and macronutrient intake across education levels is significant, but the difference in micronutrient intake is not significant. The gap in dietary structure across educational levels is relatively stable. These results indicate that the dietary structures of different groups in China are uncoordinated. We propose policies covering agricultural production, supply chain infrastructure, public institutions, education, and public awareness to build a sustainable food system with a healthy dietary pattern