Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization

Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern. Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase. To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) - a simple yet effective method that combines the advantages of PEFT with quantized LLMs. By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage. We employ PEQA-tuning for task-specific adaptation on LLMs with up to 65 billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.Comment: Published at NeurIPS 2023. Camera-ready versio

Onion peel extracts ameliorate hyperglycemia and insulin resistance in high fat diet/streptozotocin-induced diabetic rats

<p>Abstract</p> <p>Background</p> <p>Quercetin derivatives in onions have been regarded as the most important flavonoids to improve diabetic status in cells and animal models. The present study was aimed to examine the hypoglycemic and insulin-sensitizing capacity of onion peel extract (OPE) containing high quercetin in high fat diet/streptozotocin-induced diabetic rats and to elucidate the mechanism of its insulin-sensitizing effect.</p> <p>Methods</p> <p>Male Sprague-Dawley rats were fed the AIN-93G diet modified to contain 41.2% fat and intraperitoneally injected with a single dose of streptozotocin (40 mg/kg body weight). One week after injection, the rats with fasting blood glucose levels above 126 mg/dL were randomly divided into 4 groups to treat with high fat diet containing 0 (diabetic control), 0.5, or 1% of OPE or 0.1% quercetin (quercetin equivalent to 1% of OPE) for 8 weeks. To investigate the mechanism for the effects of OPE, we examined biochemical parameters (insulin sensitivity and oxidative stresses) and protein and gene expressions (pro-inflammatory cytokines and receptors).</p> <p>Results</p> <p>Compared to the diabetic control, hypoglycemic and insulin-sensitizing capability of 1% OPE were demonstrated by significant improvement of glucose tolerance as expressed in incremental area under the curve (<it>P </it>= 0.0148). The insulin-sensitizing effect of OPE was further supported by increased glycogen levels in liver and skeletal muscle (<it>P </it>< 0.0001 and <it>P </it>= 0.0089, respectively). Quantitative RT-PCR analysis showed increased expression of insulin receptor (<it>P </it>= 0.0408) and GLUT4 (<it>P </it>= 0.0346) in muscle tissues. The oxidative stress, as assessed by superoxide dismutase activity and malondialdehyde formation, plasma free fatty acids, and hepatic protein expressions of IL-6 were significantly reduced by 1% OPE administration (<it>P </it>= 0.0393, 0.0237, 0.0148 and 0.0025, respectively).</p> <p>Conclusion</p> <p>OPE might improve glucose response and insulin resistance associated with type 2 diabetes by alleviating metabolic dysregulation of free fatty acids, suppressing oxidative stress, up-regulating glucose uptake at peripheral tissues, and/or down-regulating inflammatory gene expression in liver. Moreover, in most cases, OPE showed greater potency than pure quercetin equivalent. These findings provide a basis for the use of onion peel to improve insulin insensitivity in type 2 diabetes.</p

Effect of biochars pyrolyzed in N2 and CO2, and feedstock on microbial community in metal(loid)s contaminated soils

Little is known about the effects of applying amendments on soil for immobilizing metal(loid)s on the soil microbial community. Alterations in the microbial community were examined after incubation of treated contaminated soils. One soil was contaminated with Pb and As, a second soil with Cd and Zn. Red pepper stalk (RPS) and biochars produced from RPS in either N2 atmosphere (RPSN) or CO2 atmosphere (RPSC) were applied at a rate of 2.5% to the two soils and incubated for 30 days. Bacterial communities of control and treated soils were characterized by sequencing 16S rRNA genes using the Illumina MiSeq sequencing. In both soils, bacterial richness increased in the amended soils, though somewhat differently between the treatments. Evenness values decreased significantly, and the final overall diversities were reduced. The neutralization of pH, reduced available concentrations of Pb or Cd, and supplementation of available carbon and surface area could be possible factors affecting the community changes. Biochar amendments caused the soil bacterial communities to become more similar than those in the not amended soils. The bacterial community structures at the phylum and genus levels showed that amendment addition might restore the normal bacterial community of soils, and cause soil bacterial communities in contaminated soils to normalize and stabilize

The limited immunomodulatory effects of escharectomy on the kinetics of endotoxin, cytokines, and adhesion molecules in major burns.

Escharectomy has been shown to improve the survival rates and the outcomes in burns. This observational study was conducted to assess the role of escharectomy on the inflammatory mediators in major burns. Seventeen ASA physical status II or status III adult surviving major burn patients were recruited. When the escharectomy was scheduled, a series of blood samples was obtained at -3 and -1 days preoperation, and +1 and +3 postoperation. The changing levels of endotoxin, cytokines, and adhesion molecules were measured with a quantitative sandwich immunoassay. Extensive escharectomy did not appear to have any significant impact on the levels of tumor necrosis factor alpha, interleukin-10, soluble intracellular adhesion molecule-1 and soluble vascular adhesion molecule-1. Meanwhile, endotoxin and E-selectin were significantly decreased after escharectomy. Escharectomy appeared to have a limited immunomodulatory effect on the inflammatory mediators in systemic inflammatory responses induced by major burns. This is probably related to the timing and extent of surgery, and the complex nature of burn-related inflammation

Titanium dioxide induces apoptotic cell death through reactive oxygen species-mediated Fas upregulation and Bax activation

Background: Titanium dioxide (TiO2) has been widely used in many areas, including biomedicine, cosmetics, and environmental engineering. Recently, it has become evident that some TiO2 particles have a considerable cytotoxic effect in normal human cells. However, the molecular basis for the cytotoxicity of TiO2 has yet to be defined.Methods and results: In this study, we demonstrated that combined treatment with TiO2 nanoparticles sized less than 100 nm and ultraviolet A irradiation induces apoptotic cell death through reactive oxygen species-dependent upregulation of Fas and conformational activation of Bax in normal human cells. Treatment with P25 TiO2 nanoparticles with a hydrodynamic size distribution centered around 70 nm (TiO2P25-70) together with ultraviolet A irradiation-induced caspase-dependent apoptotic cell death, accompanied by transcriptional upregulation of the death receptor, Fas, and conformational activation of Bax. In line with these results, knockdown of either Fas or Bax with specific siRNA significantly inhibited TiO2-induced apoptotic cell death. Moreover, inhibition of reactive oxygen species with an antioxidant, N-acetyl-L-cysteine, clearly suppressed upregulation of Fas, conformational activation of Bax, and subsequent apoptotic cell death in response to combination treatment using TiO2P25-70 and ultraviolet A irradiation.Conclusion: These results indicate that sub-100 nm sized TiO2 treatment under ultraviolet A irradiation induces apoptotic cell death through reactive oxygen species-mediated upregulation of the death receptor, Fas, and activation of the preapoptotic protein, Bax. Elucidating the molecular mechanisms by which nanosized particles induce activation of cell death signaling pathways would be critical for the development of prevention strategies to minimize the cytotoxicity of nanomaterials.This work was supported by the Korea Ministry of Environment and The Eco-Technopia 21 Project (091-091-081)