Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains. We demonstrate the efficacy of our approach by presenting the Sheared-LLaMA series, pruning the LLaMA2-7B model down to 1.3B and 2.7B parameters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, and OpenLLaMA models, on a wide range of downstream and instruction tuning evaluations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMs with structured pruning is a far more cost-effective approach for building smaller LLMs.Comment: The code and models are available at https://github.com/princeton-nlp/LLM-Shearin

Knowledge-Based Open Performance Measurement System (KBO-PMS) for a Garment Product Development Process in Big Data Environment

Globally, customers are getting increasingly demanding in terms of personalization of products and are asking for shorter product development periods with more predictable product performance, especially in fashion industry. Current market pressures drive firms to adapt new design process in product development (PD) processes. Nevertheless, choosing the effective PD process is a challenging, complex decision. There is a critical need to develop a performance measurements system (PMS) for choosing appropriate product development (PD) processes in garment design to support product mangers to effectively respond to market. This paper presents a knowledge-based open performance measurement system (KBO-PMS) in big data environment, in order to support complex industrial decision-making for new product development. Its dynamic and flexible structure enables the whole system to be more adapted to knowledge sharing of product managers and processing of various time-varying data. The proposed KBO-PMS is composed of an interactive structure, capable of both integrating new KPIs from the open resource and tracking the evolution of the KBO-PMS components with time. The proposed KBO-PMS has been validated by realizing the performance evaluation of product development (PD) in fashion industry. It can be regarded as an application of open-resource based dynamic group decision-making in fashion big data environment

Lifestyle intervention Tai Chi for adult patients with type 2 diabetes mellitus: a PRIO-harms based overview of 17 systematic reviews

ObjectiveTo systematically summarize current evidence and determine the clinical effectiveness and safety of Tai Chi for type 2 diabetes mellitus (T2DM) in adults by conducting an overview of systematic reviews (SRs).MethodsA systematic search encompassing five electronic databases was conducted until July 30, 2023, to identify relevant systematic reviews (SRs) based on randomized controlled trials (RCTs) concerning Tai Chi for T2DM. The methodological quality of the included SRs was assessed using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2) and the Risk of Bias in Systematic Reviews (ROBIS) tool. The Preferred Reporting Items for Overview of Systematic Review (PRIO-harms) checklist was used to promote a more balanced reporting of benefits and harms in this overview. Corrected covered area (CCA) was used to calculate the degree of overlapping primary studies. Primary outcome measures were glycosylated hemoglobin (HbA1c) and fasting blood glucose (FBG), while secondary outcomes encompassed health-related quality measures. The GRADE (Grades of Recommendations, Assessment, Development, and Evaluation) framework was utilized to assess the quality of evidence for the outcome measures.ResultsA total of 17 eligible SRs were included in this overview. One SR reported negative conclusions, while the remaining 16 reported positive ones on different outcomes. A total of 4 SRs reported adverse events, either absent or minor. Most of the SRs exhibited critically low quality (15/17) and a high risk of bias (14/17), as indicated by AMSTAR2 and ROBIS, respectively. The CCA was 12.14%, indicating a high degree of primary study overlapping. Evidence from 135 results for 24 outcomes concerning Tai Chi for T2DM was evaluated using the GRADE approach, most of which were rated very low.ConclusionTai Chi shows promise as a potentially effective and safe lifestyle intervention for adults with T2DM, particularly in improving HbA1c, FBG, BMI, and overall quality of life (QoL). However, these results should be cautiously interpreted due to methodological flaws observed in the current SRs and the low quality of the SRs based on GRADE. Furthermore, there is a compelling need for additional well-designed, high-quality RCTs and SRs to establish robust and conclusive evidence regarding the efficacy of Tai Chi for managing T2DM in the future.Systematic Review Registrationhttps://www.crd.york.ac.uk/PROSPERO/, identifier CRD 42019140988

Congestive Heart Failure Exhibited Higher BMI With Lower Energy Intake and Lower Physical Activity Level: Data From the National Health and Examination Nutrition Survey

Background: Despite that nutritional deficiency existed in congestive heart failure (CHF), there is a large amount of CHF patients suffering from obesity. This study aimed to identify the differences for increased BMI or obesity in CHF patients.Methods: This cross-sectional study included adults from the National Health and Nutrition Examination Survey 2007–2016. Differences were compared between CHF participants vs. non-CHF participants, and BMI ≥ 30 kg/m2 vs. BMI < 30 kg/m2 CHF participants.Results: CHF participants were with higher BMI, lower energy and macronutrient intake, lower physical activity level and longer rest time, and lower hematocrit and hemoglobin level (all P < 0.05) than non-CHF participants. The prevalence of BMI ≥ 30 kg/m2 in participants with CHF was 53.48%. There was no significant difference observed in energy and macronutrient intake between CHF participants with BMI ≥ 30 kg/m2 or <30 kg/m2. The water intake (P = 0.032), sedentary time (P = 0.002), and hematocrit (P = 0.028) were significantly different between CHF with BMI ≥ 30 kg/m2 and with <30 kg/m2.Conclusion: Compared with non-CHF participants, CHF participants exhibited higher BMI with lower energy and macronutrient intake, lower physical activity level, longer rest time, and hemodilution with lower hematocrit and hemoglobin level. Among CHF participants with BMI ≥ 30 kg/m2, higher sedentary time and hematocrit were observed

Effect of Sow Intestinal Flora on the Formation of Endometritis

Endometritis is the main cause of decreased reproductive performance of sows, while one of the most important factors in the etiology of sow endometritis is an aberration of birth canal microbiota. Therefore, people began to pay attention to the microbiota structure and composition of the birth canal of sows with endometritis. Interestingly, we found that the risk of endometritis was increased in the sows with constipation in clinical practice, which may imply that the intestinal flora is related to the occurrence of endometritis. Therefore, understanding the relationship between birth canal microbiota and intestinal microbiota of the host has become exceptionally crucial. In this study, the microbiota of birth canal secretions and fresh feces of four healthy and four endometritis sows were analyzed via sequencing the V3 + V4 region of bacterial 16S ribosomal (rDNA) gene. The results showed a significant difference between endometritis and healthy sows birth canal flora in composition and abundance. Firmicutes (74.36%) and Proteobacteria were the most dominant phyla in birth canal microbiota of healthy sows. However, the majority of beneficial bacteria that belonging to Firmicutes phylum (e.g., Lactobacillus and Enterococcus) declined in endometritis sow. The abundance of Porphyromonas, Clostridium sensu stricto 1, Streptococcus, Fusobacterium, Actinobacillus, and Bacteroides increased significantly in the birth canal microbiota of endometritis sows. Escherichia–Shigella and Bacteroides were the common genera in the birth canal and intestinal flora of endometritis sows. The abundance of Escherichia–Shigella and Bacteroides in the intestines of sows suffering from endometritis were significantly increased than the intestinal microbiota of the healthy sows. We speculated that some intestinal bacteria (such as Escherichia–Shigella and Bacteroides) might be bound up with the onset of sow endometritis based on intestinal microbiota analysis in sows with endometritis and healthy sows. The above results can supply a theoretical basis to research the pathogenesis of endometritis and help others understand the relationship with the microbiota of sow's birth canal and gut

The regulations on cortical activation and functional connectivity of the dorsolateral prefrontal cortex-primary somatosensory cortex elicited by acupuncture with reinforcing-reducing manipulation

IntroductionTraditional acupuncture with reinforcing-reducing manipulation is essential for clinical effectiveness, whereas the underlying central mechanism of it remains unknown. This study with multiple-channels functional near-infrared spectroscopy (fNIRS) aims to explore cerebral-response modes during acupuncture with reinforcing-reducing manipulations.Materials and methodsFunctional near-infrared spectroscopy data were recorded from 35 healthy participants during the lifting-thrusting reinforcing manipulation, the lifting-thrusting reducing manipulation, and the even reinforcing-reducing manipulation with lifting-thrusting. The general linear model based (GLM) cortical activation analysis and the functional connectivity (FC) based on region of interest (ROI) analysis were combined to be conducted.ResultsIn comparison with the baseline, the results showed that three acupuncture with reinforcing-reducing manipulations similarly induced the hemodynamic responses in the bilateral dorsolateral prefrontal cortex (DLPFC) and increased FC between the DLPFC and primary somatosensory cortex (S1). Specifically, the even reinforcing-reducing manipulation deactivated the bilateral DLPFC, the frontopolar area (FP), the right primary motor cortex (M1), the bilateral S1, and the bilateral secondary somatosensory cortex (S2); The reducing manipulation deactivated the bilateral DLPFC; The reinforcing manipulation activated the bilateral DLPFC, the left S1, and the right S2. The between-group comparisons indicated that the reinforcing-reducing manipulation induced opposite hemodynamic responses in the bilateral DLPFC and the left S1 and exhibited different FC patterns in the left DLPFC-S1, within the right DLPFC, and between the left S1 and the left orbitofrontal cortex (OFC).ConclusionThese findings verified the feasibility of fNIRS for investigating cerebral functional activities of acupuncture manipulations, suggesting that the regulations on the DLPFC-S1 cortex may be the potential central mechanism for the realization of acupuncture with reinforcing-reducing manipulation’s effect.Clinical trial registrationClinicalTrials.gov, identifier, ChiCTR2100051893