Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains. We demonstrate the efficacy of our approach by presenting the Sheared-LLaMA series, pruning the LLaMA2-7B model down to 1.3B and 2.7B parameters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, and OpenLLaMA models, on a wide range of downstream and instruction tuning evaluations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMs with structured pruning is a far more cost-effective approach for building smaller LLMs.Comment: The code and models are available at https://github.com/princeton-nlp/LLM-Shearin

Software Architecture in Practice: Challenges and Opportunities

Software architecture has been an active research field for nearly four decades, in which previous studies make significant progress such as creating methods and techniques and building tools to support software architecture practice. Despite past efforts, we have little understanding of how practitioners perform software architecture related activities, and what challenges they face. Through interviews with 32 practitioners from 21 organizations across three continents, we identified challenges that practitioners face in software architecture practice during software development and maintenance. We reported on common software architecture activities at software requirements, design, construction and testing, and maintenance stages, as well as corresponding challenges. Our study uncovers that most of these challenges center around management, documentation, tooling and process, and collects recommendations to address these challenges.Comment: Preprint of Full Research Paper, the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '23

Temporal and bidirectional association between blood pressure variability and arterial stiffness: Cross-lagged cohort study

BACKGROUND: The causal relationship between blood pressure variability (BPV) and arterial stiffness remains debated. OBJECTIVE: This study aimed to explore the temporal and bidirectional associations between long-term BPV and arterial stiffness using a cohort design with multiple surveys. METHODS: Participants from the Beijing Health Management Cohort who underwent health examinations from visit 1 (2010-2011) to visit 5 (2018-2019) were enrolled in this study. Long-term BPV was defined as intraindividual variation using the coefficient of variation (CV) and SD. Arterial stiffness was measured by brachial-ankle pulse wave velocity (baPWV). The bidirectional relationship between BPV and arterial stiffness was explored using cross-lagged analysis and linear regression models, with records before and after visit 3 categorized as phase 1 and phase 2, respectively. RESULTS: Of the 1506 participants, who were a mean of 56.11 (SD 8.57) years old, 1148 (76.2%) were male. The cross-lagged analysis indicated that the standardized coefficients of BPV at phase 1 directing to the baPWV level at phase 2 were statistically significant but not vice-versa. The adjusted regression coefficients of the CV were 4.708 (95% CI 0.946-8.470) for systolic blood pressure, 3.119 (95% 0.166-6.073) for diastolic pressure, and 2.205 (95% CI 0.300-4.110) for pulse pressure. The coefficients of the SD were 4.208 (95% CI 0.177-8.239) for diastolic pressure and 4.247 (95% CI 0.448-8.046) for pulse pressure. The associations were predominant in the subgroup with hypertension, but we did not observe any significant association of baPWV level with subsequent BPV indices. CONCLUSIONS: The findings supported a temporal relationship between long-term BPV and arterial stiffness level, especially among people with hypertension

Proteomic Profile of Carbonylated Proteins Screen Regulation of Apoptosis via CaMK Signaling in Response to Regular Aerobic Exercise

To research carbonylated proteins and screen molecular targets in the rat striatum on regular aerobic exercise, male SpragueDawley rats (13 months old, n = 24) were randomly divided into middle-aged sedentary control (M-SED) and aerobic exercise (M-EX) groups (n = 12 each). Maximum oxygen consumption (VO2max) gradually increased from 50%–55% to 65%–70% for a total of 10 weeks. A total of 36 carbonylated proteins with modifed oxidative sites were identifed by Electrospray IonizationQuadrupole-Time of Flight-Mass Spectrometer (ESI-Q-TOF-MS), including 17 carbonylated proteins unique to the M-SED group, calcium/calmodulin-dependent protein kinase type II subunit beta (CaMKII�), and heterogeneous nuclear ribonucleoprotein A2/B1 (Hnrnpa2b1), among others, and 19 specifc to the M-EX group, ubiquitin carboxyl-terminal hydrolase isozyme L1 (UCHL1), and malic enzyme, among others. Regular aerobic exercise improved behavioral and stereological indicators, promoted normal apoptosis (P \u3c 0.01), alleviated carbonylation of the CaMKII� and Hnrnpa2b1, but induced carbonylation of the UCH-L1, and signifcantly upregulated the expression levels of CaMKII�, CaMKII�, and Vdac1 (p \u3c 0.01) and Hnrnpa2b1 and UCH-L1 (p \u3c 0.01), as well as the phosphoinositide 3-kinase/protein kinase B/mammalian target of rapamycin pathways (PI3K/Akt/mTOR) pathway-related genes Akt and mTOR. Regular aerobic exercise for 10 weeks (incremental for the frst 6 weeks followed by constant loading for 4 weeks) enhanced carbonylation of CaMKII�, Hnrnpa2b1, and modulated apoptosis via activation of CaMK and phosphoinositide 3-kinase/protein kinase B/mTOR signaling. It also promoted normal apoptosis in the rat striatum, which may have protective efects in neurons

Bug characteristics in blockchain systems: A large-scale empirical study

NSF

MicroRNA Expression Profiling Screen miR-3557/324-targeted CaMK/mTOR in the Rat Striatum of Parkinson\u27s Disease in Regular Aerobic Exercise

This study aimed to screen the target miRNAs and to investigate the differential miR-3557/324-targeted signal mechanisms in the rats’ model of Parkinson’s disease (PD) with regular aerobic exercise. Rats were divided into sedentary control PD group (SED-PD, n = 18) and aerobic exercise PD group (EX-PD, n = 22). After 8 weeks of regular aerobic exercise, a 6-hydroxydopamine- (6-OHDA-) induced PD lesion model was constructed. Preregular aerobic exercises enhanced the injury resistance of rats with 6-OHDA-induced PD. The rotational behavior after injection of apomorphine hydrochloride was alleviated. Under the scanning electron microscopy, we found the neurons, axons, and villi of the striatum were clearly and tightly arranged, and neurons and axons significantly becoming larger. Tyrosine hydroxylase (TH) was increased significantly and α-synuclein protein expression was reduced in the EX-PD group compared to the SED-PD group. Screening from miRNA microarray chip, we further found upregulation of miR-3557 and downregulation of miR-324 were closely related to the calcium-modulating signaling pathway, remitting the progress of Parkinson’s disease on aerobic exercise. Compared to the SED-PD group, Ca2+/calmodulin dependent protein kinase II (CaMK2α) was upregulated, but CaMKV and voltage-dependent anion-selective channel protein 1 (Vdac1) were significantly downregulated in the EX-PD group. Additionally, phosphatidylinositol-3-kinase (PI3K)/mammalian target of rapamycin (mTOR) expression were activated, and ubiquitin carboxy-terminal hydrolase L1 (UCH-L1) expression was upregulated in the EX-PD group. Conclusions: the adaptive mechanism of regular aerobic exercise delaying neurodegenerative diseases and lesions was that miR-3557/324 was activated to regulate one of its targets CaMKs signaling pathways. CaMKs, coordinated with mTOR pathway-related gene expression, improved UCH-L1 level to favor for delaying neurodegeneration or improving the pathogenesis of PD lesions