DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.Comment: Accepted to EACL 202

Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization

Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. Our method (Continuation-KD) achieves state-of-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).Comment: Published at EMNLP 2022 (Findings

Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)

The rapid advancement of large language models (LLMs) has revolutionized natural language processing (NLP). While these models excel at understanding and generating human-like text, their widespread deployment can be prohibitively expensive. SortedNet is a recent training technique for enabling dynamic inference for deep neural networks. It leverages network modularity to create sub-models with varying computational loads, sorting them based on computation/accuracy characteristics in a nested manner. We extend SortedNet to generative NLP tasks, making large language models dynamic without any pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model efficiency, eliminating the need for multiple models for various scenarios during inference. We show that using this approach, we are able to unlock the potential of intermediate layers of transformers in generating the target output. Our sub-models remain integral components of the original model, minimizing storage requirements and transition costs between different computational/latency budgets. By applying this approach on LLaMa 2 13B for tuning on the Stanford Alpaca dataset and comparing it to normal tuning and early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver models twice as fast as the original model while maintaining or exceeding performance

SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks

As the size of deep learning models continues to grow, finding optimal models under memory and computation constraints becomes increasingly more important. Although usually the architecture and constituent building blocks of neural networks allow them to be used in a modular way, their training process is not aware of this modularity. Consequently, conventional neural network training lacks the flexibility to adapt the computational load of the model during inference. This paper proposes SortedNet, a generalized and scalable solution to harness the inherent modularity of deep neural networks across various dimensions for efficient dynamic inference. Our training considers a nested architecture for the sub-models with shared parameters and trains them together with the main model in a sorted and probabilistic manner. This sorted training of sub-networks enables us to scale the number of sub-networks to hundreds using a single round of training. We utilize a novel updating scheme during training that combines random sampling of sub-networks with gradient accumulation to improve training efficiency. Furthermore, the sorted nature of our training leads to a search-free sub-network selection at inference time; and the nested architecture of the resulting sub-networks leads to minimal storage requirement and efficient switching between sub-networks at inference. Our general dynamic training approach is demonstrated across various architectures and tasks, including large language models and pre-trained vision models. Experimental results show the efficacy of the proposed approach in achieving efficient sub-networks while outperforming state-of-the-art dynamic training approaches. Our findings demonstrate the feasibility of training up to 160 different sub-models simultaneously, showcasing the extensive scalability of our proposed method while maintaining 96% of the model performance

Protocol for systematic review: peak bone mass pattern in different parts of the world

Copyright: © 2015 Mohammadi Z. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Peak bone mass, which can be defined as the amount of bone tissue present at the end of the skeletal maturation, and also it is an important determinant of osteoporotic fracture risk. The peak bone mass of a given part of the skeleton is directly dependent upon both its genetics and environmental factors. Therefore, the aim of the proposed research is a comprehensive systematic assessment of the pattern of peak bone mass in different countries across the globe. The present article explains the protocol for conducting such a research

Re-admission Rate of Patients with Ureteral Stone: A Descriptive Study

Introduction: Patients with acute renal colic need to choose between undergoing medical treatments and receiving interventions. The Aim of this study is to evaluate the outcomes of patients who are discharged from emergency departments with ureteral stones lesser than 6 millimeters. In doing so, the effect of diagnostic treatment approaches on clinical outcomes and referral rate is to be assessed.Patients and Methods: This study was performed on patients with ureteral stones referred to emergency department of Shohadaye Tajrish Hospital between May2015 to June 2018. A checklist was filled out for each patient and it included their complete medical history, physical examination results and paraclinical data. Patients were then studied for 4 weeks to determine referral times to hospital and clinical outcomes.Results: 105 patients include 81 men (77.14%) with average age of 37.1±12.4 years were studied. The mean stone diameter was 4.2±2.1 mm.  Most of ureteral stones were seen in the right-hand side (60 percent). 71 patients (67.6%) did not have any history of nephrolithiasis and 73 (69.5%) did not have positive family history for nephrolithiasis. Ureteral stones were still observed in 42 patients (40%) after two weeks of studies and only one patient (1.1%) had stone in Ultrasound Imaging after 4 weeks of observations.Conclusion: Most Patients (95%) with stones smaller than 6 mm responded to Medical Expulsive Therapy (MET) after 4 weeks and passed spontaneously ureteral calculi

Design, Synthesis and Biological Evaluation of New 5,5-Diarylhydantoin Derivatives as Selective Cyclooxygenase-2 Inhibitors

A new group of 5,5-diarylhydantoin derivatives bearing a methylsulfonyl COX-2 pharmacophore at the para position of the C-5 phenyl ring were designed and synthesized as selective COX-2 inhibitors. In vitro COX-1/COX-2 inhibition structure-activity relationships identified 5-[4-(methylsulfonyl)phenyl]-5-phenyl-hydantoin (4) as a highly potent and selective COX-2 inhibitor (COX-2 IC50 = 0.077 μM; selectivity index > 1298). It was more selective than the reference drug celecoxib (COX-2 IC50 = 0.060 μM; selectivity index = 405). A molecular modeling study where 4 was docked in the binding site of COX-2 indicated that the p-MeSO2 COX-2 pharmacophore group on the C-5 phenyl ring is oriented in the vicinity of the COX-2 secondary pocket. The results of this study showed that the type of substituent on the N-3 hydantoin ring substituent is important for COX-2 inhibitory activity

Flange Wrinkling in Flexible Roll Forming Process

AbstractFlexible roll forming is an advanced sheet metal forming process for producing variable cross section profiles. Flange wrinkling at the transition zone where the cross section changes is a major defect in the flexible roll forming process. In this paper, the flange wrinkling at the transition zone is studied using finite element analysis. The results showed that the strip deformation at the transition zone can be considered as a combination of two strip deformations observed in the conventional roll forming process and the flanging process. According to finite element analysis results, when the flange wrinkling occurs, compressive longitudinal strain is smaller than the necessary compressive longitudinal strain calculated by mathematical modeling to obtain the intended profile geometry in the compression zone. Therefore, comparison of compressive longitudinal strain obtained from the finite element analysis and the necessary compressive longitudinal strain is a good criterion to predict the flange wrinkling occurrence. A flexible roll forming setup was developed. Longitudinal strain history is obtained from the finite element simulation and is compared with the experimental data from the flexible roll forming setup. Results show a good agreement and confirm the finite element analysis

Distributions of High-Sensitivity C-Reactive Protein, Total Cholesterol-HDL Ratio and 10-Year Cardiovascular Risk: National Population-Based Study

The present study aimed to evaluate the distributions of High-Sensitivity C-reactive protein, TC-HDL ratio and 10-year risk of cardiovascular diseases among Iranian adult population. We conducted a cross-sectional study on a total of 2125 adults aged 25 to 65. Data of the Third National Surveillance of Risk Factors of Non-Communicable Diseases (SuRFNCD-2007) was used. Anthropometric indices, blood pressure and biochemical measurements had been obtained. Ten-year risk of cardiovascular events was also calculated using different models. Median (interquartile range) and geometric means (95% CI) of hs-CRP were 5.1(3.9) and 4.1(4.38-4.85), respectively. Mean TC-HDL ratio±(SD) was 5.94±2.84 in men and 5.37±1.97 in women (P<0.001). In spite of risk scores (FRS and SCORE), no significant gender and age-related differences were observed in hs-CRP levels. Exclusion of CRP levels≥10 did not change the results. The proportion of high-risk categories using SCORE and FRS models were 3.6 % and 8.8 %, respectively. In comparison with other published data, greater means and median values of High-Sensitivity C-reactive protein were observed. Higher TC-HDL ratio and cardiovascular risk in men than in women were also demonstrated. The issue of screening for cardiovascular diseases has yet to be addressed due to considerable prevalence of elevated CRP and increased risk of cardiovascular events among various subgroups