Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models

Due to the substantial scale of Large Language Models (LLMs), the direct application of conventional compression methodologies proves impractical. The computational demands associated with even minimal gradient updates present challenges, particularly on consumer-grade hardware. This paper introduces an innovative approach for the parametric and practical compression of LLMs based on reduced order modelling, which entails low-rank decomposition within the feature space and re-parameterization in the weight space. Notably, this compression technique operates in a layer-wise manner, obviating the need for a GPU device and enabling the compression of billion-scale models within stringent constraints of both memory and time. Our method represents a significant advancement in model compression by leveraging matrix decomposition, demonstrating superior efficacy compared to the prevailing state-of-the-art structured pruning method.Comment: Brief technical report; Code will be made available at https://github.com/transmuteAI/trailmet/tree/main/trailmet/algorithms/llm-ro

Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architectures

Conventional scaling of neural networks typically involves designing a base network and growing different dimensions like width, depth, etc. of the same by some predefined scaling factors. We introduce an automated scaling approach leveraging second-order loss landscape information. Our method is flexible towards skip connections a mainstay in modern vision transformers. Our training-aware method jointly scales and trains transformers without additional training iterations. Motivated by the hypothesis that not all neurons need uniform depth complexity, our approach embraces depth heterogeneity. Extensive evaluations on DeiT-S with ImageNet100 show a 2.5% accuracy gain and 10% parameter efficiency improvement over conventional scaling. Scaled networks demonstrate superior performance upon training small scale datasets from scratch. We introduce the first intact scaling mechanism for vision transformers, a step towards efficient model scaling.Comment: Accepted At ICLR 2024 (Tiny Paper Track

A prospective study on geriatric abdominal surgical emergencies

Background: Geriatric population is a special subgroup of population undergoing emergency abdominal surgeries. Both higher age group and emergency surgical procedure are considered as high risk factors. In this study, we study the most common cause for geriatric population to undergo an emergency abdominal surgery and the therapeutic outcomes.Methods: All the patients aged more than 60 years coming to surgical department, BLDEU’s hospital with acute abdominal conditions. Study period was from Jan 2010 to Jan 2013. All patients aged more than 60 years old admitted with abdominal emergency conditions in department of surgery. Geriatric patients coming with blunt trauma of abdomen also included. Exclusion criteria were immunocompromised patients.Results:128 patients aged 60 years or more who presented with abdominal emergency surgical conditions were studied. Most common cause for emergency abdominal surgery was perforated peptic ulcer (38%) followed by intestinal obstruction (17%). The most common post-operative complication was surgical site infection (29%). Mortality rate was 17%. Most common cause of death was septic shock with multi organ dysfunction.Conclusion:Geriatric population is an important subgroup of population undergoing emergency abdominal surgeries. Most common cause is peptic ulcer perforation followed by intestinal obstruction due to adhesions. More than the age per say, the delay in presentation may be the cause for mortality in this age group. The therapeutic outcome in patients with co morbid factors like hypertension and diabetes mellitus in control, were similar to other patients.

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.Comment: Technical report. v2: Add LLaMA-1&2 results. Code and models at https://github.com/Arnav0400/ViT-Slim/tree/master/GLoR

Study the effect of metformin, voglibose alone and in combination on body mass index in non-diabetic obese Indian subjects- A hospital based study

Background: Early detection and therapy of the obese adolescent with a family history of type 2 diabetes may interrupt the cycle of weight gain and insulin resistance that leads to glucose intolerance in adulthood. Materials & Methods: The objective of our study was to observe the effect of metformin and voglibose on BMI, as it provides a simple and convenient anthropometric index for classification of obesity. 60 non diabetic obese subjects were selected on the basis of inclusion and exclusion criteria, and divided into three groups of 20 subjects each. The first group received metformin 500 mg BD, second group received voglibose 0.3 mg and the third group received a combination of metformin 500 mg and voglibose 0.3mg. For the comparison we applied paired and unpaired t test. Paired t test was applied for intra group comparison and unpaired t test was applied for inter group comparison. Results: After six months of treatment with Metformin 500 mg BD alone, Voglibose 0.3mg BD alone and Metformin 500 mg with Voglibose 0.3 mg BD in combination, all three groups showed statistically significant reduction in BMI values from baseline. When we compared results of metformin group with voglibose group there was no statistically significant difference. But when we compared results of metformin alone with metformin and voglibose combination and voglibose alone with metformin and voglibose combination, the combination group showed statistically significant reduction in BMI base line values. Conclusion: Therefore, it can be concluded that Metformin + Voglibose combination is very effective in reducing body weight, but further long term studies with large sample size are needed to assess the safety and efficacy of Metformin+ Voglibose combination in treatment of obesity in non-diabetic population

TO ACCESS THE EFFECT OF PIPALLI CHOORNA AND SHATAVARI CHOORNA IN STANYAKSHAYA

Aim: To access the effect of Pipalli and Shatavari choorna with Shukhoshna godugdha in Stanyakshaya. Objective: Study of literature regarding Stanyakshaya and Pippli choorna. Observation on the effect of Pippali choorna with Godugdha on lactating mother and observe if any adverse effect.Method: Study Group: 60 patients were observed & treated, Study divided in two groups 30 patients in each group. In Group A (Trial Group) 30 patients are randomly selected in which Stanyakshaya will be treated with Pipplichoorna, 500 mg twice daily after meals with Sukhoshana Godugdha. In Group B (Control Group) 30 patients are Group of randomly selected in which Stanya Jananan drug Shatavarimool choorna 2 gm twice daily with Godugdha after meal. A follow up was done on each patient after every 7 days. Initially all the signs and symptoms were noted thoroughly. Change in signs and symptoms in each follow up were observed and noted in case paper. Follow up was done for 3 weeks during treatment and for 2 weeks after treatment. The total duration of treatment was 21 days.Results: Comparing all the symptoms before and after treatment had positive results of  treatment given to group B (Shatavari Choorna) showed slightly better results over treatment given to group A (Pippali choorna). The Statistical Analysis reveals that Shatavari Choorna with Shukhoshna godugdha in the management of Stanyakshaya is more effective than Pippalichoorna with Shukhoshna godugdha. Conclusion: Treatment given to group B (Shatavari Choorna) showed better results over treatment given to group A (Pippali choorna), we can conclude that treatment Shatavari Choorna given to group B is better for this disease Stanyakshaya

On designing light-weight object trackers through network pruning: Use CNNs or transformers?

Object trackers deployed on low-power devices need to be light-weight, however, most of the current state-of-the-art (SOTA) methods rely on using compute-heavy backbones built using CNNs or transformers. Large sizes of such models do not allow their deployment in low-power conditions and designing compressed variants of large tracking models is of great importance. This paper demonstrates how highly compressed light-weight object trackers can be designed using neural architectural pruning of large CNN and transformer based trackers. Further, a comparative study on architectural choices best suited to design light-weight trackers is provided. A comparison between SOTA trackers using CNNs, transformers as well as the combination of the two is presented to study their stability at various compression ratios. Finally results for extreme pruning scenarios going as low as 1% in some cases are shown to study the limits of network pruning in object tracking. This work provides deeper insights into designing highly efficient trackers from existing SOTA methods.Comment: Submitted at IEEE ICASSP 202