Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.Comment: AAAI 202

An increase in long non-coding RNA PANDAR is associated with poor prognosis in clear cell renal cell carcinoma

Abstract Background Nearly 30% of clear cell renal cell carcinoma (ccRCC) patients present with metastasis at the time of diagnosis, and the prognosis for these patients is poor. Therefore, novel potential prognostic biomarkers and therapeutic targets for ccRCC could be helpful. Emerging evidence indicates that lncRNAs play important roles in cancer tumorigenesis and could be used as potential biomarkers or therapeutic targets. PANDAR (promoter of CDKN1A antisense DNA damage activated RNA) is a relatively novel lncRNA that plays an important role in the development of multiple cancers. However, the clinical significance and molecular mechanism of PANDAR in ccRCC are still elusive. In the present study, we attempted to elucidate the role of PANDAR in ccRCC. Methods The relative expression level of lncRNA PANDAR was quantified by real-time qPCR in 62 paired ccRCC tissues and in renal cancer cell lines, and its association with overall survival was assessed by statistical analysis. The biological functions of lncRNA PANDAR on ccRCC cells were determined both in vitro and in vivo. Results PANDAR expression was significantly upregulated in tumor tissues and cell lines compared with normal counterparts. Moreover, PANDAR served as an independent predictor of overall survival, and increased PANDAR expression was positively correlated with an advanced TNM stage. Further experiments demonstrated that PANDAR silencing can significantly inhibit cell proliferation and invasion, induce cell cycle arrest in the G1 phase and significantly promote apoptosis in 7860 and Caki-1 cell lines. In addition, in vivo experiments confirmed that downregulation of PANDAR inhibited the tumorigenic ability of 7860 cells in nude mice. Silencing of PANDAR also inhibited the expression of Bcl-2 and Mcl-1 and upregulated the expression of Bax in vivo. Conclusions Our results suggest that PANDAR is involved in ccRCC progression and may serve as a potential prognostic biomarker and therapeutic target