Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT), which enables adaptive feature extraction from global contexts. However, existing self-attention methods either adopt sparse global attention or window attention to reduce the computation complexity, which may compromise the local feature learning or subject to some handcrafted designs. In contrast, local attention, which restricts the receptive field of each query to its own neighboring pixels, enjoys the benefits of both convolution and self-attention, namely local inductive bias and dynamic feature selection. Nevertheless, current local attention modules either use inefficient Im2Col function or rely on specific CUDA kernels that are hard to generalize to devices without CUDA support. In this paper, we propose a novel local attention module, Slide Attention, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Specifically, we first re-interpret the column-based Im2Col function from a new row-based perspective and use Depthwise Convolution as an efficient substitution. On this basis, we propose a deformed shifting module based on the re-parameterization technique, which further relaxes the fixed key/value positions to deformed features in the local region. In this way, our module realizes the local attention paradigm in both efficient and flexible manner. Extensive experiments show that our slide attention module is applicable to a variety of advanced Vision Transformer models and compatible with various hardware devices, and achieves consistently improved performances on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/Slide-Transformer.Comment: Accepted to CVPR202

Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.Comment: Accepted to CVPR202

Treatment with ML385 mitigated the protective effects of β-PAE on cognitive function and neuroinflammation.

ML385 reversed β-PAE mediated neuroprotective effects as shown in the behavioral test. ML385(1mg/kg in 3μL) was i.c.v injected 2 hours post-CLP. The inhibitory avoidance test and open field test were performed at 24h post-surgery (n = 6–8 in each group). (B) ML385 partially mitigated the anti-inflammatory effects of β-PAE in the brain. TNF-α, IL-1β and IL-6 were measured in brain homogenate by ELISA (n = 6–8 in each group). (C) The protein expression of Sirt1, Nrf2, HO-1 and cleaved-caspase 3 was determined by western-blotting. Samples of hippocampus were collected and prepared in homogenate at 24h after procedure. Summary data are obtained from three independent experiments (n = 4 in each group). *P<0.05 vs.CLP control; #P<0.05 vs. CLP+β-PAE group.</p

β-PAE improved the cognitive function in SAE mice.

Mice were i.c.v injected with a low dose(0.2mg/kg) of β-PAE, high dose(1mg/kg) of β-PAE or saline, respectively. Reagents were all prepared in equal volume of 3μL. Step down inhibitory avoidance test and open field test were performed at 24h, 48h and 7days post-surgery. All presented data are a composite of three independent experiments (n = 6–10 in each group). *P<0.05 vs. Sham control; #P<0.05 vs. CLP control; β-PAE(L), low dose of β-PAE group; β-PAE(H), high dose of β-PAE group.</p

Enhanced and unenhanced: Radiomics models for discriminating between benign and malignant cystic renal masses on CT images: A multi-center study.

BackgroundMachine learning algorithms used to classify cystic renal masses (CRMs) nave not been applied to unenhanced CT images, and their diagnostic accuracy had not been compared against radiologists.MethodThis retrospective study aimed to develop radiomics models that discriminate between benign and malignant CRMs in a triple phase computed tomography (CT) protocol and compare the diagnostic accuracy of the radiomics approach with experienced radiologists. Predictive models were established using a training set and validation set of unenhanced and enhanced (arterial phase [AP] and venous phase [VP]) CT images of benign and malignant CRMs. The diagnostic capabilities of the models and experienced radiologists were compared using Receiver Operating Characteristic (ROC) curves.ResultsOn unenhanced, AP and VP CT images in the validation set, the AUC, specificity, sensitivity and accuracy for discriminating between benign and malignant CRMs were 90.0 (95%CI: 81-98%), 90.0%, 90.5% and 90.2%; 93.0% (95%CI: 86-99%), 86.7%, 95.2% and 88.3%; and 95.0% (95%CI: 90%-100%), 93.3%, 90.5% and 92.1%, respectively, for the radiomics models. Diagnostic accuracy of the radiomics models differed significantly on unenhanced images in the training set vs. each radiologist (p = 0.001 and 0.003) but not in the validation set (p = 0.230 and 0.590); differed significantly on AP images in the validation set vs. each radiologist (p = 0.007 and 0.007) but not in the training set (p = 0.663 and 0.663); and there were no differences on VP images in the training or validation sets vs. each radiologist (training set: p = 0.453 and 0.051, validation set: p = 0.236 and 0.786).ConclusionsRadiomics models may have clinical utility for discriminating between benign and malignant CRMs on unenhanced and enhanced CT images. The performance of the radiomics model on unenhanced CT images was similar to experienced radiologists, implying it has potential as a screening and diagnostic tool for CRMs

Endosymbiotic Fungal Diversity and Dynamics of the Brown Planthopper across Developmental Stages, Tissues, and Sexes Revealed Using Circular Consensus Sequencing

Endosymbiotic fungi play an important role in the growth and development of insects. Understanding the endosymbiont communities hosted by the brown planthopper (BPH; Nilaparvata lugens Stål), the most destructive pest in rice, is a prerequisite for controlling BPH rice infestations. However, the endosymbiont diversity and dynamics of the BPH remain poorly studied. Here, we used circular consensus sequencing (CCS) to obtain 87,131 OTUs (operational taxonomic units), which annotated 730 species of endosymbiotic fungi in the various developmental stages and tissues. We found that three yeast-like symbionts (YLSs), Polycephalomyces prolificus, Ophiocordyceps heteropoda, and Hirsutella proturicola, were dominant in almost all samples, which was especially pronounced in instar nymphs 4–5, female adults, and the fat bodies of female and male adult BPH. Interestingly, honeydew as the only in vitro sample had a unique community structure. Various diversity indices might indicate the different activity of endosymbionts in these stages and tissues. The biomarkers analyzed using LEfSe suggested some special functions of samples at different developmental stages of growth and the active functions of specific tissues in different sexes. Finally, we found that the incidence of occurrence of three species of Malassezia and Fusarium sp. was higher in males than in females in all comparison groups. In summary, our study provides a comprehensive survey of symbiotic fungi in the BPH, which complements the previous research on YLSs. These results offer new theoretical insights and practical implications for novel pest management strategies to understand the BPH–microbe symbiosis and devise effective pest control strategies

Comparison of radiomics models in the training and validation sets.

Comparison of radiomics models in the training and validation sets.</p