Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT), which enables adaptive feature extraction from global contexts. However, existing self-attention methods either adopt sparse global attention or window attention to reduce the computation complexity, which may compromise the local feature learning or subject to some handcrafted designs. In contrast, local attention, which restricts the receptive field of each query to its own neighboring pixels, enjoys the benefits of both convolution and self-attention, namely local inductive bias and dynamic feature selection. Nevertheless, current local attention modules either use inefficient Im2Col function or rely on specific CUDA kernels that are hard to generalize to devices without CUDA support. In this paper, we propose a novel local attention module, Slide Attention, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Specifically, we first re-interpret the column-based Im2Col function from a new row-based perspective and use Depthwise Convolution as an efficient substitution. On this basis, we propose a deformed shifting module based on the re-parameterization technique, which further relaxes the fixed key/value positions to deformed features in the local region. In this way, our module realizes the local attention paradigm in both efficient and flexible manner. Extensive experiments show that our slide attention module is applicable to a variety of advanced Vision Transformer models and compatible with various hardware devices, and achieves consistently improved performances on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/Slide-Transformer.Comment: Accepted to CVPR202

Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.Comment: Accepted to CVPR202

Treatment with ML385 mitigated the protective effects of β-PAE on cognitive function and neuroinflammation.

ML385 reversed β-PAE mediated neuroprotective effects as shown in the behavioral test. ML385(1mg/kg in 3μL) was i.c.v injected 2 hours post-CLP. The inhibitory avoidance test and open field test were performed at 24h post-surgery (n = 6–8 in each group). (B) ML385 partially mitigated the anti-inflammatory effects of β-PAE in the brain. TNF-α, IL-1β and IL-6 were measured in brain homogenate by ELISA (n = 6–8 in each group). (C) The protein expression of Sirt1, Nrf2, HO-1 and cleaved-caspase 3 was determined by western-blotting. Samples of hippocampus were collected and prepared in homogenate at 24h after procedure. Summary data are obtained from three independent experiments (n = 4 in each group). *P<0.05 vs.CLP control; #P<0.05 vs. CLP+β-PAE group.</p

Angiogenesis unveiled: Insights into its role and mechanisms in cartilage injury

Osteoarthritis (OA) commonly results in compromised mobility and disability, thereby imposing a significant burden on healthcare systems. Cartilage injury is a prevalent pathological manifestation in OA and constitutes a central focus for the development of treatment strategies. Despite the considerable number of studies aimed at delaying this degenerative process, their outcomes remain unvalidated in preclinical settings. Recently, therapeutic strategies focused on angiogenesis have attracted the growing interest from researchers. Thus, we conducted a comprehensive literature review to elucidate the current progress in research and pinpoint research gaps in this domain. Additionally, it provides theoretical guidance for future research endeavors and the development of treatment strategies

Blockade of IL-17A/IL-17R Pathway Protected Mice from Sepsis-Associated Encephalopathy by Inhibition of Microglia Activation

Sepsis-associated encephalopathy (SAE) is a poorly understood condition that leads to long-term cognitive impairment and increased mortality in survivors. Recent research revealed that IL-17A/IL-17R might serve as a checkpoint in microglia-mediated neuroinflammation. The present study was designed to determine the specific role of IL-17A-mediated microglia activation in the development of SAE. A mouse model of SAE was induced by cecal ligation and puncture (CLP), and behavior performance was evaluated by the inhibitory avoidance test and the open field test. Cytokine expression and microglia activation in brain tissue were determined at 6 h, 12 h, 24 h, 48 h, and day 7 post surgery. Further, septic mice were intracerebral ventricle- (i.c.v.-) injected with recombinant IL-17A, anti-IL-17A ab, anti-IL-17R ab, or isotype controls to evaluate the potential effects of IL-17A/IL-17R blockade in the prevention of SAE. Septic peritonitis induced significant impairment of learning memory and exploratory activity, which was associated with a higher expression of IL-17A, IL-1β, and TNF-α in the brain homogenate. Fluorescence intensity of Iba-1 and IL-17R in the hippocampus was significantly increased following CLP. Treatment with recombinant IL-17A enhanced the neuroinflammation and microglia activation in CLP mice. On the contrary, neutralizing anti-IL-17A or anti-IL-17R antibodies mitigated the CNS inflammation and microglia activation, thus alleviating the cognitive dysfunction. Furthermore, as compared to the sham control, microglia cultured from CLP mice produced significantly higher levels of cytokines and expressed with higher fluorescence intensity of Iba-1 in response to IL-17A or LPS. Pretreatment with anti-IL-17R ab suppressed the Iba-1 expression and cytokine production in microglia stimulated by IL-17A. In conclusion, blockade of the IL-17A/IL-17R pathway inhibited microglia activation and neuroinflammation, thereby partially reversing sepsis-induced cognitive impairment. The present study suggested that the IL-17A/IL-17R signaling pathway had an important, nonredundant role in the development of SAE

β-PAE improved the cognitive function in SAE mice.

Mice were i.c.v injected with a low dose(0.2mg/kg) of β-PAE, high dose(1mg/kg) of β-PAE or saline, respectively. Reagents were all prepared in equal volume of 3μL. Step down inhibitory avoidance test and open field test were performed at 24h, 48h and 7days post-surgery. All presented data are a composite of three independent experiments (n = 6–10 in each group). *P<0.05 vs. Sham control; #P<0.05 vs. CLP control; β-PAE(L), low dose of β-PAE group; β-PAE(H), high dose of β-PAE group.</p

Enhanced and unenhanced: Radiomics models for discriminating between benign and malignant cystic renal masses on CT images: A multi-center study.

BackgroundMachine learning algorithms used to classify cystic renal masses (CRMs) nave not been applied to unenhanced CT images, and their diagnostic accuracy had not been compared against radiologists.MethodThis retrospective study aimed to develop radiomics models that discriminate between benign and malignant CRMs in a triple phase computed tomography (CT) protocol and compare the diagnostic accuracy of the radiomics approach with experienced radiologists. Predictive models were established using a training set and validation set of unenhanced and enhanced (arterial phase [AP] and venous phase [VP]) CT images of benign and malignant CRMs. The diagnostic capabilities of the models and experienced radiologists were compared using Receiver Operating Characteristic (ROC) curves.ResultsOn unenhanced, AP and VP CT images in the validation set, the AUC, specificity, sensitivity and accuracy for discriminating between benign and malignant CRMs were 90.0 (95%CI: 81-98%), 90.0%, 90.5% and 90.2%; 93.0% (95%CI: 86-99%), 86.7%, 95.2% and 88.3%; and 95.0% (95%CI: 90%-100%), 93.3%, 90.5% and 92.1%, respectively, for the radiomics models. Diagnostic accuracy of the radiomics models differed significantly on unenhanced images in the training set vs. each radiologist (p = 0.001 and 0.003) but not in the validation set (p = 0.230 and 0.590); differed significantly on AP images in the validation set vs. each radiologist (p = 0.007 and 0.007) but not in the training set (p = 0.663 and 0.663); and there were no differences on VP images in the training or validation sets vs. each radiologist (training set: p = 0.453 and 0.051, validation set: p = 0.236 and 0.786).ConclusionsRadiomics models may have clinical utility for discriminating between benign and malignant CRMs on unenhanced and enhanced CT images. The performance of the radiomics model on unenhanced CT images was similar to experienced radiologists, implying it has potential as a screening and diagnostic tool for CRMs

High-Affinity Anti-VISTA Antibody Protects against Sepsis by Inhibition of T Lymphocyte Apoptosis and Suppression of the Inflammatory Response

Background. B7 family members and ligands have been identified as critical checkpoints in orchestrating the immune response during sepsis. V-domain Ig suppressor of T cell activation (VISTA) is a new inhibitory immune checkpoint involved in restraining T cell response. Previous studies demonstrated that VISTA engagement on T cells and myeloid cells could transmit inhibitory signals, resulting in reduced activation and function. The current study was designed to determine the potential therapeutic effects of a high-affinity anti-VISTA antibody (clone MH5A) in a murine model of sepsis. Methods. Polymicrobial sepsis was induced in male C57BL/6 mice via cecal ligation and puncture. Expression profiles of VISTA on T lymphocytes and macrophage were examined at 24 and 72 h postsurgery. The effects of anti-VISTA mAb on the 7-day survival, lymphocyte apoptosis, cytokine expression, bacterial burden, and vital organ damage were determined. Furthermore, the effects of anti-VISTA mAb on CD3+ T cell apoptosis and macrophage activation were determined in vitro. Results. VISTA was substantially expressed on T cells and macrophages in sham-operated mice; septic peritonitis did not induce significant changes in the expression profiles. Treatment with MH5A improved the survival of septic mice, accompanied by reduced lymphocyte apoptosis, decreased cytokine expression, and enhanced bacterial clearance. Engagement of VISTA receptor with MH5A mitigated CD3+ T cell apoptosis cultured from CLP mice and suppressed LPS-induced cytokine production by macrophage in vitro. Conclusion. The present study identified VISTA as a novel immune checkpoint in the regulation of T cell and macrophage response during sepsis. Modulation of the VISTA pathway might offer a promising opportunity in the immunotherapy for sepsis

Endosymbiotic Fungal Diversity and Dynamics of the Brown Planthopper across Developmental Stages, Tissues, and Sexes Revealed Using Circular Consensus Sequencing

Endosymbiotic fungi play an important role in the growth and development of insects. Understanding the endosymbiont communities hosted by the brown planthopper (BPH; Nilaparvata lugens Stål), the most destructive pest in rice, is a prerequisite for controlling BPH rice infestations. However, the endosymbiont diversity and dynamics of the BPH remain poorly studied. Here, we used circular consensus sequencing (CCS) to obtain 87,131 OTUs (operational taxonomic units), which annotated 730 species of endosymbiotic fungi in the various developmental stages and tissues. We found that three yeast-like symbionts (YLSs), Polycephalomyces prolificus, Ophiocordyceps heteropoda, and Hirsutella proturicola, were dominant in almost all samples, which was especially pronounced in instar nymphs 4–5, female adults, and the fat bodies of female and male adult BPH. Interestingly, honeydew as the only in vitro sample had a unique community structure. Various diversity indices might indicate the different activity of endosymbionts in these stages and tissues. The biomarkers analyzed using LEfSe suggested some special functions of samples at different developmental stages of growth and the active functions of specific tissues in different sexes. Finally, we found that the incidence of occurrence of three species of Malassezia and Fusarium sp. was higher in males than in females in all comparison groups. In summary, our study provides a comprehensive survey of symbiotic fungi in the BPH, which complements the previous research on YLSs. These results offer new theoretical insights and practical implications for novel pest management strategies to understand the BPH–microbe symbiosis and devise effective pest control strategies