Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.Comment: Accepted to CVPR202

From Artifacts to Outcomes: Comparison of HMD VR, Desktop, and Slides Lectures for Food Microbiology Laboratory Instruction

Despite the value of VR (Virtual Reality) for educational purposes, the instructional power of VR in Biology Laboratory education remains under-explored. Laboratory lectures can be challenging due to students' low motivation to learn abstract scientific concepts and low retention rate. Therefore, we designed a VR-based lecture on fermentation and compared its effectiveness with lectures using PowerPoint slides and a desktop application. Grounded in the theory of distributed cognition and motivational theories, our study examined how learning happens in each condition from students' learning outcomes, behaviors, and perceptions. Our result indicates that VR facilitates students' long-term retention to learn by cultivating their longer visual attention and fostering a higher sense of immersion, though students' short-term retention remains the same across all conditions. This study extends current research on VR studies by identifying the characteristics of each teaching artifact and providing design implications for integrating VR technology into higher education

Excitation of extraordinary modes inside the source of Saturn's kilometric radiation

The electron cyclotron maser instability (ECMI) of extraordinary mode waves was investigated with the parameters observed in Saturn's kilometric radiation (SKR) sources. Previous studies employed simplified dispersion relations, and did not consider the excitation of the relativistic (R) mode. This mode is introduced by considering the relativistic effect in plasmas consisting of both cold and hot electrons. Using particle-in-cell simulations, we investigated the excitation of R and X modes based on the measured data. Using the reported value of the density ratio of energetic to total electrons $n_e/n_0=24\%$, the most unstable mode is the R mode. The escaping X-mode emissions are amplified only if the energetic electrons are dominant with $n_e/n_0 \ge 90\%$. For these cases, only the X mode is excited and the R mode disappears due to its strong coupling. The results are well in line with the linear kinetic theory of ECMI. The properties of both the R and X modes are consistent with the observed SKR emissions. This raises questions about the nature of the measured electric field fluctuations within ``presumed'' SKR sources. The study provides new insights into the ECMI process relevant to SKR emission mechanisms

Highly efficient visible-light photocatalytic ethane oxidation into ethyl hydroperoxide as a radical reservoir

Photocatalytic ethane conversion into value-added chemicals is a great challenge especially under visible light irradiation. The production of ethyl hydroperoxide (CH CH OOH), which is a promising radical reservoir for regulating the oxidative stress in cells, is even more challenging due to its facile decomposition. Here, we demonstrated a design of a highly efficient visible-light-responsive photocatalyst, Au/WO , for ethane oxidation into CH CH OOH, achieving an impressive yield of 1887 μmol g in two hours under visible light irradiation at room temperature for the first time. Furthermore, thermal energy was introduced into the photocatalytic system to increase the driving force for ethane oxidation, enhancing CH CH OOH production by six times to 11 233 μmol g at 100 °C and achieving a significant apparent quantum efficiency of 17.9% at 450 nm. In addition, trapping active species and isotope-labeling reactants revealed the reaction pathway. These findings pave the way for scalable ethane conversion into CH CH OOH as a potential anticancer drug

Unraveling the Prognostic Significance of Rgs Gene Family in Gastric Cancer and the Potential Implication of Rgs4 in Regulating Tumor-infiltrating Fibroblast

Regulator of G-protein signaling (RGS) proteins are regulators of signal transduction mediated by G protein-coupled receptors (GPCRs). Current studies have shown that some molecules in the RGS gene family are related to the occurrence, development and poor prognosis of malignant tumors. However, the RGS gene family has been rarely studied in gastric cancer. In this study, we explored the mutation and expression profile of RGS gene family in gastric cancer, and evaluated the prognostic value of RGS expression. Then we established a prognostic model based on RGS gene family and performed functional analysis. Further studies showed that RGS4, as an independent prognostic predictor, may play an important role in regulating fibroblasts in the immune microenvironment. In conclusion, this study explores the value of RGS gene family in gastric cancer, which is of great significance for predicting the prognosis and guiding the treatment of gastric cancer

Improving Robust Fairness via Balance Adversarial Training

Adversarial training (AT) methods are effective against adversarial attacks, yet they introduce severe disparity of accuracy and robustness between different classes, known as the robust fairness problem. Previously proposed Fair Robust Learning (FRL) adaptively reweights different classes to improve fairness. However, the performance of the better-performed classes decreases, leading to a strong performance drop. In this paper, we observed two unfair phenomena during adversarial training: different difficulties in generating adversarial examples from each class (source-class fairness) and disparate target class tendencies when generating adversarial examples (target-class fairness). From the observations, we propose Balance Adversarial Training (BAT) to address the robust fairness problem. Regarding source-class fairness, we adjust the attack strength and difficulties of each class to generate samples near the decision boundary for easier and fairer model learning; considering target-class fairness, by introducing a uniform distribution constraint, we encourage the adversarial example generation process for each class with a fair tendency. Extensive experiments conducted on multiple datasets (CIFAR-10, CIFAR-100, and ImageNette) demonstrate that our method can significantly outperform other baselines in mitigating the robust fairness problem (+5-10\% on the worst class accuracy

GEOGLE: context mining tool for the correlation between gene expression and the phenotypic distinction

<p>Abstract</p> <p>Background</p> <p>In the post-genomic era, the development of high-throughput gene expression detection technology provides huge amounts of experimental data, which challenges the traditional pipelines for data processing and analyzing in scientific researches.</p> <p>Results</p> <p>In our work, we integrated gene expression information from Gene Expression Omnibus (GEO), biomedical ontology from Medical Subject Headings (MeSH) and signaling pathway knowledge from sigPathway entries to develop a context mining tool for gene expression analysis – GEOGLE. GEOGLE offers a rapid and convenient way for searching relevant experimental datasets, pathways and biological terms according to multiple types of queries: including biomedical vocabularies, GDS IDs, gene IDs, pathway names and signature list. Moreover, GEOGLE summarizes the signature genes from a subset of GDSes and estimates the correlation between gene expression and the phenotypic distinction with an integrated p value.</p> <p>Conclusion</p> <p>This approach performing global searching of expression data may expand the traditional way of collecting heterogeneous gene expression experiment data. GEOGLE is a novel tool that provides researchers a quantitative way to understand the correlation between gene expression and phenotypic distinction through meta-analysis of gene expression datasets from different experiments, as well as the biological meaning behind. The web site and user guide of GEOGLE are available at: <url>http://omics.biosino.org:14000/kweb/workflow.jsp?id=00020</url></p

Editing Large Language Models: Problems, Methods, and Opportunities

Despite the ability to train capable LLMs, the methodology for maintaining their relevancy and rectifying errors remains elusive. To this end, the past few years have witnessed a surge in techniques for editing LLMs, the objective of which is to efficiently alter the behavior of LLMs within a specific domain without negatively impacting performance across other inputs. This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs. In particular, we provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal. We also build a new benchmark dataset to facilitate a more robust evaluation and pinpoint enduring issues intrinsic to existing techniques. Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context. Code and datasets are available at https://github.com/zjunlp/EasyEdit.Comment: EMNLP 2023. Updated with new experiment

A Mild Dyssynchronous Contraction Pattern Detected by SPECT Myocardial Perfusion Imaging Predicts Super-Response to Cardiac Resynchronization Therapy

Background: Using single photon emission computed tomography myocardial perfusion imaging (SPECT MPI) with phase analysis (PA), we aimed to identify the predictive value of a new contraction pattern in cardiac resynchronization therapy (CRT) response. Methods: Left ventricular mechanical dyssynchrony (LVMD) was evaluated using SPECT MPI with PA in non-ischemic dilated cardiomyopathy (DCM) patients with left bundle branch block (LBBB) indicated for CRT. CRT super-response was defined as LV ejection fraction (EF) ≥50% or an absolute increase of LVEF \u3e15%. The LV contraction was categorized as the mild dyssynchronous pattern when the phase standard deviation (PSD) ≤ 40.3° and phase histogram bandwidth (PBW) ≤ 111.9°, otherwise it was defined as severe dyssynchronous pattern which was further characterized as U-shaped, heterogeneous or homogenous pattern. Results: The final cohort comprised 74 patients, including 32 (43.2%) in mild dyssynchronous group, 17 (23%) in U-shaped group, 19 (25.7%) in heterogeneous group, and 6 (8.1%) in homogenous group. The mild dyssynchronous group had lower PSD and PBW than U-shaped, heterogeneous, and homogenous groups ( \u3c 0.0001). Compared to patients with the heterogeneous pattern, the odds ratios (ORs) with 95% confidence intervals (CIs) for CRT super-response were 10.182(2.43-42.663), 12.8(2.545-64.372), and 2.667(0.327-21.773) for patients with mild dyssynchronous, U-shaped, and homogenous pattern, respectively. After multivariable adjustment, mild dyssynchronous group remained associated with increased CRT super-response (adjusted OR 5.709, 95% CI 1.152-28.293). Kaplan-Meier curves showed that mild dyssynchronous group demonstrated a better long-term prognosis. Conclusions: The mild dyssynchronous pattern in patients with DCM is associated with an increased CRT super-response and better long-term prognosis