LipsFormer: Introducing Lipschitz Continuity to Vision Transformers

We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny based on Swin Transformer training for 300 epochs can obtain 82.7\% without any learning rate warmup. Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training for 300 epochs achieves a top-1 accuracy of 83.5\% with 4.7G FLOPs and 24M parameters. The code will be released at \url{https://github.com/IDEA-Research/LipsFormer}.Comment: To appear in ICLR 2023, our code will be public at https://github.com/IDEA-Research/LipsForme

Fight Fire with Fire: Combating Adversarial Patch Attacks using Pattern-randomized Defensive Patches

Object detection has found extensive applications in various tasks, but it is also susceptible to adversarial patch attacks. Existing defense methods often necessitate modifications to the target model or result in unacceptable time overhead. In this paper, we adopt a counterattack approach, following the principle of "fight fire with fire," and propose a novel and general methodology for defending adversarial attacks. We utilize an active defense strategy by injecting two types of defensive patches, canary and woodpecker, into the input to proactively probe or weaken potential adversarial patches without altering the target model. Moreover, inspired by randomization techniques employed in software security, we employ randomized canary and woodpecker injection patterns to defend against defense-aware attacks. The effectiveness and practicality of the proposed method are demonstrated through comprehensive experiments. The results illustrate that canary and woodpecker achieve high performance, even when confronted with unknown attack methods, while incurring limited time overhead. Furthermore, our method also exhibits sufficient robustness against defense-aware attacks, as evidenced by adaptive attack experiments

Extracellular vesicles for breast cancer diagnosis and therapy

Breast cancer is still suffering from its poor diagnosis and the lack of effective treatment. Despite of recent development of some novel chemicals, which are found to have inspiring therapeutic effects in vitro, their outcomes in clinic trails are disappointing, mainly due to the lack of suitable therapeutic vehicles. Thanks to their ability to encapsulate bio-molecules, extracellular vesicles (EVs), including exosomes, microvesicles, and apoptotic bodies, hold great promise in becoming a suitable candidate in the breast cancer diagnosis and therapy. Currently, EVs are increasingly evaluated as potential indicators in the diagnosis of breast cancer since they are actively involved in different stages of breast cancer development, including promoting cancer occurrence and metastasis, establishing tumor ecology, and promoting tumor growth. Moreover, they are also considered as promising new platforms in breast cancer therapy. Here, we discuss the potential therapeutic applications of EVs, including EVs as biomarkers for diagnosis and therapeutic drug delivery to tumor sites. The promising data and technologies indicate the potential applicability of EVs in clinical management of patients with breast cancer.publishedVersio

CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds

We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches. Then, to recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module to directly aggregate fine-grained spatial information from backbone for further proposal refinement. It is memory-and-computation efficient and can better encode the geometry-specific features of each 3D proposal. Our model achieves state-of-the-art 3D detection performance with remarkable gains of +\textit{3.6\%} on ScanNet V2 and +\textit{2.6}\% on SUN RGB-D in term of [email protected]. Code will be available at https://github.com/Haiyang-W/CAGroup3D.Comment: Accept by NeurIPS202

Small Object Detection Based on Two-Stage Calculation Transformer

Despite the current small object detection task has achieved significant improvements, it still suffers from some problems. For example, it is a challenge to extract small object features because of little information in the scene of small objects, which may lose the original feature information of small object, resulting in poor detection results. To address this problem, this paper proposes a two-stage calculation Transformer (TCT) based small object detection network. Firstly, a two-stage calculation Transformer is embedded in the backbone feature extraction network for feature enhancement. Based on the traditional Transformer values computation, multiple 1D dilated convolutional layer branches with different feature fusions are utilized to implement global self-attention for the purpose of improving the feature representation and information interaction. Secondly, this paper proposes an effective residual connection module to improve the low-efficiency convolution and activation of the current CSPLayer, which helps to advance the information flow and learn more rich contextual details. Finally, this paper proposes a feature fusion and refinement module for fusing multi-scale features and improving the target feature representation capability. Quantitative and qualitative experiments on PASCAL VOC2007+2012 dataset, COCO2017 dataset and TinyPerson dataset show that the proposed algorithm has better ability of target feature extraction and higher detection accuracy for small target detection, compared with YOLOX

Advances of deep learning in electrical impedance tomography image reconstruction

Electrical impedance tomography (EIT) has been widely used in biomedical research because of its advantages of real-time imaging and nature of being non-invasive and radiation-free. Additionally, it can reconstruct the distribution or changes in electrical properties in the sensing area. Recently, with the significant advancements in the use of deep learning in intelligent medical imaging, EIT image reconstruction based on deep learning has received considerable attention. This study introduces the basic principles of EIT and summarizes the application progress of deep learning in EIT image reconstruction with regards to three aspects: a single network reconstruction, deep learning combined with traditional algorithm reconstruction, and multiple network hybrid reconstruction. In future, optimizing the datasets may be the main challenge in applying deep learning for EIT image reconstruction. Adopting a better network structure, focusing on the joint reconstruction of EIT and traditional algorithms, and using multimodal deep learning-based EIT may be the solution to existing problems. In general, deep learning offers a fresh approach for improving the performance of EIT image reconstruction and could be the foundation for building an intelligent integrated EIT diagnostic system in the future

Resistin induces multidrug resistance in myeloma by inhibiting cell death and upregulating ABC transporter expression

Despite advances in therapy, multiple myeloma remains incurable, with a high frequency of relapse. This suggests the need to identify additional factors that contribute to drug resistance. Our previous studies revealed that bone marrow adipocytes promote resistance to chemotherapy in myeloma through adipocyte-secreted adipokines, but the mechanism underlying this effect and the specific adipokines involved are not well understood. We proposed to determine the role of resistin, an adipokine that is secreted by adipocytes, in chemotherapy resistance in myeloma. We found that resistin abrogated chemotherapy-induced apoptosis in established myeloma cell lines and primary myeloma samples. Resistin inhibited chemotherapy-induced caspase cleavage through the NF-κB and PI3K/Akt pathways. Resistin also increased the expression and drug efflux function of ATP-binding cassette (ABC) transporters in myeloma cells through decreasing the expression of both DNA methyltransferases DNMT1 and DNMT3a and the methylation levels of ABC gene promoters. In vivo studies further demonstrated the protective effect of resistin in chemotherapy-induced apoptosis. Our study thus reveals a new biological function of resistin in the pathogenesis of myeloma, with the implication that targeting resistin could be a potential strategy to prevent or overcome multidrug resistance in myeloma