The Lottery Ticket Hypothesis for Vision Transformers

The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method, called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input images consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the winning tickets, which represent a significant amount of information in the input. Furthermore, we present a simple yet effective method to find the winning tickets in input patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. More specifically, we use a ticket selector to generate the winning tickets based on the informativeness of patches. Meanwhile, we build another randomly selected subset of patches for comparison, and the experiments show that there is clear difference between the performance of models trained with winning tickets and randomly selected subsets

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.Comment: AAAI 202

Alternating levels versus all levels mini-plate fixation in open door cervical laminoplasty for treatment of degenerative cervical myelopathy

Objectiveː To investigate clinical and radiological results of alternating levels compared with all levels mini-plate fixation in open door cervical laminoplasty for treatment of degenerative cervical myelopathy. METHODSː From January 2011 to April 2014, 64 patients sustained degenerative cervical myelopathy, who underwent cervical laminoplasty with alternating levels (31 patients in group A) or all levels plate fixation (33 patients in group B) were included in this retrospectively study. Clinical and radiological results were calculated.  RESULTSːThe mean cost for group B was higher than group A(P < 0.05). No statistical difference was found in the mean operation time, blood loss, axial symptoms，C5 palsy, improvement in JOA and NDI score between group A and B. Open angle in mini-plate fixed levels was significantly more than that in suture fixed levels(P < 0.05). No statistical difference was found in drift back of spinal cord, APD, Pavlov's ratio, CCI and ROM between mini-plate fixed levels and suture fixed levels.CONCLUSIONSː Open door laminoplasty at alternating levels mini-plate fixation is an economical surgical method and can obtain similar satisfactory clinical and radiological results compared to all levels mini-plate fixation. 

Modeling the initial mechanical response and yielding behavior of gelled crude oil

The initial mechanical response and yielding behavior of gelled crude oil under constant shear rate conditions were investigated. By putting the Maxwell mechanical analog and a special dashpot in parallel, a quasi-Jeffreys model was obtained. The kinetic equation of the structural parameter in the Houska model was simplified reasonably so that a simplified constitutive equation of the special dashpot was expressed. By introducing a damage factor into the constitutive equation of the special dashpot and the Maxwell mechanical analog, we established a constitutive equation of the quasi-Jeffreys model. Rheological tests of gelled crude oil were conducted by imposing constant shear rates and the relationship between the shear stress and shear strain under different shear rates was plotted. It is found that the constitutive equation can fit the experimental data well under a wide range of shear rates. Based on the fitted parameters in the quasi-Jeffreys model, the shear stress changing rules of the Maxwell mechanical analog and the special dashpot were calculated and analyzed. It is found that the critical yield strain and the corresponding shear strain where shear stress of the Maxwell analog is the maximum change slightly under different shear rates. And then a critical damage softening strain which is irrelevant to the shearing conditions was put forward to describe the yielding behavior of gelled crude oil

Hydrogel-based treatments for spinal cord injuries

Spinal cord injury (SCI) is characterized by damage resulting in dysfunction of the spinal cord. Hydrogels are common biomaterials that play an important role in the treatment of SCI. Hydrogels are biocompatible, and some have electrical conductivity that are compatible with spinal cord tissues. Hydrogels have a high drug-carrying capacity, allowing them to be used for SCI treatment through the loading of various types of active substances, drugs, or cells. We first discuss the basic anatomy and physiology of the human spinal cord and briefly discuss SCI and its treatment. Then, we describe different treatment strategies for SCI. We further discuss the crosslinking methods and classification of hydrogels and detail hydrogel biomaterials prepared using different processing methods for the treatment of SCI. Finally, we analyze the future applications and limitations of hydrogels for SCI. The development of biomaterials opens up new possibilities and options for the treatment of SCI. Thus, our findings will inspire scholars in related fields and promote the development of hydrogel therapy for SCI

A Novel NADP(H)-Dependent 7alpha-HSDH: Discovery and Construction of Substrate Selectivity Mutant by C-Terminal Truncation

7α-Hydroxysteroid dehydrogenase (7α-HSDH) plays an important role in the biosynthesis of tauroursodeoxycholic acid (TUDCA) using complex substrate chicken bile powder as raw material. However, chicken bile powder contains 4.74% taurocholic acid (TCA), and a new by-product tauroursocholic acid (TUCA) will be produced, having the risk of causing colorectal cancer. Here, we obtained a novel NADP(H)-dependent 7α-HSDH with good thermostability from Ursus thibetanus gut microbiota (named St-2-2). St-2-2 could catalyze taurochenodeoxycholic acid (TCDCA) and TCA with the catalytic activity of 128.13 and 269.39 U/mg, respectively. Interestingly, by a structure-based C-terminal truncation strategy, St-2-2△C10 only remained catalytic activity on TCDCA (14.19 U/mg) and had no activity on TCA. As a result, it can selectively catalyze TCDCA in waste chicken bile powder. MD simulation and structural analysis indicated that enhanced surface hydrophilicity and improved C-terminal rigidity affected the entry and exit of substrates. Hydrogen bond interactions between different subunits and interaction changes in Phe249 of the C-terminal loop inverted the substrate catalytic activity. This is the first report on substrate selectivity of 7α-HSDH by C-terminal truncation strategy and it can be extended to other 7α-HSDHs (J-1-1, S1-a-1)

sj-docx-1-pie-10.1177_09544089231165128 - Supplemental material for Casing failure mechanism and slip anchoring optimization of the compression packer in the oil and gas well

Supplemental material, sj-docx-1-pie-10.1177_09544089231165128 for Casing failure mechanism and slip anchoring optimization of the compression packer in the oil and gas well by Yang Tang, Jie Wang, Minghai Zhou, Guangyao Li and Xiao Xiao in Proceedings of the Institution of Mechanical Engineers, Part E: Journal of Process Mechanical Engineering</p

Studies on the anti-psoriasis effects and its mechanism of a dual JAK2/FLT3 inhibitor flonoltinib maleate

Psoriasis is a chronic, inflammatory autoimmune disease mediated by T cells, and characterized with abnormal proliferation and differentiation of keratinocytes, and inflammatory infiltration. The Janus kinase-signal transducer and activator of transcription (JAK-STAT) pathway has been identified to play essential roles in mediating various of biological processes, and is closely related to autoimmune diseases. Dendritic cells (DCs) are important antigen presenting cells and play an important regulatory role in T cells. The proliferation, differentiation and function of DCs are regulated by JAK and FMS-like tyrosine kinase 3 (FLT3) signal pathways. Flonoltinib maleate (FM), a high selectivity dual JAK2/FLT3 inhibitor with IC50 values of 0.8 nM and 15 nM for JAK2 and FLT3, respectively, was developed by our laboratory. Moreover, FM was a potent JAK2 inhibitor with 863-fold and 696-fold selectivity over JAK1 and JAK3, respectively. In this study, the anti-psoriasis activity of FM was evaluated both in vitro and in vivo. FM effectively inhibited the proliferation of HaCaT, the inflammatory keratinocyte induced by M5 and markedly suppressed the generation and differentiation of DCs from bone marrow (BM), and inhibited the expression of FLT3 in DCs in vitro. FM effectively inhibited the ear thickening and improved the pathological changes of the ear in interleukin (IL)-23-induced psoriasis-like acanthosis mouse model. Further in keratin 14-vascular endothelial growth factor (K14-VEGF) transgenic homozygous mice model, FM could obviously improve the psoriatic symptom and pathological changes, significantly inhibit the generations of Th1 and Th17 cells in the spleen, and the accumulations of DCs in the ears. FM could also significantly reduce the expression of various inflammatory factors both in C57BL/6 and K14-VEGF mice ears, and the serum of K14-VEGF mice. Mechanism revealed that FM effectively suppressed the phosphorylation of JAK2, STAT3 and STAT5 in inflammatory keratinocytes and the mice ears of C57BL/6 and K14-VEGF, as well as the phosphorylation of FLT3 in K14-VEGF mice ears. In conclusion, FM plays an excellent anti-psoriasis activity, including inhibiting keratinocyte proliferation and regulating inflammatory response through inhibiting JAK2 and FLT3 signaling pathway

<i>Piper nigrum</i> Extract Inhibits the Growth of Human Colorectal Cancer HT-29 Cells by Inducing p53-Mediated Apoptosis

Colorectal cancer (CRC) is a prevalent malignancy of the digestive tract with the second highest mortality rate globally. Piper nigrum is a widely used traditional medicinal plant, exhibiting antitumor activity against various tumor cells. At present, research on the effect of Piper nigrum on CRC is limited to in vitro cytotoxicity, lacking comprehensive mechanism investigations. This study aimed to explore the inhibitory effect and mechanism of Piper nigrum extract (PNE) on HT-29 cells. Firstly, we identified the chemical components of PNE. Then, MTT assay, colony formation assay, JC-1 staining, and flow cytometry were used to analyze the effect of PNE on HT-29 cells in vitro. A xenograft model, histopathological examination, immunohistochemistry, and western blot were used to evaluate the tumor growth inhibitory activity and mechanism of PNE in vivo. The results indicated that PNE could inhibit cell proliferation and colony formation, reduce mitochondrial membrane potential, induce cell apoptosis in vitro, and inhibit tumor growth in vivo. Furthermore, PNE could regulate p53 and its downstream proteins, and subsequently activate the caspase-3 pathway. In summary, PNE probably induced apoptosis of HT-29 cells through the mitochondrial pathway mediated by p53. All these results suggested that PNE might be a potential natural-origin anti-CRC drug candidate