The Lottery Ticket Hypothesis for Vision Transformers

The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method, called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input images consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the winning tickets, which represent a significant amount of information in the input. Furthermore, we present a simple yet effective method to find the winning tickets in input patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. More specifically, we use a ticket selector to generate the winning tickets based on the informativeness of patches. Meanwhile, we build another randomly selected subset of patches for comparison, and the experiments show that there is clear difference between the performance of models trained with winning tickets and randomly selected subsets

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training

Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage at both training and inference time limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference, while time-consuming training is still unavoidable. In contrast, this paper points out that the million-scale training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy reduction scheme, by exploring the sparsity under three levels: number of training examples in the dataset, number of patches (tokens) in each example, and number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate training for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 15.2% speedup with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.Comment: AAAI 202

The Efficient and Convenient Extracting Uranium from Water by a Uranyl-Ion Affine Microgel Container

Uranium is an indispensable part of the nuclear industry that benefits us, but its consequent pollution of water bodies also makes a far-reaching impact on human society. The rapid, efficient and convenient extraction of uranium from water is to be a top priority. Thanks to the super hydrophilic and fast adsorption rate of microgel, it has been the ideal adsorbent in water; however, it was too difficult to recover the microgel after adsorption, which limited its practical applications. Here, we developed a uranyl-ion affine and recyclable microgel container that has not only the rapid swelling rate of microgel particles but also allows the detection of the adsorption saturation process by the naked eye

Effects of dentifrice containing hydroxyapatite on dentinal tubule occlusion and aqueous hexavalent chromium cations sorption: a preliminary study.

In order to endow environmental protection features to dentifrice, hydroxyapatite (HA) was added to ordinary dentifrice. The effects on dentinal tubule occlusion and surface mineralization were compared after brushing dentine discs with dentifrice with or without HA. The two types of dentifrice were then added to 100 µg/ml of hexavalent chromium cation (Cr(6+)) solution in order to evaluate their capacities of adsorbing Cr(6+) from water. Our results showed that the dentifrice containing HA was significantly better than the ordinary dentifrice in occluding the dentinal tubules with a plugging rate greater than 90%. Moreover, the effect of the HA dentifrice was persistent and energy-dispersive spectrometer (EDS) revealed that the atomic percentages of calcium and phosphorus on the surface of dentine discs increased significantly. Adding HA to ordinary dentifrice significantly enhanced the ability of dentifrice to adsorb Cr(6+) from water with the removal rate up to 52.36%. In addition, the sorption was stable. Our study suggests that HA can be added to ordinary dentifrice to obtain dentifrice that has both relieving dentin hypersensitivity benefits and also helps to control environmental pollution

Multi-Factors Cooperatively Actuated Photonic Hydrogel Aptasensors for Facile, Label-Free and Colorimetric Detection of Lysozyme

Responsive two-dimensional photonic crystal (2DPC) hydrogels have been widely used as smart sensing materials for constructing various optical sensors to accurately detect different target analytes. Herein, we report photonic hydrogel aptasensors based on aptamer-functionalized 2DPC poly(acrylamide-acrylic acid-N-tert-butyl acrylamide) hydrogels for facile, label-free and colorimetric detection of lysozyme in human serum. The constructed photonic hydrogel aptasensors undergo shrinkage upon exposure to lysozyme solution through multi-factors cooperative actuation. Here, the specific binding between the aptamer and lysozyme, and the simultaneous interactions between carboxyl anions and N-tert-butyl groups with lysozyme, increase the cross-linking density of the hydrogel, leading to its shrinkage. The aptasensors’ shrinkage decreases the particle spacing of the 2DPC embedded in the hydrogel network. It can be simply monitored by measuring the Debye diffraction ring of the photonic hydrogel aptasensors using a laser pointer and a ruler without needing sophisticated apparatus. The significant shrinkage of the aptasensors can be observed by the naked eye via the hydrogel size and color change. The aptasensors show good sensitivity with a limit of detection of 1.8 nM, high selectivity and anti-interference for the detection of lysozyme. The photonic hydrogel aptasensors have been successfully used to accurately determine the concentration of lysozyme in human serum. Therefore, novel photonic hydrogel aptasensors can be constructed by designing functional monomers and aptamers that can specifically bind target analytes

Dentinal tubule occlusion after mechanical brushing with distilled water.

<p>(SEM, 5000×) (a, d) Dentinal tubules were empty with no occlusion. (b, e) Almost all dentinal tubules were empty with very few dentinal tubules showing little occluding materials. (c, f) Most of the dentinal tubules showed various levels of sediment blockage.</p