34 research outputs found
The Lottery Ticket Hypothesis for Vision Transformers
The conventional lottery ticket hypothesis (LTH) claims that there exists a
sparse subnetwork within a dense neural network and a proper random
initialization method, called the winning ticket, such that it can be trained
from scratch to almost as good as the dense counterpart. Meanwhile, the
research of LTH in vision transformers (ViTs) is scarcely evaluated. In this
paper, we first show that the conventional winning ticket is hard to find at
weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs
to input images consisting of image patches inspired by the input dependence of
ViTs. That is, there exists a subset of input image patches such that a ViT can
be trained from scratch by using only this subset of patches and achieve
similar accuracy to the ViTs trained by using all image patches. We call this
subset of input patches the winning tickets, which represent a significant
amount of information in the input. Furthermore, we present a simple yet
effective method to find the winning tickets in input patches for various types
of ViT, including DeiT, LV-ViT, and Swin Transformers. More specifically, we
use a ticket selector to generate the winning tickets based on the
informativeness of patches. Meanwhile, we build another randomly selected
subset of patches for comparison, and the experiments show that there is clear
difference between the performance of models trained with winning tickets and
randomly selected subsets
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Vision transformers (ViTs) have recently obtained success in many
applications, but their intensive computation and heavy memory usage at both
training and inference time limit their generalization. Previous compression
algorithms usually start from the pre-trained dense models and only focus on
efficient inference, while time-consuming training is still unavoidable. In
contrast, this paper points out that the million-scale training data is
redundant, which is the fundamental reason for the tedious training. To address
the issue, this paper aims to introduce sparsity into data and proposes an
end-to-end efficient training framework from three sparse perspectives, dubbed
Tri-Level E-ViT. Specifically, we leverage a hierarchical data redundancy
reduction scheme, by exploring the sparsity under three levels: number of
training examples in the dataset, number of patches (tokens) in each example,
and number of connections between tokens that lie in attention weights. With
extensive experiments, we demonstrate that our proposed technique can
noticeably accelerate training for various ViT architectures while maintaining
accuracy. Remarkably, under certain ratios, we are able to improve the ViT
accuracy rather than compromising it. For example, we can achieve 15.2% speedup
with 72.6% (+0.4) Top-1 accuracy on Deit-T, and 15.7% speedup with 79.9% (+0.1)
Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.Comment: AAAI 202
The Efficient and Convenient Extracting Uranium from Water by a Uranyl-Ion Affine Microgel Container
Uranium is an indispensable part of the nuclear industry that benefits us, but its consequent pollution of water bodies also makes a far-reaching impact on human society. The rapid, efficient and convenient extraction of uranium from water is to be a top priority. Thanks to the super hydrophilic and fast adsorption rate of microgel, it has been the ideal adsorbent in water; however, it was too difficult to recover the microgel after adsorption, which limited its practical applications. Here, we developed a uranyl-ion affine and recyclable microgel container that has not only the rapid swelling rate of microgel particles but also allows the detection of the adsorption saturation process by the naked eye
Effects of dentifrice containing hydroxyapatite on dentinal tubule occlusion and aqueous hexavalent chromium cations sorption: a preliminary study.
In order to endow environmental protection features to dentifrice, hydroxyapatite (HA) was added to ordinary dentifrice. The effects on dentinal tubule occlusion and surface mineralization were compared after brushing dentine discs with dentifrice with or without HA. The two types of dentifrice were then added to 100 µg/ml of hexavalent chromium cation (Cr(6+)) solution in order to evaluate their capacities of adsorbing Cr(6+) from water. Our results showed that the dentifrice containing HA was significantly better than the ordinary dentifrice in occluding the dentinal tubules with a plugging rate greater than 90%. Moreover, the effect of the HA dentifrice was persistent and energy-dispersive spectrometer (EDS) revealed that the atomic percentages of calcium and phosphorus on the surface of dentine discs increased significantly. Adding HA to ordinary dentifrice significantly enhanced the ability of dentifrice to adsorb Cr(6+) from water with the removal rate up to 52.36%. In addition, the sorption was stable. Our study suggests that HA can be added to ordinary dentifrice to obtain dentifrice that has both relieving dentin hypersensitivity benefits and also helps to control environmental pollution
Multi-Factors Cooperatively Actuated Photonic Hydrogel Aptasensors for Facile, Label-Free and Colorimetric Detection of Lysozyme
Responsive two-dimensional photonic crystal (2DPC) hydrogels have been widely used as smart sensing materials for constructing various optical sensors to accurately detect different target analytes. Herein, we report photonic hydrogel aptasensors based on aptamer-functionalized 2DPC poly(acrylamide-acrylic acid-N-tert-butyl acrylamide) hydrogels for facile, label-free and colorimetric detection of lysozyme in human serum. The constructed photonic hydrogel aptasensors undergo shrinkage upon exposure to lysozyme solution through multi-factors cooperative actuation. Here, the specific binding between the aptamer and lysozyme, and the simultaneous interactions between carboxyl anions and N-tert-butyl groups with lysozyme, increase the cross-linking density of the hydrogel, leading to its shrinkage. The aptasensors’ shrinkage decreases the particle spacing of the 2DPC embedded in the hydrogel network. It can be simply monitored by measuring the Debye diffraction ring of the photonic hydrogel aptasensors using a laser pointer and a ruler without needing sophisticated apparatus. The significant shrinkage of the aptasensors can be observed by the naked eye via the hydrogel size and color change. The aptasensors show good sensitivity with a limit of detection of 1.8 nM, high selectivity and anti-interference for the detection of lysozyme. The photonic hydrogel aptasensors have been successfully used to accurately determine the concentration of lysozyme in human serum. Therefore, novel photonic hydrogel aptasensors can be constructed by designing functional monomers and aptamers that can specifically bind target analytes
Dentinal tubule occlusion after mechanical brushing with distilled water.
<p>(SEM, 5000×) (a, d) Dentinal tubules were empty with no occlusion. (b, e) Almost all dentinal tubules were empty with very few dentinal tubules showing little occluding materials. (c, f) Most of the dentinal tubules showed various levels of sediment blockage.</p