Bridging The Gaps Between Token Pruning and Full Pre-training via Masked Fine-tuning

Despite the success of transformers on various computer vision tasks, they suffer from excessive memory and computational cost. Some works present dynamic vision transformers to accelerate inference by pruning redundant tokens. A key to improving token pruning is using well-trained models as initialization for faster convergence and better performance. However, current base models usually adopt full image training, i.e., using full images as inputs and keeping the whole feature maps through the forward process, which causes inconsistencies with dynamic models that gradually reduce tokens, including calculation pattern, information amount and token selection strategy inconsistencies. Inspired by MAE which performs masking and reconstruction self-supervised task, we devise masked fine-tuning to bridge the gaps between pre-trained base models used for initialization and token pruning based dynamic vision transformers, by masking image patches and predicting the image class label based on left unmasked patches. Extensive experiments on ImageNet demonstrate that base models via masked fine-tuning gain strong occlusion robustness and ability against information loss. With this better initialization, Dynamic ViT achieves higher accuracies, especially under large token pruning ratios (e.g., 81.9% vs. 81.3%, and 62.3% vs. 58.9% for DeiT based Dynamic ViT/0.8 and Dynamic ViT/0.3). Moreover, we apply our method into different token pruning based dynamic vision transformers, different pre-trained models and randomly initialized models to demonstrate the generalization ability.Comment: Submitted to TI

Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding

Multimodal transformer exhibits high capacity and flexibility to align image and text for visual grounding. However, the existing encoder-only grounding framework (e.g., TransVG) suffers from heavy computation due to the self-attention operation with quadratic time complexity. To address this issue, we present a new multimodal transformer architecture, coined as Dynamic Mutilmodal DETR (Dynamic MDETR), by decoupling the whole grounding process into encoding and decoding phases. The key observation is that there exists high spatial redundancy in images. Thus, we devise a new dynamic multimodal transformer decoder by exploiting this sparsity prior to speed up the visual grounding process. Specifically, our dynamic decoder is composed of a 2D adaptive sampling module and a text guided decoding module. The sampling module aims to select these informative patches by predicting the offsets with respect to a reference point, while the decoding module works for extracting the grounded object information by performing cross attention between image features and text features. These two modules are stacked alternatively to gradually bridge the modality gap and iteratively refine the reference point of grounded object, eventually realizing the objective of visual grounding. Extensive experiments on five benchmarks demonstrate that our proposed Dynamic MDETR achieves competitive trade-offs between computation and accuracy. Notably, using only 9% feature points in the decoder, we can reduce ~44% GFLOPs of the multimodal transformer, but still get higher accuracy than the encoder-only counterpart. In addition, to verify its generalization ability and scale up our Dynamic MDETR, we build the first one-stage CLIP empowered visual grounding framework, and achieve the state-of-the-art performance on these benchmarks.Comment: Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) in October 202

Reusing recycled fibers in high-value fiber-reinforced polymer composites: Improving bending strength by surface cleaning

Glass fiber-reinforced polymer (GFRP) composites and carbon fiber-reinforced polymer (CFRP) composites were recycled using superheated steam. Recycled glass fibers (R-GFs) and recycled carbon fibers (R-CFs) were surface treated for reuse as fiber-reinforced polymer (FRP) composites. Treated R-GFs (TR-GFs) and treated R-CFs (TR-CFs) were characterized by scanning electron microscopy (SEM) and remanufactured by vacuum-assisted resin transfer molding (VARTM). Most residual resin impurities were removed by surface treatment. Analysis indicated no adverse effect of surface treatment on bending strength. The mechanical properties of the TR-GF reinforced polymer (TR-GFRP) and TR-CF reinforced polymer (TR-CFRP) composites were determined and compared with those of R-GF reinforced polymer (R-GFRP) and R-CF reinforced polymer (R-CFRP). The bending strengths of R-GFRP (26%) and R-CFRP (49%) were very low, compared to that of virgin glass fiber-reinforced polymer (V-GFRP) and that of virgin carbon fiber-reinforced polymer (V-CFRP). The bending strength of TR-GFRP composites was improved to about 90% of that of V-GFRP, and the bending strength of TR-CFRP composites was improved to about 80% of that of V-CFRP.ArticleCOMPOSITES SCIENCE AND TECHNOLOGY. 72(11):1298-1303 (2012)journal articl

Maternal Influenza Infection Causes Marked Behavioral and Pharmacological Changes in the Offspring

Maternal viral infection is known to increase the risk for schizophrenia and autism in the offspring. Using this observation in an animal model, we find that respiratory infection of pregnant mice (both BALB/c and C57BL/6 strains) with the human influenza virus yields offspring that display highly abnormal behavioral responses as adults. As in schizophrenia and autism, these offspring display deficits in prepulse inhibition (PPI) in the acoustic startle response. Compared with control mice, the infected mice also display striking responses to the acute administration of antipsychotic (clozapine and chlorpromazine) and psychomimetic (ketamine) drugs. Moreover, these mice are deficient in exploratory behavior in both open-field and novel-object tests, and they are deficient in social interaction. At least some of these behavioral changes likely are attributable to the maternal immune response itself. That is, maternal injection of the synthetic double-stranded RNA polyinosinic-polycytidylic acid causes a PPI deficit in the offspring in the absence of virus. Therefore, maternal viral infection has a profound effect on the behavior of adult offspring, probably via an effect of the maternal immune response on the fetus

A scaling law for distinct electrocaloric cooling performance in low-dimensional organic, relaxor and anti-ferroelectrics.

Electrocaloric (EC) materials show promise in eco-friendly solid-state refrigeration and integrable on-chip thermal management. While direct measurement of EC thin-films still remains challenging, a generic theoretical framework for quantifying the cooling properties of rich EC materials including normal-, relaxor-, organic- and anti-ferroelectrics is imperative for exploiting new flexible and room-temperature cooling alternatives. Here, we present a versatile theory that combines Master equation with Maxwell relations and analytically relates the macroscopic cooling responses in EC materials with the intrinsic diffuseness of phase transitions and correlation characteristics. Under increased electric fields, both EC entropy and adiabatic temperature changes increase quadratically initially, followed by further linear growth and eventual gradual saturation. The upper bound of entropy change (∆Smax) is limited by distinct correlation volumes (V cr ) and transition diffuseness. The linearity between V cr and the transition diffuseness is emphasized, while ∆Smax = 300 kJ/(K.m3) is obtained for Pb0.8Ba0.2ZrO3. The ∆Smax in antiferroelectric Pb0.95Zr0.05TiO3, Pb0.8Ba0.2ZrO3 and polymeric ferroelectrics scales proportionally with V cr-2.2, owing to the one-dimensional structural constraint on lattice-scale depolarization dynamics; whereas ∆Smax in relaxor and normal ferroelectrics scales as ∆Smax ~ V cr-0.37, which tallies with a dipolar interaction exponent of 2/3 in EC materials and the well-proven fractional dimensionality of 2.5 for ferroelectric domain walls

Improving abdomen tumor low-dose CT images using a fast dictionary learning based processing.

International audienceIn abdomen computed tomography (CT), repeated radiation exposures are often inevitable for cancer patients who receive surgery or radiotherapy guided by CT images. Low-dose scans should thus be considered in order to avoid the harm of accumulative x-ray radiation. This work is aimed at improving abdomen tumor CT images from low-dose scans by using a fast dictionary learning (DL) based processing. Stemming from sparse representation theory, the proposed patch-based DL approach allows effective suppression of both mottled noise and streak artifacts. The experiments carried out on clinical data show that the proposed method brings encouraging improvements in abdomen low-dose CT images with tumors

Activation of the maternal immune system alters cerebellar development in the offspring

A common pathological finding in autism is a localized deficit in Purkinje cells (PCs). Cerebellar abnormalities have also been reported in schizophrenia. Using a mouse model that exploits a known risk factor for these disorders, maternal infection, we asked if the offspring of pregnant mice given a mid-gestation respiratory infection have cerebellar pathology resembling that seen in these disorders. We also tested the effects of maternal immune activation in the absence of virus by injection of the synthetic dsRNA, poly(I:C). We infected pregnant mice with influenza on embryonic day 9.5 (E9.5), or injected poly(I:C) i.p. on E12.5, and assessed the linear density of PCs in the cerebellum of adult or postnatal day 11 (P11) offspring. To study granule cell migration, we also injected BrdU on P11. Adult offspring of influenza- or poly(I:C)-exposed mice display a localized deficit in PCs in lobule VII of the cerebellum, as do P11 offspring. Coincident with this are heterotopic PCs, as well as delayed migration of granule cells in lobules VI and VII. The cerebellar pathology observed in the offspring of influenza- or poly(I:C)-exposed mice is strikingly similar to that observed in autism. The poly(I:C) findings indicate that deficits are likely caused by the activation of the maternal immune system. Finally, our data suggest that cerebellar abnormalities occur during embryonic development, and may be an early deficit in autism and schizophrenia