47 research outputs found
Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
Representation learning has been evolving from traditional supervised
training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous
works have demonstrated their pros and cons in specific scenarios, i.e., CL and
supervised pre-training excel at capturing longer-range global patterns and
enabling better feature discrimination, while MIM can introduce more local and
diverse attention across all transformer layers. In this paper, we explore how
to obtain a model that combines their strengths. We start by examining previous
feature distillation and mask feature reconstruction methods and identify their
limitations. We find that their increasing diversity mainly derives from the
asymmetric designs, but these designs may in turn compromise the discrimination
ability. In order to better obtain both discrimination and diversity, we
propose a simple but effective Hybrid Distillation strategy, which utilizes
both the supervised/CL teacher and the MIM teacher to jointly guide the student
model. Hybrid Distill imitates the token relations of the MIM teacher to
alleviate attention collapse, as well as distills the feature maps of the
supervised/CL teacher to enable discrimination. Furthermore, a progressive
redundant token masking strategy is also utilized to reduce the distilling
costs and avoid falling into local optima. Experiment results prove that Hybrid
Distill can achieve superior performance on different benchmarks
AiluRus: A Scalable ViT Framework for Dense Prediction
Vision transformers (ViTs) have emerged as a prevalent architecture for
vision tasks owing to their impressive performance. However, when it comes to
handling long token sequences, especially in dense prediction tasks that
require high-resolution input, the complexity of ViTs increases significantly.
Notably, dense prediction tasks, such as semantic segmentation or object
detection, emphasize more on the contours or shapes of objects, while the
texture inside objects is less informative. Motivated by this observation, we
propose to apply adaptive resolution for different regions in the image
according to their importance. Specifically, at the intermediate layer of the
ViT, we utilize a spatial-aware density-based clustering algorithm to select
representative tokens from the token sequence. Once the representative tokens
are determined, we proceed to merge other tokens into their closest
representative token. Consequently, semantic similar tokens are merged together
to form low-resolution regions, while semantic irrelevant tokens are preserved
independently as high-resolution regions. This strategy effectively reduces the
number of tokens, allowing subsequent layers to handle a reduced token sequence
and achieve acceleration. We evaluate our proposed method on three different
datasets and observe promising performance. For example, the "Segmenter ViT-L"
model can be accelerated by 48% FPS without fine-tuning, while maintaining the
performance. Additionally, our method can be applied to accelerate fine-tuning
as well. Experimental results demonstrate that we can save 52% training time
while accelerating 2.46 times FPS with only a 0.09% performance drop. The code
is available at https://github.com/caddyless/ailurus/tree/main.Comment: Accepted by NeurIPS 202
Corticosteroids for the prevention of bronchopulmonary dysplasia in preterm infants: a network meta-analysis
Objective: To determine the comparative efficacy and safety of corticosteroids in the prevention of bronchopulmonary dysplasia (BPD) in preterm infants. Study design: We systematically searched PubMed, EMBASE and the Cochrane Library. Two reviewers independently selected randomised controlled trials (RCTs) of postnatal corticosteroids in preterm infants. A Bayesian network meta-analysis and subgroup analyses were performed. Results: We included 47 RCTs with 6747 participants. The use of dexamethasone at either high dose or low dose decreased the risk of BPD (OR 0.29, 95% credible interval (CrI) 0.14 to 0.52; OR 0.58, 95% CrI 0.39 to 0.76, respectively). High-dose dexamethasone was more effective than hydrocortisone, beclomethasone and low-dose dexamethasone. Early and long-term dexamethasone at either high dose or low dose decreased the risk of BPD (OR 0.11, 95% CrI 0.02 to 0.4; OR 0.37, 95% CrI 0.16 to 0.67, respectively). There were no statistically significant differences in the risk of cerebral palsy (CP) between different corticosteroids. However, high-dose and long-term dexamethasone ranked lower than placebo and other regimens in terms of CP. Subgroup analyses indicated budesonide was associated with a decreased risk of BPD in extremely preterm and extremely low birthweight infants (OR 0.60, 95% CrI 0.36 to 0.93). Conclusions: Dexamethasone can reduce the risk of BPD in preterm infants. Of the different dexamethasone regimens, aggressive initiation seems beneficial, while a combination of high-dose and long-term use should be avoided because of the possible adverse neurodevelopmental outcome. Dexamethasone and inhaled corticosteroids need to be further evaluated in large-scale RCTs with long-term follow-ups
Exact dipole solitary wave solution in metamaterials with higher-order dispersion
We present an exact dipole solitary wave solution in a mutual modulation form of bright and dark
solitons for a higher-order nonlinear Schrödinger equation with third- and fourth-order dispersion
in metamaterials (MMs) using an ansatz method. Based on the Drude model, the formation
conditions, existence regions and propagation properties are discussed. The results reveal that the
solitary wave may exist in a few parameter regions of MMs, different from those in optical fibres, and
its propagation properties can be controlled by adjusting the frequency of incident waves in each
existence region
Theoretical Analysis of a Surface Plasmonic Waveguide With a Double-Petal-Shaped Air Core
Temperature correction method for dielectric response of high moisture content and aging degree oil impregnated paper based on segmented activation energy
Frequency Domain Spectroscopy (FDS) is widely used to estimate oil–paper insulation. The high moisture and long aging change the FDS characteristics of the insulating oil–paper, so normalization processing is required. Still, the choice of single activation energy needs further research. This paper studies the FDS characteristic of oil–paper insulation samples with different moisture and aging degrees and improves the traditional temperature correction method. The enhanced Arrhenius model uses the segmented activation energy to make the normalized FDS curve coincide better with the target curve, and the error is reduced. In addition, to verify the method’s effectiveness, this paper proposes an iterative correction process. It corrects the tan δ-f curve of the bushing with an aging time of 800 h based on segmented activation energy, and the overall normalization effect is improved