167 research outputs found
RUSH: Robust Contrastive Learning via Randomized Smoothing
Recently, adversarial training has been incorporated in self-supervised
contrastive pre-training to augment label efficiency with exciting adversarial
robustness. However, the robustness came at a cost of expensive adversarial
training. In this paper, we show a surprising fact that contrastive
pre-training has an interesting yet implicit connection with robustness, and
such natural robustness in the pre trained representation enables us to design
a powerful robust algorithm against adversarial attacks, RUSH, that combines
the standard contrastive pre-training and randomized smoothing. It boosts both
standard accuracy and robust accuracy, and significantly reduces training costs
as compared with adversarial training. We use extensive empirical studies to
show that the proposed RUSH outperforms robust classifiers from adversarial
training, by a significant margin on common benchmarks (CIFAR-10, CIFAR-100,
and STL-10) under first-order attacks. In particular, under
-norm perturbations of size 8/255 PGD attack on CIFAR-10, our
model using ResNet-18 as backbone reached 77.8% robust accuracy and 87.9%
standard accuracy. Our work has an improvement of over 15% in robust accuracy
and a slight improvement in standard accuracy, compared to the
state-of-the-arts.Comment: 12 pages, 2 figure
Study on the Indicators Evaluating Innovation Abilities of High-end Equipment Manufacturing Industry in Sichuan Province
Innovation is the only way to enhance the competitiveness of high-end equipment manufacturing industry in Sichuan province. However, due to the lack of relevant evaluation standards, many high-end equipment
manufacturing enterprises cannot evaluate their own innovation abilities effectively. Using the expert-interview
method, this paper constructs an evaluation index system composed of three primary indicators, six secondary
indicators and nine tertiary indicators. This paper determines the weights for each indicator through order-relationship analysis, based on which suggestions are put forward to improve the innovation ability for such enterprises
FCG-ASpredictor: An Approach for the Prediction of Average Speed of Road Segments with Floating Car GPS Data
The average speed (AS) of a road segment is an important factor for predicting traffic congestion, because the accuracy of AS can directly affect the implementation of traffic management. The traffic environment, spatiotemporal information, and the dynamic interaction between these two factors impact the predictive accuracy of AS in the existing literature, and floating car data comprehensively reflect the operation of urban road vehicles. In this paper, we proposed a novel road segment AS predictive model, which is based on floating car data. First, the impact of historical AS, weather, and date attributes on AS prediction has been analyzed. Then, through spatiotemporal correlations calculation based on the data from Global Positioning System (GPS), the predictive method utilizes the recursive least squares method to fuse the historical AS with other factors (such as weather, date attributes, etc.) and adopts an extended Kalman filter algorithm to accurately predict the AS of the target segment. Finally, we applied our approach on the traffic congestion prediction on four road segments in Chengdu, China. The results showed that the proposed predictive model is highly feasible and accurate.
Document type: Articl
What is valued most by patients with type 2 diabetes mellitus when selecting second-line antihyperglycemic medications in China
Objective: To estimate patient preferences for second-line antihyperglycemic medications in China. Methods: A face to face survey with the best-worst scaling (BWS) choices was administered in patients with diagnosed type 2 diabetes mellitus (T2DM). Study participants were asked to indicate which attribute they valued most and which attribute they valued least in 11 choice sets, each of which consisted of five alternatives out of 11 antihyperglycemic medication-specific attributes (treatment efficacy, weight change, hypoglycemic events, gastrointestinal side effects, cardiovascular health, urinary tract infection and genital infection side effects, edema, mode of administration, bone fracture, dosing frequency and out-of-pocket cost). A counting approach, a conditional logit model, and K-means clustering were used to estimate the relative importance of items and preference heterogeneity. Results: A total of 362 participants were included with a mean age of 63.6 (standard deviation: 11.8) years. There were 56.4% of participants were women, and 56.3% being diagnosed with diabetes for at least 5 years. Efficacy, cardiovascular health and hypoglycemic events were valued most, while dosing frequency, mode of administration and bone fracture were valued least. The K-means clustering further showed preference heterogeneity in out-of-pocket cost across the participants. Conclusion: Our study suggests that treatment efficacy, cardiovascular health and hypoglycemic events are valued most by Chinese patients with T2DM when selecting second-line antihyperglycemic medications. The study improves the understanding of patients’ preferences for second-line antihyperglycemic medications in China
NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers
The complicated architecture and high training cost of vision transformers
urge the exploration of post-training quantization. However, the heavy-tailed
distribution of vision transformer activations hinders the effectiveness of
previous post-training quantization methods, even with advanced quantizer
designs. Instead of tuning the quantizer to better fit the complicated
activation distribution, this paper proposes NoisyQuant, a quantizer-agnostic
enhancement for the post-training activation quantization performance of vision
transformers. We make a surprising theoretical discovery that for a given
quantizer, adding a fixed Uniform noisy bias to the values being quantized can
significantly reduce the quantization error under provable conditions. Building
on the theoretical insight, NoisyQuant achieves the first success on actively
altering the heavy-tailed activation distribution with additive noisy bias to
fit a given quantizer. Extensive experiments show NoisyQuant largely improves
the post-training quantization performance of vision transformer with minimal
computation overhead. For instance, on linear uniform 6-bit activation
quantization, NoisyQuant improves SOTA top-1 accuracy on ImageNet by up to
1.7%, 1.1% and 0.5% for ViT, DeiT, and Swin Transformer respectively, achieving
on-par or even higher performance than previous nonlinear, mixed-precision
quantization.Comment: Accepted to CVPR202
Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning
Large Language Models (LLMs) have demonstrated significant potential in
performing multiple tasks in multimedia applications, ranging from content
generation to interactive entertainment, and artistic creation. However, the
diversity of downstream tasks in multitask scenarios presents substantial
adaptation challenges for LLMs. While traditional methods often succumb to
knowledge confusion on their monolithic dense models, Mixture-of-Experts (MoE)
has been emerged as a promising solution with its sparse architecture for
effective task decoupling. Inspired by the principles of human cognitive
neuroscience, we design a novel framework \texttt{Intuition-MoR1E} that
leverages the inherent semantic clustering of instances to mimic the human
brain to deal with multitask, offering implicit guidance to router for
optimized feature allocation. Moreover, we introduce cutting-edge Rank-1
Experts formulation designed to manage a spectrum of intuitions, demonstrating
enhanced parameter efficiency and effectiveness in multitask LLM finetuning.
Extensive experiments demonstrate that Intuition-MoR1E achieves superior
efficiency and 2.15\% overall accuracy improvement across 14 public datasets
against other state-of-the-art baselines.Comment: 13 pages, 5 figure
- …