686 research outputs found
Approximated Prompt Tuning for Vision-Language Pre-trained Models
Prompt tuning is a parameter-efficient way to deploy large-scale pre-trained
models to downstream tasks by adding task-specific tokens. In terms of
vision-language pre-trained (VLP) models, prompt tuning often requires a large
number of learnable tokens to bridge the gap between the pre-training and
downstream tasks, which greatly exacerbates the already high computational
overhead. In this paper, we revisit the principle of prompt tuning for
Transformer-based VLP models and reveal that the impact of soft prompt tokens
can be actually approximated via independent information diffusion steps,
thereby avoiding the expensive global attention modeling and reducing the
computational complexity to a large extent. Based on this finding, we propose a
novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer
learning. To validate APT, we apply it to two representative VLP models, namely
ViLT and METER, and conduct extensive experiments on a bunch of downstream
tasks. Meanwhile, the generalization of APT is also validated on CLIP for image
classification. The experimental results not only show the superior performance
gains and computation efficiency of APT against the conventional prompt tuning
methods, e.g., +6.6% accuracy and -64.62% additional computation overhead on
METER, but also confirm its merits over other parameter-efficient transfer
learning approaches
Towards Efficient Visual Adaption via Structural Re-parameterization
Parameter-efficient transfer learning (PETL) is an emerging research spot
aimed at inexpensively adapting large-scale pre-trained models to downstream
tasks. Recent advances have achieved great success in saving storage costs for
various vision tasks by updating or injecting a small number of parameters
instead of full fine-tuning. However, we notice that most existing PETL methods
still incur non-negligible latency during inference. In this paper, we propose
a parameter-efficient and computationally friendly adapter for giant vision
models, called RepAdapter. Specifically, we prove that the adaption modules,
even with a complex structure, can be seamlessly integrated into most giant
vision models via structural re-parameterization. This property makes
RepAdapter zero-cost during inference. In addition to computation efficiency,
RepAdapter is more effective and lightweight than existing PETL methods due to
its sparse structure and our careful deployment. To validate RepAdapter, we
conduct extensive experiments on 27 benchmark datasets of three vision tasks,
i.e., image and video classifications and semantic segmentation. Experimental
results show the superior performance and efficiency of RepAdapter than the
state-of-the-art PETL methods. For instance, by updating only 0.6% parameters,
we can improve the performance of ViT from 38.8 to 55.1 on Sun397. Its
generalizability is also well validated by a bunch of vision models, i.e., ViT,
CLIP, Swin-Transformer and ConvNeXt. Our source code is released at
https://github.com/luogen1996/RepAdapter
Bayesian Optimized 1-Bit CNNs
Deep convolutional neural networks (DCNNs) have dominated the recent
developments in computer vision through making various record-breaking models.
However, it is still a great challenge to achieve powerful DCNNs in
resource-limited environments, such as on embedded devices and smart phones.
Researchers have realized that 1-bit CNNs can be one feasible solution to
resolve the issue; however, they are baffled by the inferior performance
compared to the full-precision DCNNs. In this paper, we propose a novel
approach, called Bayesian optimized 1-bit CNNs (denoted as BONNs), taking the
advantage of Bayesian learning, a well-established strategy for hard problems,
to significantly improve the performance of extreme 1-bit CNNs. We incorporate
the prior distributions of full-precision kernels and features into the
Bayesian framework to construct 1-bit CNNs in an end-to-end manner, which have
not been considered in any previous related methods. The Bayesian losses are
achieved with a theoretical support to optimize the network simultaneously in
both continuous and discrete spaces, aggregating different losses jointly to
improve the model capacity. Extensive experiments on the ImageNet and CIFAR
datasets show that BONNs achieve the best classification performance compared
to state-of-the-art 1-bit CNNs
CSPM: A Contrastive Spatiotemporal Preference Model for CTR Prediction in On-Demand Food Delivery Services
Click-through rate (CTR) prediction is a crucial task in the context of an
online on-demand food delivery (OFD) platform for precisely estimating the
probability of a user clicking on food items. Unlike universal e-commerce
platforms such as Taobao and Amazon, user behaviors and interests on the OFD
platform are more location and time-sensitive due to limited delivery ranges
and regional commodity supplies. However, existing CTR prediction algorithms in
OFD scenarios concentrate on capturing interest from historical behavior
sequences, which fails to effectively model the complex spatiotemporal
information within features, leading to poor performance. To address this
challenge, this paper introduces the Contrastive Sres under different search
states using three modules: contrastive spatiotemporal representation learning
(CSRL), spatiotemporal preference extractor (StPE), and spatiotemporal
information filter (StIF). CSRL utilizes a contrastive learning framework to
generate a spatiotemporal activation representation (SAR) for the search
action. StPE employs SAR to activate users' diverse preferences related to
location and time from the historical behavior sequence field, using a
multi-head attention mechanism. StIF incorporates SAR into a gating network to
automatically capture important features with latent spatiotemporal effects.
Extensive experiments conducted on two large-scale industrial datasets
demonstrate the state-of-the-art performance of CSPM. Notably, CSPM has been
successfully deployed in Alibaba's online OFD platform Ele.me, resulting in a
significant 0.88% lift in CTR, which has substantial business implications
X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance
Text-driven 3D stylization is a complex and crucial task in the fields of
computer vision (CV) and computer graphics (CG), aimed at transforming a bare
mesh to fit a target text. Prior methods adopt text-independent multilayer
perceptrons (MLPs) to predict the attributes of the target mesh with the
supervision of CLIP loss. However, such text-independent architecture lacks
textual guidance during predicting attributes, thus leading to unsatisfactory
stylization and slow convergence. To address these limitations, we present
X-Mesh, an innovative text-driven 3D stylization framework that incorporates a
novel Text-guided Dynamic Attention Module (TDAM). The TDAM dynamically
integrates the guidance of the target text by utilizing text-relevant spatial
and channel-wise attentions during vertex feature extraction, resulting in more
accurate attribute prediction and faster convergence speed. Furthermore,
existing works lack standard benchmarks and automated metrics for evaluation,
often relying on subjective and non-reproducible user studies to assess the
quality of stylized 3D assets. To overcome this limitation, we introduce a new
standard text-mesh benchmark, namely MIT-30, and two automated metrics, which
will enable future research to achieve fair and objective comparisons. Our
extensive qualitative and quantitative experiments demonstrate that X-Mesh
outperforms previous state-of-the-art methods.Comment: Technical repor
- …