83 research outputs found
Cross-position Activity Recognition with Stratified Transfer Learning
Human activity recognition aims to recognize the activities of daily living
by utilizing the sensors on different body parts. However, when the labeled
data from a certain body position (i.e. target domain) is missing, how to
leverage the data from other positions (i.e. source domain) to help learn the
activity labels of this position? When there are several source domains
available, it is often difficult to select the most similar source domain to
the target domain. With the selected source domain, we need to perform accurate
knowledge transfer between domains. Existing methods only learn the global
distance between domains while ignoring the local property. In this paper, we
propose a \textit{Stratified Transfer Learning} (STL) framework to perform both
source domain selection and knowledge transfer. STL is based on our proposed
\textit{Stratified} distance to capture the local property of domains. STL
consists of two components: Stratified Domain Selection (STL-SDS) can select
the most similar source domain to the target domain; Stratified Activity
Transfer (STL-SAT) is able to perform accurate knowledge transfer. Extensive
experiments on three public activity recognition datasets demonstrate the
superiority of STL. Furthermore, we extensively investigate the performance of
transfer learning across different degrees of similarities and activity levels
between domains. We also discuss the potential applications of STL in other
fields of pervasive computing for future research.Comment: Submit to Pervasive and Mobile Computing as an extension to PerCom 18
paper; First revision. arXiv admin note: substantial text overlap with
arXiv:1801.0082
Joint Air Quality and Weather Prediction Based on Multi-Adversarial Spatiotemporal Networks
Accurate and timely air quality and weather predictions are of great
importance to urban governance and human livelihood. Though many efforts have
been made for air quality or weather prediction, most of them simply employ one
another as feature input, which ignores the inner-connection between two
predictive tasks. On the one hand, the accurate prediction of one task can help
improve another task's performance. On the other hand, geospatially distributed
air quality and weather monitoring stations provide additional hints for
city-wide spatiotemporal dependency modeling. Inspired by the above two
insights, in this paper, we propose the Multi-adversarial spatiotemporal
recurrent Graph Neural Networks (MasterGNN) for joint air quality and weather
predictions. Specifically, we first propose a heterogeneous recurrent graph
neural network to model the spatiotemporal autocorrelation among air quality
and weather monitoring stations. Then, we develop a multi-adversarial graph
learning framework to against observation noise propagation introduced by
spatiotemporal modeling. Moreover, we present an adaptive training strategy by
formulating multi-adversarial learning as a multi-task learning problem.
Finally, extensive experiments on two real-world datasets show that MasterGNN
achieves the best performance compared with seven baselines on both air quality
and weather prediction tasks.Comment: 9 pages, 6 figure
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Various adaptation methods, such as LoRA, prompts, and adapters, have been
proposed to enhance the performance of pre-trained vision-language models in
specific domains. The robustness of these adaptation methods against
distribution shifts have not been studied. In this study, we assess the
robustness of 11 widely-used adaptation methods across 4 vision-language
datasets under multimodal corruptions. Concretely, we introduce 7 benchmark
datasets, including 96 visual and 87 textual corruptions, to investigate the
robustness of different adaptation methods, the impact of available adaptation
examples, and the influence of trainable parameter size during adaptation. Our
analysis reveals that: 1) Adaptation methods are more sensitive to text
corruptions than visual corruptions. 2) Full fine-tuning does not consistently
provide the highest robustness; instead, adapters can achieve better robustness
with comparable clean performance. 3) Contrary to expectations, our findings
indicate that increasing the number of adaptation data and parameters does not
guarantee enhanced robustness; instead it results in even lower robustness. We
hope this study could benefit future research in the development of robust
multimodal adaptation methods. The benchmark, code, and dataset used in this
study can be accessed at \url{https://adarobustness.github.io}
Boosting Cross-Domain Speech Recognition with Self-Supervision
The cross-domain performance of automatic speech recognition (ASR) could be
severely hampered due to the mismatch between training and testing
distributions. Since the target domain usually lacks labeled data, and domain
shifts exist at acoustic and linguistic levels, it is challenging to perform
unsupervised domain adaptation (UDA) for ASR. Previous work has shown that
self-supervised learning (SSL) or pseudo-labeling (PL) is effective in UDA by
exploiting the self-supervisions of unlabeled data. However, these
self-supervisions also face performance degradation in mismatched domain
distributions, which previous work fails to address. This work presents a
systematic UDA framework to fully utilize the unlabeled data with
self-supervision in the pre-training and fine-tuning paradigm. On the one hand,
we apply continued pre-training and data replay techniques to mitigate the
domain mismatch of the SSL pre-trained model. On the other hand, we propose a
domain-adaptive fine-tuning approach based on the PL technique with three
unique modifications: Firstly, we design a dual-branch PL method to decrease
the sensitivity to the erroneous pseudo-labels; Secondly, we devise an
uncertainty-aware confidence filtering strategy to improve pseudo-label
correctness; Thirdly, we introduce a two-step PL approach to incorporate target
domain linguistic knowledge, thus generating more accurate target domain
pseudo-labels. Experimental results on various cross-domain scenarios
demonstrate that the proposed approach effectively boosts the cross-domain
performance and significantly outperforms previous approaches.Comment: Accepted by IEEE/ACM Transactions on Audio, Speech and Language
Processing (TASLP), 202
OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization
Vision-language pre-training (VLP) models demonstrate impressive abilities in
processing both images and text. However, they are vulnerable to multi-modal
adversarial examples (AEs). Investigating the generation of
high-transferability adversarial examples is crucial for uncovering VLP models'
vulnerabilities in practical scenarios. Recent works have indicated that
leveraging data augmentation and image-text modal interactions can enhance the
transferability of adversarial examples for VLP models significantly. However,
they do not consider the optimal alignment problem between dataaugmented
image-text pairs. This oversight leads to adversarial examples that are overly
tailored to the source model, thus limiting improvements in transferability. In
our research, we first explore the interplay between image sets produced
through data augmentation and their corresponding text sets. We find that
augmented image samples can align optimally with certain texts while exhibiting
less relevance to others. Motivated by this, we propose an Optimal
Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method
formulates the features of image and text sets as two distinct distributions
and employs optimal transport theory to determine the most efficient mapping
between them. This optimal mapping informs our generation of adversarial
examples to effectively counteract the overfitting issues. Extensive
experiments across various network architectures and datasets in image-text
matching tasks reveal that our OT-Attack outperforms existing state-of-the-art
methods in terms of adversarial transferability
Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR
Self-supervised pre-training could effectively improve the performance of
low-resource automatic speech recognition (ASR). However, existing
self-supervised pre-training are task-agnostic, i.e., could be applied to
various downstream tasks. Although it enlarges the scope of its application,
the capacity of the pre-trained model is not fully utilized for the ASR task,
and the learned representations may not be optimal for ASR. In this work, in
order to build a better pre-trained model for low-resource ASR, we propose a
pre-training approach called wav2vec-S, where we use task-specific
semi-supervised pre-training to refine the self-supervised pre-trained model
for the ASR task thus more effectively utilize the capacity of the pre-trained
model to generate task-specific representations for ASR. Experiments show that
compared to wav2vec 2.0, wav2vec-S only requires a marginal increment of
pre-training time but could significantly improve ASR performance on in-domain,
cross-domain and cross-lingual datasets. Average relative WER reductions are
24.5% and 6.6% for 1h and 10h fine-tuning, respectively. Furthermore, we show
that semi-supervised pre-training could close the representation gap between
the self-supervised pre-trained model and the corresponding fine-tuned model
through canonical correlation analysis.Comment: Accepted by Interspeech 202
Understanding and Improving In-Context Learning on Vision-language Models
Recently, in-context learning (ICL) on large language models (LLMs) has
received great attention, and this technique can also be applied to
vision-language models (VLMs) built upon LLMs. These VLMs can respond to
queries by conditioning responses on a series of multimodal demonstrations,
which comprise images, queries, and answers. Though ICL has been extensively
studied on LLMs, its research on VLMs remains limited. The inclusion of
additional visual information in the demonstrations motivates the following
research questions: which of the two modalities in the demonstration is more
significant? How can we select effective multimodal demonstrations to enhance
ICL performance? This study investigates the significance of both visual and
language information. Our findings indicate that ICL in VLMs is predominantly
driven by the textual information in the demonstrations whereas the visual
information in the demonstrations barely affects the ICL performance.
Subsequently, we provide an understanding of the findings by analyzing the
model information flow and comparing model inner states given different ICL
settings. Motivated by our analysis, we propose a simple yet effective
approach, termed Mixed Modality In-Context Example Selection (MMICES), which
considers both visual and language modalities when selecting demonstrations and
shows better ICL performance. Extensive experiments are conducted to support
our findings, understanding, and improvement of the ICL performance of VLMs.Comment: 8 pages, 10 figure
- …