7,083 research outputs found
Wasserstein Distance Guided Representation Learning for Domain Adaptation
Domain adaptation aims at generalizing a high-performance learner on a target
domain via utilizing the knowledge distilled from a source domain which has a
different but related data distribution. One solution to domain adaptation is
to learn domain invariant feature representations while the learned
representations should also be discriminative in prediction. To learn such
representations, domain adaptation frameworks usually include a domain
invariant representation learning approach to measure and reduce the domain
discrepancy, as well as a discriminator for classification. Inspired by
Wasserstein GAN, in this paper we propose a novel approach to learn domain
invariant feature representations, namely Wasserstein Distance Guided
Representation Learning (WDGRL). WDGRL utilizes a neural network, denoted by
the domain critic, to estimate empirical Wasserstein distance between the
source and target samples and optimizes the feature extractor network to
minimize the estimated Wasserstein distance in an adversarial manner. The
theoretical advantages of Wasserstein distance for domain adaptation lie in its
gradient property and promising generalization bound. Empirical studies on
common sentiment and image classification adaptation datasets demonstrate that
our proposed WDGRL outperforms the state-of-the-art domain invariant
representation learning approaches.Comment: The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI
2018
Confounder Balancing in Adversarial Domain Adaptation for Pre-Trained Large Models Fine-Tuning
The excellent generalization, contextual learning, and emergence abilities in
the pre-trained large models (PLMs) handle specific tasks without direct
training data, making them the better foundation models in the adversarial
domain adaptation (ADA) methods to transfer knowledge learned from the source
domain to target domains. However, existing ADA methods fail to account for the
confounder properly, which is the root cause of the source data distribution
that differs from the target domains. This study proposes an adversarial domain
adaptation with confounder balancing for PLMs fine-tuning (ADA-CBF). The
ADA-CBF includes a PLM as the foundation model for a feature extractor, a
domain classifier and a confounder classifier, and they are jointly trained
with an adversarial loss. This loss is designed to improve the domain-invariant
representation learning by diluting the discrimination in the domain
classifier. At the same time, the adversarial loss also balances the confounder
distribution among source and unmeasured domains in training. Compared to
existing ADA methods, ADA-CBF can correctly identify confounders in
domain-invariant features, thereby eliminating the confounder biases in the
extracted features from PLMs. The confounder classifier in ADA-CBF is designed
as a plug-and-play and can be applied in the confounder measurable,
unmeasurable, or partially measurable environments. Empirical results on
natural language processing and computer vision downstream tasks show that
ADA-CBF outperforms the newest GPT-4, LLaMA2, ViT and ADA methods
- …