57,570 research outputs found
Conditional Support Alignment for Domain Adaptation with Label Shift
Unsupervised domain adaptation (UDA) refers to a domain adaptation framework
in which a learning model is trained based on the labeled samples on the source
domain and unlabelled ones in the target domain. The dominant existing methods
in the field that rely on the classical covariate shift assumption to learn
domain-invariant feature representation have yielded suboptimal performance
under the label distribution shift between source and target domains. In this
paper, we propose a novel conditional adversarial support alignment (CASA)
whose aim is to minimize the conditional symmetric support divergence between
the source's and target domain's feature representation distributions, aiming
at a more helpful representation for the classification task. We also introduce
a novel theoretical target risk bound, which justifies the merits of aligning
the supports of conditional feature distributions compared to the existing
marginal support alignment approach in the UDA settings. We then provide a
complete training process for learning in which the objective optimization
functions are precisely based on the proposed target risk bound. Our empirical
results demonstrate that CASA outperforms other state-of-the-art methods on
different UDA benchmark tasks under label shift conditions
RLSbench: Domain Adaptation Under Relaxed Label Shift
Despite the emergence of principled methods for domain adaptation under label
shift, their sensitivity to shifts in class conditional distributions is
precariously under explored. Meanwhile, popular deep domain adaptation
heuristics tend to falter when faced with label proportions shifts. While
several papers modify these heuristics in attempts to handle label proportions
shifts, inconsistencies in evaluation standards, datasets, and baselines make
it difficult to gauge the current best practices. In this paper, we introduce
RLSbench, a large-scale benchmark for relaxed label shift, consisting of 500
distribution shift pairs spanning vision, tabular, and language modalities,
with varying label proportions. Unlike existing benchmarks, which primarily
focus on shifts in class-conditional , our benchmark also focuses on
label marginal shifts. First, we assess 13 popular domain adaptation methods,
demonstrating more widespread failures under label proportion shifts than were
previously known. Next, we develop an effective two-step meta-algorithm that is
compatible with most domain adaptation heuristics: (i) pseudo-balance the data
at each epoch; and (ii) adjust the final classifier with target label
distribution estimate. The meta-algorithm improves existing domain adaptation
heuristics under large label proportion shifts, often by 2--10\% accuracy
points, while conferring minimal effect (0.5\%) when label proportions do
not shift. We hope that these findings and the availability of RLSbench will
encourage researchers to rigorously evaluate proposed methods in relaxed label
shift settings. Code is publicly available at
https://github.com/acmi-lab/RLSbench.Comment: Accepted at ICML 2023. Paper website:
https://sites.google.com/view/rlsbench
Deep transfer learning for partial differential equations under conditional shift with DeepONet
Traditional machine learning algorithms are designed to learn in isolation,
i.e. address single tasks. The core idea of transfer learning (TL) is that
knowledge gained in learning to perform one task (source) can be leveraged to
improve learning performance in a related, but different, task (target). TL
leverages and transfers previously acquired knowledge to address the expense of
data acquisition and labeling, potential computational power limitations, and
the dataset distribution mismatches. Although significant progress has been
made in the fields of image processing, speech recognition, and natural
language processing (for classification and regression) for TL, little work has
been done in the field of scientific machine learning for functional regression
and uncertainty quantification in partial differential equations. In this work,
we propose a novel TL framework for task-specific learning under conditional
shift with a deep operator network (DeepONet). Inspired by the conditional
embedding operator theory, we measure the statistical distance between the
source domain and the target feature domain by embedding conditional
distributions onto a reproducing kernel Hilbert space. Task-specific operator
learning is accomplished by fine-tuning task-specific layers of the target
DeepONet using a hybrid loss function that allows for the matching of
individual target samples while also preserving the global properties of the
conditional distribution of target data. We demonstrate the advantages of our
approach for various TL scenarios involving nonlinear PDEs under conditional
shift. Our results include geometry domain adaptation and show that the
proposed TL framework enables fast and efficient multi-task operator learning,
despite significant differences between the source and target domains.Comment: 19 pages, 3 figure
Domain adaptation with conditional transferable components
© 2016 by the author(s). Domain adaptation arises in supervised learning when the training (source domain) and test (target domain) data have different distribution- s. Let X and Y denote the features and target, respectively, previous work on domain adaptation mainly considers the covariate shift situation where the distribution of the features P(X) changes across domains while the conditional distribution P(Y\X) stays the same. To reduce domain discrepancy, recent methods try to find invariant components T(X) that have similar P(T(X)) on different domains by explicitly minimizing a distribution discrepancy measure. However, it is not clear if P(Y\T(X)) in different domains is also similar when P(Y/X)changes. Furthermore, transferable components do not necessarily have to be invariant. If the change in some components is identifiable, we can make use of such components for prediction in the target domain. In this paper, we focus on the case where P{X ,Y) and P(Y') both change in a causal system in which Y is the cause for X. Under appropriate assumptions, we aim to extract conditional transferable components whose conditional distribution P(T{X)\Y) is invariant after proper location-scale (LS) transformations, and identify how P{Y) changes between domains simultaneously. We provide theoretical analysis and empirical evaluation on both synthetic and real-world data to show the effectiveness of our method
Match and Reweight Strategy for Generalized Target Shift
We address the problem of unsupervised domain adaptation under the setting of generalized target shift (both class-conditional and label shifts occur). We show that in that setting, for good generalization, it is necessary to learn with similar source and target label distributions and to match the class-conditional probabilities. For this purpose, we propose an estimation of target label proportion by blending mixture estimation and optimal transport. This estimation comes with theoretical guarantees of correctness. Based on the estimation, we learn a model by minimizing a importance weighted loss and a Wasserstein distance between weighted marginals. We prove that this minimization allows to match class-conditionals given mild assumptions on their geometry. Our experimental results show that our method performs better on average than competitors accross a range domain adaptation problems including digits,VisDA and Office
Identifying Latent Causal Content for Multi-Source Domain Adaptation
Multi-source domain adaptation (MSDA) learns to predict the labels in target
domain data, under the setting that data from multiple source domains are
labelled and data from the target domain are unlabelled. Most methods for this
task focus on learning invariant representations across domains. However, their
success relies heavily on the assumption that the label distribution remains
consistent across domains, which may not hold in general real-world problems.
In this paper, we propose a new and more flexible assumption, termed
\textit{latent covariate shift}, where a latent content variable
and a latent style variable are introduced in the generative
process, with the marginal distribution of changing across
domains and the conditional distribution of the label given
remaining invariant across domains. We show that although (completely)
identifying the proposed latent causal model is challenging, the latent content
variable can be identified up to scaling by using its dependence with labels
from source domains, together with the identifiability conditions of nonlinear
ICA. This motivates us to propose a novel method for MSDA, which learns the
invariant label distribution conditional on the latent content variable,
instead of learning invariant representations. Empirical evaluation on
simulation and real data demonstrates the effectiveness of the proposed method
A review of domain adaptation without target labels
Domain adaptation has become a prominent problem setting in machine learning
and related fields. This review asks the question: how can a classifier learn
from a source domain and generalize to a target domain? We present a
categorization of approaches, divided into, what we refer to as, sample-based,
feature-based and inference-based methods. Sample-based methods focus on
weighting individual observations during training based on their importance to
the target domain. Feature-based methods revolve around on mapping, projecting
and representing features such that a source classifier performs well on the
target domain and inference-based methods incorporate adaptation into the
parameter estimation procedure, for instance through constraints on the
optimization procedure. Additionally, we review a number of conditions that
allow for formulating bounds on the cross-domain generalization error. Our
categorization highlights recurring ideas and raises questions important to
further research.Comment: 20 pages, 5 figure
- …