73 research outputs found
Bias Reduction via End-to-End Shift Learning: Application to Citizen Science
Citizen science projects are successful at gathering rich datasets for
various applications. However, the data collected by citizen scientists are
often biased --- in particular, aligned more with the citizens' preferences
than with scientific objectives. We propose the Shift Compensation Network
(SCN), an end-to-end learning scheme which learns the shift from the scientific
objectives to the biased data while compensating for the shift by re-weighting
the training data. Applied to bird observational data from the citizen science
project eBird, we demonstrate how SCN quantifies the data distribution shift
and outperforms supervised learning models that do not address the data bias.
Compared with competing models in the context of covariate shift, we further
demonstrate the advantage of SCN in both its effectiveness and its capability
of handling massive high-dimensional data
Transformer Based Multi-Source Domain Adaptation
In practical machine learning settings, the data on which a model must make
predictions often come from a different distribution than the data it was
trained on. Here, we investigate the problem of unsupervised multi-source
domain adaptation, where a model is trained on labelled data from multiple
source domains and must make predictions on a domain for which no labelled data
has been seen. Prior work with CNNs and RNNs has demonstrated the benefit of
mixture of experts, where the predictions of multiple domain expert classifiers
are combined; as well as domain adversarial training, to induce a domain
agnostic representation space. Inspired by this, we investigate how such
methods can be effectively applied to large pretrained transformer models. We
find that domain adversarial training has an effect on the learned
representations of these models while having little effect on their
performance, suggesting that large transformer-based models are already
relatively robust across domains. Additionally, we show that mixture of experts
leads to significant performance improvements by comparing several variants of
mixing functions, including one novel mixture based on attention. Finally, we
demonstrate that the predictions of large pretrained transformer based domain
experts are highly homogenous, making it challenging to learn effective
functions for mixing their predictions.Comment: 12 pages, 3 figures, 5 table
- …