5,159 research outputs found
Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentation
Domain adaptation is an important task to enable learning when labels are
scarce. While most works focus only on the image modality, there are many
important multi-modal datasets. In order to leverage multi-modality for domain
adaptation, we propose cross-modal learning, where we enforce consistency
between the predictions of two modalities via mutual mimicking. We constrain
our network to make correct predictions on labeled data and consistent
predictions across modalities on unlabeled target-domain data. Experiments in
unsupervised and semi-supervised domain adaptation settings prove the
effectiveness of this novel domain adaptation strategy. Specifically, we
evaluate on the task of 3D semantic segmentation using the image and point
cloud modality. We leverage recent autonomous driving datasets to produce a
wide variety of domain adaptation scenarios including changes in scene layout,
lighting, sensor setup and weather, as well as the synthetic-to-real setup. Our
method significantly improves over previous uni-modal adaptation baselines on
all adaption scenarios. Code will be made available.Comment: arXiv admin note: text overlap with arXiv:1911.1267
Dynamic Adaptation on Non-Stationary Visual Domains
Domain adaptation aims to learn models on a supervised source domain that
perform well on an unsupervised target. Prior work has examined domain
adaptation in the context of stationary domain shifts, i.e. static data sets.
However, with large-scale or dynamic data sources, data from a defined domain
is not usually available all at once. For instance, in a streaming data
scenario, dataset statistics effectively become a function of time. We
introduce a framework for adaptation over non-stationary distribution shifts
applicable to large-scale and streaming data scenarios. The model is adapted
sequentially over incoming unsupervised streaming data batches. This enables
improvements over several batches without the need for any additionally
annotated data. To demonstrate the effectiveness of our proposed framework, we
modify associative domain adaptation to work well on source and target data
batches with unequal class distributions. We apply our method to several
adaptation benchmark datasets for classification and show improved classifier
accuracy not only for the currently adapted batch, but also when applied on
future stream batches. Furthermore, we show the applicability of our
associative learning modifications to semantic segmentation, where we achieve
competitive results
Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation
We present an approach for encoding visual task relationships to improve
model performance in an Unsupervised Domain Adaptation (UDA) setting. Semantic
segmentation and monocular depth estimation are shown to be complementary
tasks; in a multi-task learning setting, a proper encoding of their
relationships can further improve performance on both tasks. Motivated by this
observation, we propose a novel Cross-Task Relation Layer (CTRL), which encodes
task dependencies between the semantic and depth predictions. To capture the
cross-task relationships, we propose a neural network architecture that
contains task-specific and cross-task refinement heads. Furthermore, we propose
an Iterative Self-Learning (ISL) training scheme, which exploits semantic
pseudo-labels to provide extra supervision on the target domain. We
experimentally observe improvements in both tasks' performance because the
complementary information present in these tasks is better captured.
Specifically, we show that: (1) our approach improves performance on all tasks
when they are complementary and mutually dependent; (2) the CTRL helps to
improve both semantic segmentation and depth estimation tasks performance in
the challenging UDA setting; (3) the proposed ISL training scheme further
improves the semantic segmentation performance. The implementation is available
at https://github.com/susaha/ctrl-uda.Comment: Accepted at CVPR 2021; updated results according to the released
source cod
- …