11 research outputs found
Unsupervised Model Adaptation for Continual Semantic Segmentation
We develop an algorithm for adapting a semantic segmentation model that is
trained using a labeled source domain to generalize well in an unlabeled target
domain. A similar problem has been studied extensively in the unsupervised
domain adaptation (UDA) literature, but existing UDA algorithms require access
to both the source domain labeled data and the target domain unlabeled data for
training a domain agnostic semantic segmentation model. Relaxing this
constraint enables a user to adapt pretrained models to generalize in a target
domain, without requiring access to source data. To this end, we learn a
prototypical distribution for the source domain in an intermediate embedding
space. This distribution encodes the abstract knowledge that is learned from
the source domain. We then use this distribution for aligning the target domain
distribution with the source domain distribution in the embedding space. We
provide theoretical analysis and explain conditions under which our algorithm
is effective. Experiments on benchmark adaptation task demonstrate our method
achieves competitive performance even compared with joint UDA approaches.Comment: 12 pages, 5 figure
MLAN: Multi-Level Adversarial Network for Domain Adaptive Semantic Segmentation
Recent progresses in domain adaptive semantic segmentation demonstrate the
effectiveness of adversarial learning (AL) in unsupervised domain adaptation.
However, most adversarial learning based methods align source and target
distributions at a global image level but neglect the inconsistency around
local image regions. This paper presents a novel multi-level adversarial
network (MLAN) that aims to address inter-domain inconsistency at both global
image level and local region level optimally. MLAN has two novel designs,
namely, region-level adversarial learning (RL-AL) and co-regularized
adversarial learning (CR-AL). Specifically, RL-AL models prototypical regional
context-relations explicitly in the feature space of a labelled source domain
and transfers them to an unlabelled target domain via adversarial learning.
CR-AL fuses region-level AL and image-level AL optimally via mutual
regularization. In addition, we design a multi-level consistency map that can
guide domain adaptation in both input space (, image-to-image
translation) and output space (, self-training) effectively. Extensive
experiments show that MLAN outperforms the state-of-the-art with a large margin
consistently across multiple datasets.Comment: Submitted to P
Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers
Detection transformers have recently shown promising object detection results
and attracted increasing attention. However, how to develop effective domain
adaptation techniques to improve its cross-domain performance remains
unexplored and unclear. In this paper, we delve into this topic and empirically
find that direct feature distribution alignment on the CNN backbone only brings
limited improvements, as it does not guarantee domain-invariant sequence
features in the transformer for prediction. To address this issue, we propose a
novel Sequence Feature Alignment (SFA) method that is specially designed for
the adaptation of detection transformers. Technically, SFA consists of a domain
query-based feature alignment (DQFA) module and a token-wise feature alignment
(TDA) module. In DQFA, a novel domain query is used to aggregate and align
global context from the token sequence of both domains. DQFA reduces the domain
discrepancy in global feature representations and object relations when
deploying in the transformer encoder and decoder, respectively. Meanwhile, TDA
aligns token features in the sequence from both domains, which reduces the
domain gaps in local and instance-level feature representations in the
transformer encoder and decoder, respectively. Besides, a novel bipartite
matching consistency loss is proposed to enhance the feature discriminability
for robust object detection. Experiments on three challenging benchmarks show
that SFA outperforms state-of-the-art domain adaptive object detection methods.
Code has been made available at: https://github.com/encounter1997/SFA.Comment: Fix a typo in Eq. 1
KRADA: Known-region-aware Domain Alignment for Open World Semantic Segmentation
In semantic segmentation, we aim to train a pixel-level classifier to assign
category labels to all pixels in an image, where labeled training images and
unlabeled test images are from the same distribution and share the same label
set. However, in an open world, the unlabeled test images probably contain
unknown categories and have different distributions from the labeled images.
Hence, in this paper, we consider a new, more realistic, and more challenging
problem setting where the pixel-level classifier has to be trained with labeled
images and unlabeled open-world images -- we name it open world semantic
segmentation (OSS). In OSS, the trained classifier is expected to identify
unknown-class pixels and classify known-class pixels well. To solve OSS, we
first investigate which distribution that unknown-class pixels obey. Then,
motivated by the goodness-of-fit test, we use statistical measurements to show
how a pixel fits the distribution of an unknown class and select highly-fitted
pixels to form the unknown region in each image. Eventually, we propose an
end-to-end learning framework, known-region-aware domain alignment (KRADA), to
distinguish unknown classes while aligning distributions of known classes in
labeled and unlabeled open-world images. The effectiveness of KRADA has been
verified on two synthetic tasks and one COVID-19 segmentation task
Pixel-level intra-domain adaptation for semantic segmentation
Recent advances in unsupervised domain adaptation have achieved remarkable performance on semantic segmentation tasks. Despite such progress, existing works mainly focus on bridging the inter-domain gaps between the source and target domain, while only few of them noticed the intra-domain gaps within the target data. In this work, we propose a pixel-level intra-domain adaptation approach to reduce the intra-domain gaps within the target data. Compared with image-level methods, ours treats each pixel as an instance, which adapts the segmentation model at a more fine-grained level. Specifically, we first conduct the inter-domain adaptation between the source and target domain; Then, we separate the pixels in target images into the easy and hard subdomains; Finally, we propose a pixel-level adversarial training strategy to adapt a segmentation network from the easy to the hard subdomain. Moreover, we show that the segmentation accuracy can be further improved by incorporating a continuous indexing technique in the adversarial training. Experimental results show the effectiveness of our method against existing state-of-the-art approaches