2 research outputs found
Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval
Sketch-based image retrieval (SBIR) is a challenging task due to the large
cross-domain gap between sketches and natural images. How to align abstract
sketches and natural images into a common high-level semantic space remains a
key problem in SBIR. In this paper, we propose a novel semi-heterogeneous
three-way joint embedding network (Semi3-Net), which integrates three branches
(a sketch branch, a natural image branch, and an edgemap branch) to learn more
discriminative cross-domain feature representations for the SBIR task. The key
insight lies with how we cultivate the mutual and subtle relationships amongst
the sketches, natural images, and edgemaps. A semi-heterogeneous feature
mapping is designed to extract bottom features from each domain, where the
sketch and edgemap branches are shared while the natural image branch is
heterogeneous to the other branches. In addition, a joint semantic embedding is
introduced to embed the features from different domains into a common
high-level semantic space, where all of the three branches are shared. To
further capture informative features common to both natural images and the
corresponding edgemaps, a co-attention model is introduced to conduct common
channel-wise feature recalibration between different domains. A hybrid-loss
mechanism is designed to align the three branches, where an alignment loss and
a sketch-edgemap contrastive loss are presented to encourage the network to
learn invariant cross-domain representations. Experimental results on two
widely used category-level datasets (Sketchy and TU-Berlin Extension)
demonstrate that the proposed method outperforms state-of-the-art methods.Comment: Accepted by IEEE Transactions on Circuits and Systems for Video
Technolog
Deep Learning for Free-Hand Sketch: A Survey and A Toolbox
Free-hand sketches are highly illustrative, and have been widely used by
humans to depict objects or stories from ancient times to the present. The
recent prevalence of touchscreen devices has made sketch creation a much easier
task than ever and consequently made sketch-oriented applications increasingly
popular. The progress of deep learning has immensely benefited free-hand sketch
research and applications. This paper presents a comprehensive survey of the
deep learning techniques oriented at free-hand sketch data, and the
applications that they enable. The main contents of this survey include: (i) A
discussion of the intrinsic traits and unique challenges of free-hand sketch,
to highlight the essential differences between sketch data and other data
modalities, e.g., natural photos. (ii) A review of the developments of
free-hand sketch research in the deep learning era, by surveying existing
datasets, research topics, and the state-of-the-art methods through a detailed
taxonomy and experimental evaluation. (iii) Promotion of future work via a
discussion of bottlenecks, open problems, and potential research directions for
the community. Finally, to support future sketch research and applications, we
contribute TorchSketch -- the first sketch-oriented open-source deep learning
library, which is built on PyTorch and available at
https://github.com/PengBoXiangShang/torchsketch/