9 research outputs found
SHREC'16 Track: 3D Sketch-Based 3D Shape Retrieval
Sketch-based 3D shape retrieval has unique representation availability of the queries and vast applications. Therefore, it has received more and more attentions in the research community of content-based 3D object retrieval. However, sketch-based 3D shape retrieval is a challenging research topic due to the semantic gap existing between the inaccurate representation of sketches and accurate representation of 3D models. In order to enrich and advance the study of sketch-based 3D shape retrieval, we initialize the research on 3D sketch-based 3D model retrieval and collect a 3D sketch dataset based on a developed 3D sketching interface which facilitates us to draw 3D sketches in the air while standing in front of a Microsoft Kinect. The objective of this track is to evaluate the performance of different 3D sketch-based 3D model retrieval algorithms using the hand-drawn 3D sketch query dataset and a generic 3D model target dataset. The benchmark contains 300 sketches that are evenly divided into 30 classes, as well as 1 258 3D models that are classified into 90 classes. In this track, nine runs have been submitted by five groups and their retrieval performance has been evaluated using seven commonly used retrieval performance metrics. We wish this benchmark, the comparative evaluation results and the corresponding evaluation code will further promote sketch-based 3D shape retrieval and its applications
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling
Face modeling has been paid much attention in the field of visual computing.
There exist many scenarios, including cartoon characters, avatars for social
media, 3D face caricatures as well as face-related art and design, where
low-cost interactive face modeling is a popular approach especially among
amateur users. In this paper, we propose a deep learning based sketching system
for 3D face and caricature modeling. This system has a labor-efficient
sketching interface, that allows the user to draw freehand imprecise yet
expressive 2D lines representing the contours of facial features. A novel CNN
based deep regression network is designed for inferring 3D face models from 2D
sketches. Our network fuses both CNN and shape based features of the input
sketch, and has two independent branches of fully connected layers generating
independent subsets of coefficients for a bilinear face representation. Our
system also supports gesture based interactions for users to further manipulate
initial face models. Both user studies and numerical results indicate that our
sketching system can help users create face models quickly and effectively. A
significantly expanded face database with diverse identities, expressions and
levels of exaggeration is constructed to promote further research and
evaluation of face modeling techniques.Comment: 12 pages, 16 figures, to appear in SIGGRAPH 201
Explorative Study on Asymmetric Sketch Interactions for Object Retrieval in Virtual Reality
Drawing tools for Virtual Reality (VR) enable users to model 3D designs from within the virtual environment itself. These tools employ sketching and sculpting techniques known from desktop-based interfaces and apply them to hand-based controller interaction. While these techniques allow for mid-air sketching of basic shapes, it remains difficult for users to create detailed and comprehensive 3D models. Our work focuses on supporting the user in designing the virtual environment around them by enhancing sketch-based interfaces with a supporting system for interactive model retrieval. An immersed user can query a database containing detailed 3D models and replace them with the virtual environment through sketching. To understand supportive sketching within a virtual environment, we made an explorative comparison between asymmetric methods of sketch interaction, i.e., 3D mid-air sketching, 2D sketching on a virtual tablet, 2D sketching on a fixed virtual whiteboard, and 2D sketching on a real tablet. Our work shows that different patterns emerge when users interact with 3D sketches rather than 2D sketches to compensate for different results from the retrieval system. In particular, the user adopts strategies when drawing on canvas of different sizes or using a physical device instead of a virtual canvas. While we pose our work as a retrieval problem for 3D models of chairs, our results can be extrapolated to other sketching tasks for virtual environments
Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual Reality
Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities
3D Sketching for Interactive Model Retrieval in Virtual Reality
We describe a novel method for searching 3D model collections using free-form sketches within a virtual environment as queries. As opposed to traditional sketch retrieval, our queries are drawn directly onto an example model. Using immersive virtual reality the user can express their query through a sketch that demonstrates the desired structure, color and texture. Unlike previous sketch-based retrieval methods, users remain immersed within the environment without relying on textual queries or 2D projections which can disconnect the user from the environment. We perform a test using queries over several descriptors, evaluating the precision in order to select the most accurate one. We show how a convolutional neural network (CNN) can create multi-view representations of colored 3D sketches. Using such a descriptor representation, our system is able to rapidly retrieve models and in this way, we provide the user with an interactive method of navigating large object datasets. Through a user study we demonstrate that by using our VR 3D model retrieval system, users can perform search more quickly and intuitively than with a naive linear browsing method. Using our system users can rapidly populate a virtual environment with specific models from a very large database, and thus the technique has the potential to be broadly applicable in immersive editing systems
Deep point-to-subspace metric learning for sketch-based 3D shape retrieval
One key issue in managing a large scale 3D shape dataset is to identify an effective way to retrieve a shape-of-interest. The sketch-based query, which enjoys the flexibility in representing the user’s inten- tion, has received growing interests in recent years due to the popularization of the touchscreen tech- nology. Essentially, the sketch depicts an abstraction of a shape in a certain view while the shape con- tains the full 3D information. Matching between them is a cross-modality retrieval problem, and the state-of-the-art solution is to project the sketch and the 3D shape into a common space with which the cross-modality similarity can be calculated by the feature similarity/distance within. However, for a given query, only part of the viewpoints of the 3D shape is representative. Thus, blindly projecting a 3D shape into a feature vector without considering what is the query will inevitably bring query-unrepresentative information. To handle this issue, in this work we propose a Deep Point-to-Subspace Metric Learning (DPSML) framework to project a sketch into a feature vector and a 3D shape into a subspace spanned by a few selected basis feature vectors. The similarity between them is defined as the distance between the query feature vector and its closest point in the subspace by solving an optimization problem on the fly. Note that, the closest point is query-adaptive and can reflect the viewpoint information that is rep- resentative to the given query. To efficiently learn such a deep model, we formulate it as a classification problem with a special classifier design. To reduce the redundancy of 3D shapes, we also introduce a Representative-View Selection (RVS) module to select the most representative views of a 3D shape. By conducting extensive experiments on various datasets, we show that the proposed method can achieve superior performance over its competitive baseline methods and attain the state-of-the-art performance.Yinjie Lei, Ziqin Zhou, Pingping Zhang, Yulan Guo, Zijun Ma, Lingqiao Li
Cross-Modal Learning for Sketch Visual Understanding.
PhD Theses.As touching devices have rapidly proliferated, sketch has gained much popularity as an
alternative input to text descriptions and speeches. This is due to the fact that sketch
has the advantage of being informative and convenient, which have stimulated sketchrelated
research in areas such as sketch recognition, sketch segmentation, sketch-based
image retrieval, and photo-to-sketch synthesis. Though these eld has been well touched,
existing sketch works still su er from aligning the sketch and photo domains, resulting
in unsatisfactory quality for both ne-grained retrieval and synthesis between sketch and
photo modalities. To address these problems, in this thesis, we proposed a series novel
works on free-hand sketch related tasks and throw out helpful insights to help future
research.
Sketch conveys ne-grained information, making ne-grained sketch-based image retrieval
one of the most important topics for sketch research. The basic solution for this task
is learning to exploit the informativeness of sketches and link it to other modalities.
Apart from the informativeness of sketches, semantic information is also important to
understanding sketch modality and link it with other related modalities. In this thesis,
we indicate that semantic information can e ectively ll the domain gap between sketch
and photo modalities as a bridge. Based on this observation, we proposed an attributeaware
deep framework to exploit attribute information to aid ne-grained SBIR. Text
descriptions are considered as another semantic alternative to attributes, and at the same
time, with the advantage of more
exible and natural, which are exploited in our proposed
deep multi-task framework. The experimental study has shown that the semantic
attribute information can improve the ne-grained SBIR performance in a large margin.
Sketch also has its unique feature like containing temporal information. In sketch synthesis
task, the understandings from both semantic meanings behind sketches and sketching
i
process are required. The semantic meaning of sketches has been well explored in the
sketch recognition, and sketch retrieval challenges. However, the sketching process has
somehow been ignored, even though the sketching process is also very important for us
to understand the sketch modality, especially considering the unique temporal characteristics
of sketches. in this thesis, we proposed the rst deep photo-to-sketch synthesis
framework, which has provided good performance on sketch synthesis task, as shown in
the experiment section.
Generalisability is an important criterion to judge whether the existing methods are able
to be applied to the real world scenario, especially considering the di culties and costly
expense of collecting sketches and pairwise annotation. We thus proposed a generalised
ne-grained SBIR framework. In detail, we follow the meta-learning strategy, and train
a hyper-network to generate instance-level classi cation weights for the latter matching
network. The e ectiveness of the proposed method has been validated by the extensive
experimental results