783 research outputs found
Free-hand sketch synthesis with deformable stroke models
We present a generative model which can automatically summarize the stroke
composition of free-hand sketches of a given category. When our model is fit to
a collection of sketches with similar poses, it discovers and learns the
structure and appearance of a set of coherent parts, with each part represented
by a group of strokes. It represents both consistent (topology) as well as
diverse aspects (structure and appearance variations) of each sketch category.
Key to the success of our model are important insights learned from a
comprehensive study performed on human stroke data. By fitting this model to
images, we are able to synthesize visually similar and pleasant free-hand
sketches
Free-hand Sketch Understanding and Analysis
PhDWith the proliferation of touch screens, sketching input has become popular among many software
products. This phenomenon has stimulated a new round of boom in free-hand sketch research,
covering topics like sketch recognition, sketch-based image retrieval, sketch synthesis
and sketch segmentation. Comparing to previous sketch works, the newly proposed works are
generally employing more complicated sketches and sketches in much larger quantity, thanks
to the advancements in hardware. This thesis thus demonstrates some new works on free-hand
sketches, presenting novel thoughts on aforementioned topics.
On sketch recognition, Eitz et al. [32] are the first explorers, who proposed the large-scale
TU-Berlin sketch dataset [32] that made sketch recognition possible. Following their work, we
continue to analyze the dataset and find that the visual cue sparsity and internal structural complexity
are the two biggest challenges for sketch recognition. Accordingly, we propose multiple
kernel learning [45] to fuse multiple visual cues and star graph representation [12] to encode the
structures of the sketches. With the new schemes, we have achieved significant improvement
on recognition accuracy (from 56% to 65.81%). Experimental study on sketch attributes is performed
to further boost sketch recognition performance and enable novel retrieval-by-attribute
applications.
For sketch-based image retrieval, we start by carefully examining the existing works. After
looking at the big picture of sketch-based image retrieval, we highlight that studying the sketch’s
ability to distinguish intra-category object variations should be the most promising direction to
proceed on, and we define it as the fine-grained sketch-based image retrieval problem. Deformable
part-based model which addresses object part details and object deformations is raised
to tackle this new problem, and graph matching is employed to compute the similarity between
deformable part-based models by matching the parts of different models. To evaluate this new
problem, we combine the TU-Berlin sketch dataset and the PASCAL VOC photo dataset [36] to
form a new challenging cross-domain dataset with pairwise sketch-photo similarity ratings, and
our proposed method has shown promising results on this new dataset. Regarding sketch synthesis, we focus on the generating of real free-hand style sketches for
general categories, as the closest previous work [8] only managed to show efficacy on a single
category: human faces. The difficulties that impede sketch synthesis to reach other categories
include the cluttered edges and diverse object variations due to deformation. To address those
difficulties, we propose a deformable stroke model to form the sketch synthesis into a detection
process, which is directly aiming at the cluttered background and the object variations. To alleviate
the training of such a model, a perceptual grouping algorithm is further proposed that
utilizes stroke length’s relationship to stroke semantics, stroke temporal order and Gestalt principles
[58] to perform part-level sketch segmentation. The perceptual grouping provides semantic
part-level supervision automatically for the deformable stroke model training, and an iterative
learning scheme is introduced to gradually refine the supervision and the model training. With
the learned deformable stroke models, sketches with distinct free-hand style can be generated for
many categories
Towards Deep Universal Sketch Perceptual Grouper
Human free-hand sketches provide the useful data for studying human perceptual grouping, where the grouping principles such as the Gestalt laws of grouping are naturally in play during both the perception and sketching stages. In this paper, we make the first attempt to develop a universal sketch perceptual grouper. That is, a grouper that can be applied to sketches of any category created with any drawing style and ability, to group constituent strokes/segments into semantically meaningful object parts. The first obstacle to achieving this goal is the lack of large-scale datasets with grouping annotation. To overcome this, we contribute the largest sketch perceptual grouping dataset to date, consisting of 20 000 unique sketches evenly distributed over 25 object categories. Furthermore, we propose a novel deep perceptual grouping model learned with both generative and discriminative losses. The generative loss improves the generalization ability of the model, while the discriminative loss guarantees both local and global grouping consistency. Extensive experiments demonstrate that the proposed grouper significantly outperforms the state-of-the-art competitors. In addition, we show that our grouper is useful for a number of sketch analysis tasks, including sketch semantic segmentation, synthesis, and fine-grained sketch-based image retrieval. © 1992-2012 IEEE
Human motion retrieval based on freehand sketch
In this paper, we present an integrated framework of human motion retrieval based on freehand sketch. With some simple rules, the user can acquire a desired motion by sketching several key postures. To retrieve efficiently and accurately by sketch, the 3D postures are projected onto several 2D planes. The limb direction feature is proposed to represent the input sketch and the projected-postures. Furthermore, a novel index structure based on k-d tree is constructed to index the motions in the database, which speeds up the retrieval process. With our posture-by-posture retrieval algorithm, a continuous motion can be got directly or generated by using a pre-computed graph structure. What's more, our system provides an intuitive user interface. The experimental results demonstrate the effectiveness of our method. © 2014 John Wiley & Sons, Ltd
Deep Learning for Free-Hand Sketch: A Survey
Free-hand sketches are highly illustrative, and have been widely used by
humans to depict objects or stories from ancient times to the present. The
recent prevalence of touchscreen devices has made sketch creation a much easier
task than ever and consequently made sketch-oriented applications increasingly
popular. The progress of deep learning has immensely benefited free-hand sketch
research and applications. This paper presents a comprehensive survey of the
deep learning techniques oriented at free-hand sketch data, and the
applications that they enable. The main contents of this survey include: (i) A
discussion of the intrinsic traits and unique challenges of free-hand sketch,
to highlight the essential differences between sketch data and other data
modalities, e.g., natural photos. (ii) A review of the developments of
free-hand sketch research in the deep learning era, by surveying existing
datasets, research topics, and the state-of-the-art methods through a detailed
taxonomy and experimental evaluation. (iii) Promotion of future work via a
discussion of bottlenecks, open problems, and potential research directions for
the community.Comment: This paper is accepted by IEEE TPAM
Sketch Me That Shoe
This project received support from the European Union’s Horizon 2020 research and innovation programme under grant agreement #640891, the Royal Society and Natural Science Foundation of China (NSFC) joint grant #IE141387 and #61511130081, and the China Scholarship Council (CSC). We gratefully acknowledge the support of NVIDIA Corporation for the donation of the GPUs used for this research
- …