812 research outputs found
A complete hand-drawn sketch vectorization framework
Vectorizing hand-drawn sketches is a challenging task, which is of paramount
importance for creating CAD vectorized versions for the fashion and creative
workflows. This paper proposes a complete framework that automatically
transforms noisy and complex hand-drawn sketches with different stroke types in
a precise, reliable and highly-simplified vectorized model. The proposed
framework includes a novel line extraction algorithm based on a
multi-resolution application of Pearson's cross correlation and a new unbiased
thinning algorithm that can get rid of scribbles and variable-width strokes to
obtain clean 1-pixel lines. Other contributions include variants of pruning,
merging and edge linking procedures to post-process the obtained paths.
Finally, a modification of the original Schneider's vectorization algorithm is
designed to obtain fewer control points in the resulting Bezier splines. All
the proposed steps of the framework have been extensively tested and compared
with state-of-the-art algorithms, showing (both qualitatively and
quantitatively) its outperformance
Towards Practicality of Sketch-Based Visual Understanding
Sketches have been used to conceptualise and depict visual objects from
pre-historic times. Sketch research has flourished in the past decade,
particularly with the proliferation of touchscreen devices. Much of the
utilisation of sketch has been anchored around the fact that it can be used to
delineate visual concepts universally irrespective of age, race, language, or
demography. The fine-grained interactive nature of sketches facilitates the
application of sketches to various visual understanding tasks, like image
retrieval, image-generation or editing, segmentation, 3D-shape modelling etc.
However, sketches are highly abstract and subjective based on the perception of
individuals. Although most agree that sketches provide fine-grained control to
the user to depict a visual object, many consider sketching a tedious process
due to their limited sketching skills compared to other query/support
modalities like text/tags. Furthermore, collecting fine-grained sketch-photo
association is a significant bottleneck to commercialising sketch applications.
Therefore, this thesis aims to progress sketch-based visual understanding
towards more practicality.Comment: PhD thesis successfully defended by Ayan Kumar Bhunia, Supervisor:
Prof. Yi-Zhe Song, Thesis Examiners: Prof Stella Yu and Prof Adrian Hilto
Reconstruction of machine-made shapes from bitmap sketches
We propose a method of reconstructing 3D machine-made shapes from
bitmap sketches by separating an input image into individual patches and
jointly optimizing their geometry. We rely on two main observations: (1)
human observers interpret sketches of man-made shapes as a collection of
simple geometric primitives, and (2) sketch strokes often indicate occlusion
contours or sharp ridges between those primitives. Using these main observations we design a system that takes a single bitmap image of a shape, estimates image depth and segmentation into primitives with neural networks,
then fits primitives to the predicted depth while determining occlusion contours and aligning intersections with the input drawing via optimization.
Unlike previous work, our approach does not require additional input, annotation, or templates, and does not require retraining for a new category
of man-made shapes. Our method produces triangular meshes that display
sharp geometric features and are suitable for downstream applications, such
as editing, rendering, and shading
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
Self-supervised learning has gained prominence due to its efficacy at
learning powerful representations from unlabelled data that achieve excellent
performance on many challenging downstream tasks. However supervision-free
pre-text tasks are challenging to design and usually modality specific.
Although there is a rich literature of self-supervised methods for either
spatial (such as images) or temporal data (sound or text) modalities, a common
pre-text task that benefits both modalities is largely missing. In this paper,
we are interested in defining a self-supervised pre-text task for sketches and
handwriting data. This data is uniquely characterised by its existence in dual
modalities of rasterized images and vector coordinate sequences. We address and
exploit this dual representation by proposing two novel cross-modal translation
pre-text tasks for self-supervised feature learning: Vectorization and
Rasterization. Vectorization learns to map image space to vector coordinates
and rasterization maps vector coordinates to image space. We show that the our
learned encoder modules benefit both raster-based and vector-based downstream
approaches to analysing hand-drawn data. Empirical evidence shows that our
novel pre-text tasks surpass existing single and multi-modal self-supervision
methods.Comment: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021
Code : https://github.com/AyanKumarBhunia/Self-Supervised-Learning-for-Sketc
Deep Learning for Free-Hand Sketch: A Survey
Free-hand sketches are highly illustrative, and have been widely used by
humans to depict objects or stories from ancient times to the present. The
recent prevalence of touchscreen devices has made sketch creation a much easier
task than ever and consequently made sketch-oriented applications increasingly
popular. The progress of deep learning has immensely benefited free-hand sketch
research and applications. This paper presents a comprehensive survey of the
deep learning techniques oriented at free-hand sketch data, and the
applications that they enable. The main contents of this survey include: (i) A
discussion of the intrinsic traits and unique challenges of free-hand sketch,
to highlight the essential differences between sketch data and other data
modalities, e.g., natural photos. (ii) A review of the developments of
free-hand sketch research in the deep learning era, by surveying existing
datasets, research topics, and the state-of-the-art methods through a detailed
taxonomy and experimental evaluation. (iii) Promotion of future work via a
discussion of bottlenecks, open problems, and potential research directions for
the community.Comment: This paper is accepted by IEEE TPAM
Deep Vectorization of Technical Drawings
We present a new method for vectorization of technical line drawings, such as
floor plans, architectural drawings, and 2D CAD images. Our method includes (1)
a deep learning-based cleaning stage to eliminate the background and
imperfections in the image and fill in missing parts, (2) a transformer-based
network to estimate vector primitives, and (3) optimization procedure to obtain
the final primitive configurations. We train the networks on synthetic data,
renderings of vector line drawings, and manually vectorized scans of line
drawings. Our method quantitatively and qualitatively outperforms a number of
existing techniques on a collection of representative technical drawings
Differential operators on sketches via alpha contours
A vector sketch is a popular and natural geometry representation depicting
a 2D shape. When viewed from afar, the disconnected vector strokes of a
sketch and the empty space around them visually merge into positive space
and negative space, respectively. Positive and negative spaces are the key
elements in the composition of a sketch and define what we perceive as the
shape. Nevertheless, the notion of positive or negative space is mathematically ambiguous: While the strokes unambiguously indicate the interior
or boundary of a 2D shape, the empty space may or may not belong to the
shape’s exterior.
For standard discrete geometry representations, such as meshes or point
clouds, some of the most robust pipelines rely on discretizations of differential operators, such as Laplace-Beltrami. Such discretizations are not
available for vector sketches; defining them may enable numerous applications of classical methods on vector sketches. However, to do so, one needs
to define the positive space of a vector sketch, or the sketch shape.
Even though extracting this 2D sketch shape is mathematically ambiguous,
we propose a robust algorithm, Alpha Contours, constructing its conservative
estimate: a 2D shape containing all the input strokes, which lie in its interior
or on its boundary, and aligning tightly to a sketch. This allows us to define
popular differential operators on vector sketches, such as Laplacian and
Steklov operators.
We demonstrate that our construction enables robust tools for vector
sketches, such as As-Rigid-As-Possible sketch deformation and functional
maps between sketches, as well as solving partial differential equations on a
vector sketch
Automatic Structural Scene Digitalization
In this paper, we present an automatic system for the analysis and labeling
of structural scenes, floor plan drawings in Computer-aided Design (CAD)
format. The proposed system applies a fusion strategy to detect and recognize
various components of CAD floor plans, such as walls, doors, windows and other
ambiguous assets. Technically, a general rule-based filter parsing method is
fist adopted to extract effective information from the original floor plan.
Then, an image-processing based recovery method is employed to correct
information extracted in the first step. Our proposed method is fully automatic
and real-time. Such analysis system provides high accuracy and is also
evaluated on a public website that, on average, archives more than ten
thousands effective uses per day and reaches a relatively high satisfaction
rate.Comment: paper submitted to PloS On
- …