IDATER online conference: graphicacy and modelling 2010

IDATER online conference: graphicacy and modelling 201

Deep Learning for Free-Hand Sketch: A Survey

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community.Comment: This paper is accepted by IEEE TPAM

Teaching GANs to sketch in vector format

Sketching is a fundamental human cognitive ability. Deep Neural Networks (DNNs) have achieved the state-of-the-art performance in recognition tasks like image recognition, speech recognition etc. but have not made significant progress in generating stroke-based sketches a.k.a sketches in vector format. Though there are Variational Auto Encoders (VAEs) for generating sketches in vector format, there is no Generative Adversarial Network (GAN) architecture for the same. In this paper, we propose a standalone GAN architecture called SkeGAN and a hybrid VAE-GAN architecture called VASkeGAN, for sketch generation in vector format. SkeGAN is a stochastic policy in Reinforcement Learning (RL), capable of generating both multidimensional continuous and discrete outputs. VASkeGAN draws sketches by coupling the efficient representation of data by VAE with the powerful generating capabilities of GAN. We have validated that SkeGAN and VASkeGAN generate visually appealing sketches with minimal scribble effect and is comparable to a recent work titled Sketch-RNN. © 2021 ACM

The communication and recording of conceptual design information by the inclusion of visual data

This thesis reports the results of a three year, full-time research project investigating the generation and communication of product descriptions within the conceptual phase of the engineering design process. The research pays particular attention to the role played by the designer's sketch in communicating new product ideas. The investigation commences with a literature review of existing design process models (Chapter 2), which helps to define the area under investigation while presenting modern views of the process in relation to classic examples from established design research. Chapter 3 presents a literature review of the methods currently used to support communication of product descriptions. These methods of Specification are assessed and particular attention is given to new computer-based recording methods such as DOORS and Cradle. Suggestions for improving the efficiency of such models are put forward and the text-only bias of such systems is identified. This comparison of the existing systems thus identifies the research questions. Having identified the possible improvement to be gained by the incorporation of visual material in addition to the universal text description, Chapter 4 presents a literature review assessing the roles of the conceptual sketch in engineering design. As well as presenting views of drawing from philosophical, psychological and scientific standpoints, this section compares attempts made to support the engineer's sketching activity by computer means. This chapter concludes that efforts made to provide effective computer support of sketching by freehand methods are preferred to attempts made to replicate the process with current computer tools. The resulting research experiment, the methodology of which is described in Chapter 5, uses students from the final year of the Product Design Engineering course at Glasgow School of Art and the University of Glasgow. The main aim of the experiment is to identify means of including sketching within the kind of text-based support methods discussed in Chapter 3. It also observes the volume and pattern of information produced by sketch activity throughout the conceptual stages of the design process and aims to find methods which would enable sketches to indicate the general progress of a design. The findings are detailed in Chapter 6

Developing and Evaluating Visual Analogies to Support Insight and Creative Problem Solving

The primary aim of this thesis is to gain a richer understanding of visual analogies for insight problem solving, and, in particular, how they can be better developed to ensure their effectiveness as hints. While much work has explored the role of visual analogies in problem solving and their facilitative role, only a few studies have analysed how they could be designed. This thesis employs a mixed method consisting of a practice-led approach for studying how visual analogies can be designed and developed and an experimental research approach for testing their effectiveness as hints for solving visual insight problems

Shape exploration in product design: assisting transformation in pictorial representations

The sketching of pictorial representations forms a key technique for professional designers in the generation and exploration of product shape. It allows ideas of shape to be externalised and communicated, but more importantly, sketched pictorial representations can operate to assist designers' creative thinking. While computer aided design tools have a proven capability to support the development of design ideas, there is still much scope to develop computer based tools that support the free-flowing exploratory thinking that characterises shape generation and shape exploration in product design. Far from being a straight-jacket in creative design, shape rules have significant potential to bridge the gap between traditional sketching techniques and modern computational methods of design. This thesis presents an inquiry into the exploitation of shape rules within product design. It includes studies of design sketches by professional designers and these inform the development of a theoretical model for assisting design transformation. A formal model of exploration is proposed with two mechanisms; shape decomposition and shape transformation. This model is applied using pictorial representations which may be seen as the computing equivalent of freehand sketches, and reveals new strategies for systematic shape generation and exploration in product design

Exploring Co-creative Drawing Workflows

This article presents the outcomes from a mixed-methods study of drawing practitioners (e.g., professional illustrators, fine artists, and art students) that was conducted in Autumn 2018 as a preliminary investigation for the development of a physical human-AI co-creative drawing system. The aim of the study was to discover possible roles that technology could play in observing, modeling, and possibly assisting an artist with their drawing. The study had three components: a paper survey of artists' drawing practises, technology usage and attitudes, video recorded drawing exercises and a follow-up semi-structured interview which included a co-design discussion on how AI might contribute to their drawing workflow. Key themes identified from the interviews were (1) drawing with physical mediums is a traditional and primary way of creation; (2) artists' views on AI varied, where co-creative AI is preferable to didactic AI; and (3) artists have a critical and skeptical view on the automation of creative work with AI. Participants' input provided the basis for the design and technical specifications of a co-creative drawing prototype, for which details are presented in this article. In addition, lessons learned from conducting the user study are presented with a reflection on future studies with drawing practitioners

Eye Tracking Methods for Analysis of Visuo-Cognitive Behavior in Medical Imaging

Predictive modeling of human visual search behavior and the underlying metacognitive processes is now possible thanks to significant advances in bio-sensing device technology and machine intelligence. Eye tracking bio-sensors, for example, can measure psycho-physiological response through change events in configuration of the human eye. These events include positional changes such as visual fixation, saccadic movements, and scanpath, and non-positional changes such as blinks and pupil dilation and constriction. Using data from eye-tracking sensors, we can model human perception, cognitive processes, and responses to external stimuli. In this study, we investigated the visuo-cognitive behavior of clinicians during the diagnostic decision process for breast cancer screening under clinically equivalent experimental conditions involving multiple monitors and breast projection views. Using a head-mounted eye tracking device and a customized user interface, we recorded eye change events and diagnostic decisions from 10 clinicians (three breast-imaging radiologists and seven Radiology residents) for a corpus of 100 screening mammograms (comprising cases of varied pathology and breast parenchyma density). We proposed novel features and gaze analysis techniques, which help to encode discriminative pattern changes in positional and non-positional measures of eye events. These changes were shown to correlate with individual image readers' identity and experience level, mammographic case pathology and breast parenchyma density, and diagnostic decision. Furthermore, our results suggest that a combination of machine intelligence and bio-sensing modalities can provide adequate predictive capability for the characterization of a mammographic case and image readers diagnostic performance. Lastly, features characterizing eye movements can be utilized for biometric identification purposes. These findings are impactful in real-time performance monitoring and personalized intelligent training and evaluation systems in screening mammography. Further, the developed algorithms are applicable in other application domains involving high-risk visual tasks