A profile-driven sketching interface for pen-and-paper sketches

This research is funded by the University of Malta under the research grant R30 31330 and is part of the project Innovative ‘Early Design’ Product Prototyping (InPro).Sketching interface tools are developed to allow designers to benefit from the powerful computational tools avail- able in computer aided design systems. However, despite the number of sketching tools such as PDAs and Tablet PCs available on market, designers typically create a number of initial conceptual ideas using paper-based sketches and scribbles such that these tools remain inaccessible to designers in the early design stages. In this paper we describe a profile-driven, paper-based sketching interface which infers the 3D geometry of objects drawn by designers using the traditional pen and paper sketching. We show that by making full use of the shape information present in the scribbled drawing, it is possible to obtain a paper-based sketching interface that retains the simplicity of the early- stage design drawings while allowing for the modeling of a variety of object shapes.peer-reviewe

Phrasing Bimanual Interaction for Visual Design

Architects and other visual thinkers create external representations of their ideas to support early-stage design. They compose visual imagery with sketching to form abstract diagrams as representations. When working with digital media, they apply various visual operations to transform representations, often engaging in complex sequences. This research investigates how to build interactive capabilities to support designers in putting together, that is phrasing, sequences of operations using both hands. In particular, we examine how phrasing interactions with pen and multi-touch input can support modal switching among different visual operations that in many commercial design tools require using menus and tool palettes—techniques originally designed for the mouse, not pen and touch. We develop an interactive bimanual pen+touch diagramming environment and study its use in landscape architecture design studio education. We observe interesting forms of interaction that emerge, and how our bimanual interaction techniques support visual design processes. Based on the needs of architects, we develop LayerFish, a new bimanual technique for layering overlapping content. We conduct a controlled experiment to evaluate its efficacy. We explore the use of wearables to identify which user, and distinguish what hand, is touching to support phrasing together direct-touch interactions on large displays. From design and development of the environment and both field and controlled studies, we derive a set methods, based upon human bimanual specialization theory, for phrasing modal operations through bimanual interactions without menus or tool palettes

Mapping Tasks to Interactions for Graph Exploration and Graph Editing on Interactive Surfaces

Graph exploration and editing are still mostly considered independently and systems to work with are not designed for todays interactive surfaces like smartphones, tablets or tabletops. When developing a system for those modern devices that supports both graph exploration and graph editing, it is necessary to 1) identify what basic tasks need to be supported, 2) what interactions can be used, and 3) how to map these tasks and interactions. This technical report provides a list of basic interaction tasks for graph exploration and editing as a result of an extensive system review. Moreover, different interaction modalities of interactive surfaces are reviewed according to their interaction vocabulary and further degrees of freedom that can be used to make interactions distinguishable are discussed. Beyond the scope of graph exploration and editing, we provide an approach for finding and evaluating a mapping from tasks to interactions, that is generally applicable. Thus, this work acts as a guideline for developing a system for graph exploration and editing that is specifically designed for interactive surfaces.Comment: 21 pages, minor corrections (typos etc.

Hypermedia Systems Development: A Comparative Study of Software Engineers and Graphic Designers

Hypermedia systems development is, in many regards, different from conventional systems development, chief amongst these differences being its multidisciplinary nature. Foremost amongst the roles in hypermedia development are software engineering and graphic design. However, traditionally the tension between software engineers and graphic designers is pronounced. It is therefore important to gain an understanding of the differences between the two camps with a view to bringing them closer together. This paper reports on the findings of a survey of hypermedia developers conducted in Ireland. One of the objectives of the survey was to compare and contrast the development approaches, methods, and techniques used by software engineers with those used by graphic designers. It was found that software engineers and graphic designers are much closer than might be believed in their attitudes on the value and importance of processes and documented working methods. However, graphic designers primarily base development approaches around the use of specific tools, whereas software engineers are more reliant on traditional and object-oriented software development methods. Regarding diagramming methods, there is some evidence of cross-pollination, as software engineers often use informal techniques such as storyboarding and graphic designers use software engineering techniques such as use case diagrams, but graphic designers find software engineering techniques to be less useful than vice versa

Formalizing graphical notations

The thesis describes research into graphical notations for software engineering, with a principal interest in ways of formalizing them. The research seeks to provide a theoretical basis that will help in designing both notations and the software tools that process them. The work starts from a survey of literature on notation, followed by a review of techniques for formal description and for computational handling of notations. The survey concentrates on collecting views of the benefits and the problems attending notation use in software development; the review covers picture description languages, grammars and tools such as generic editors and visual programming environments. The main problem of notation is found to be a lack of any coherent, rigorous description methods. The current approaches to this problem are analysed as lacking in consensus on syntax specification and also lacking a clear focus on a defined concept of notated expression. To address these deficiencies, the thesis embarks upon an exploration of serniotic, linguistic and logical theory; this culminates in a proposed formalization of serniosis in notations, using categorial model theory as a mathematical foundation. An argument about the structure of sign systems leads to an analysis of notation into a layered system of tractable theories, spanning the gap between expressive pictorial medium and subject domain. This notion of 'tectonic' theory aims to treat both diagrams and formulae together. The research gives details of how syntactic structure can be sketched in a mathematical sense, with examples applying to software development diagrams, offering a new solution to the problem of notation specification. Based on these methods, the thesis discusses directions for resolving the harder problems of supporting notation design, processing and computer-aided generic editing. A number of future research areas are thereby opened up. For practical trial of the ideas, the work proceeds to the development and partial implementation of a system to aid the design of notations and editors. Finally the thesis is evaluated as a contribution to theory in an area which has not attracted a standard approach

The Weblog. A Log Used as a Complement to the Paper Log. Subtopic: New Pedagogical Techniques for the Teaching of Times

Within the framework of the subject of Representation in the course of studies in Architecture at Blas Pascal University (UBP), the following teaching object is proposed, which claims to continue the work began in Egrafia 2016 “Challenging the sketch, beyond a log”, regarding the new Technologies in Architectural Representation.It is presented as a complement to the freehand draw and to learn other intuitive and sensory techniques of representation by using technological devices and editing applications.It suggests the use of the mobile device, cell phones, tablets, etc., in the workshop and possible drawing APP: Adobe Draw, Adobe Ideas and Tayasui sketches.Scope and possible teaching applications:Students can work on a weblog as a complement to a paper log. It suggests the integration of other courses of studies at the university, for example with students of the course of studies in Graphic Design, Cinema, Engineering, etc.In the course of studies in Architecture, its objective is the integration of different subjects, such as Representation, Visual Language, Landscape, Interior Design, Electives of Representation, etc., by generating a common digital format for presentations in different stages of the project

Integrating Multiple Sketch Recognition Methods to Improve Accuracy and Speed

Sketch recognition is the computer understanding of hand drawn diagrams. Recognizing sketches instantaneously is necessary to build beautiful interfaces with real time feedback. There are various techniques to quickly recognize sketches into ten or twenty classes. However for much larger datasets of sketches from a large number of classes, these existing techniques can take an extended period of time to accurately classify an incoming sketch and require significant computational overhead. Thus, to make classification of large datasets feasible, we propose using multiple stages of recognition. In the initial stage, gesture-based feature values are calculated and the trained model is used to classify the incoming sketch. Sketches with an accuracy less than a threshold value, go through a second stage of geometric recognition techniques. In the second geometric stage, the sketch is segmented, and sent to shape-specific recognizers. The sketches are matched against predefined shape descriptions, and confidence values are calculated. The system outputs a list of classes that the sketch could be classified as, along with the accuracy, and precision for each sketch. This process both significantly reduces the time taken to classify such huge datasets of sketches, and increases both the accuracy and precision of the recognition

Representation, Recognition and Collaboration with Digital Ink

Pen input for computing devices is now widespread, providing a promising interaction mechanism for many purposes. Nevertheless, the diverse nature of digital ink and varied application domains still present many challenges. First, the sampling rate and resolution of pen-based devices keep improving, making input data more costly to process and store. At the same time, existing applications typically record digital ink either in proprietary formats, which are restricted to single platforms and consequently lack portability, or simply as images, which lose important information. Moreover, in certain domains such as mathematics, current systems are now achieving good recognition rates on individual symbols, in general recognition of complete expressions remains a problem due to the absence of an effective method that can reliably identify the spatial relationships among symbols. Last, but not least, existing digital ink collaboration tools are platform-dependent and typically allow only one input method to be used at a time. Together with the absence of recognition, this has placed significant limitations on what can be done. In this thesis, we investigate these issues and make contributions to each. We first present an algorithm that can accurately approximate a digital ink curve by selecting a certain subset of points from the original trace. This allows a compact representation of digital ink for efficient processing and storage. We then describe an algorithm that can automatically identify certain important features in handwritten symbols. Identifying the features can help us solve a number of problems such as improving two-dimensional mathematical recognition. Last, we present a framework for multi-user online collaboration in a pen-based and graphical environment. This framework is portable across multiple platforms and allows multimodal interactions in collaborative sessions. To demonstrate our ideas, we present InkChat, a whiteboard application, which can be used to conduct collaborative sessions on a shared canvas. It allows participants to use voice and digital ink independently and simultaneously, which has been found useful in remote collaboration