Automatic Adjacency Grammar Generation from User Drawn Sketches

http://www.ieee.orgIn this paper we present an innovative approach to automatically generate adjacency grammars describing graphical symbols. A grammar production is formulated in terms of rulesets of geometrical constraints among symbol primitives. Given a set of symbol instances sketched by a user using a digital pen, our approach infers the grammar productions consisting of the ruleset most likely to occur. The performance of our work is evaluated using a comprehensive benchmarking database of on-line symbols

Computer-aided exploration of architectural design spaces: a digital sketchbook

Het ontwerpproces van architecten vormt vaak geen lineair pad van ontwerpopgave tot eindresultaat, maar wordt veeleer gekenmerkt door exploratie of het doorzoeken van meerdere alternatieven in een (conceptuele) ontwerpruimte. Dit proces wordt in de praktijk vaak ondersteund door manueel schetsen, waarbij de ontwerpers schetsboek kan gelezen worden als een reeks exploraties. Dit soort interactie met de ontwerpruimte wordt in veel mindere mate ondersteund door hedendaagse computerondersteunde ontwerpsystemen. De metafoor van een digitaal schetsboek, waarbij menselijke exploratie wordt versterkt door de (reken)kracht van een computer, is het centrale onderzoeksthema van dit proefschrift. Hoewel het opzet van een ontwerpruimte op het eerste gezicht schatplichtig lijkt aan het onderzoeksveld van de artificiële intelligentie (AI), wordt het ontwerpen hier ruimer geïnterpreteerd dan het oplossen van problemen. Als onderzoeksmethodologie worden vormengrammatica’s ingezet, die enerzijds nauw aanleunen bij de AI en een formeel raamwerk bieden voor de exploratie van ontwerpruimtes, maar tegelijkertijd ook weerstand bieden tegen de AI en een vorm van visueel denken en ambiguïteit toelaten. De twee bijhorende onderzoeksvragen zijn hoe deze vormengrammatica’s digitaal kunnen worden gerepresenteerd, en op welke manier de ontwerper-computer interactie kan gebeuren. De resultaten van deze twee onderzoeksvragen vormen de basis van een nieuw hulpmiddel voor architecten: het digitaal schetsboek

Geometric Deep Learning for Computer-Aided Design: A Survey

Geometric Deep Learning techniques have become a transformative force in the field of Computer-Aided Design (CAD), and have the potential to revolutionize how designers and engineers approach and enhance the design process. By harnessing the power of machine learning-based methods, CAD designers can optimize their workflows, save time and effort while making better informed decisions, and create designs that are both innovative and practical. The ability to process the CAD designs represented by geometric data and to analyze their encoded features enables the identification of similarities among diverse CAD models, the proposition of alternative designs and enhancements, and even the generation of novel design alternatives. This survey offers a comprehensive overview of learning-based methods in computer-aided design across various categories, including similarity analysis and retrieval, 2D and 3D CAD model synthesis, and CAD generation from point clouds. Additionally, it provides a complete list of benchmark datasets and their characteristics, along with open-source codes that have propelled research in this domain. The final discussion delves into the challenges prevalent in this field, followed by potential future research directions in this rapidly evolving field.Comment: 26 pages, 14 figures, journal articl

Formalizing graphical notations

The thesis describes research into graphical notations for software engineering, with a principal interest in ways of formalizing them. The research seeks to provide a theoretical basis that will help in designing both notations and the software tools that process them. The work starts from a survey of literature on notation, followed by a review of techniques for formal description and for computational handling of notations. The survey concentrates on collecting views of the benefits and the problems attending notation use in software development; the review covers picture description languages, grammars and tools such as generic editors and visual programming environments. The main problem of notation is found to be a lack of any coherent, rigorous description methods. The current approaches to this problem are analysed as lacking in consensus on syntax specification and also lacking a clear focus on a defined concept of notated expression. To address these deficiencies, the thesis embarks upon an exploration of serniotic, linguistic and logical theory; this culminates in a proposed formalization of serniosis in notations, using categorial model theory as a mathematical foundation. An argument about the structure of sign systems leads to an analysis of notation into a layered system of tractable theories, spanning the gap between expressive pictorial medium and subject domain. This notion of 'tectonic' theory aims to treat both diagrams and formulae together. The research gives details of how syntactic structure can be sketched in a mathematical sense, with examples applying to software development diagrams, offering a new solution to the problem of notation specification. Based on these methods, the thesis discusses directions for resolving the harder problems of supporting notation design, processing and computer-aided generic editing. A number of future research areas are thereby opened up. For practical trial of the ideas, the work proceeds to the development and partial implementation of a system to aid the design of notations and editors. Finally the thesis is evaluated as a contribution to theory in an area which has not attracted a standard approach

Bluefish: A Relational Framework for Graphic Representations

Complex graphic representations -- such as annotated visualizations, molecular structure diagrams, or Euclidean geometry -- convey information through overlapping perceptual relations. To author such representations, users are forced to use rigid, purpose-built tools with limited flexibility and expressiveness. User interface (UI) frameworks provide only limited relief as their tree-based models are a poor fit for expressing overlaps. We present Bluefish, a diagramming framework that extends UI architectures to support overlapping perceptual relations. Bluefish graphics are instantiated as relational scenegraphs: hierarchical data structures augmented with adjacency relations. Authors specify these relations with scoped references to components found elsewhere in the scenegraph. For layout, Bluefish lazily materializes necessary coordinate transformations. We demonstrate that Bluefish enables authoring graphic representations across a diverse range of domains while preserving the compositional and abstractional affordances of traditional UI frameworks. Moreover, we show how relational scenegraphs capture previously latent semantics that can later be retargeted (e.g., for screen reader accessibility).Comment: 27 pages, 14 figure

Techniques for creating ground-truthed sketch corpora

The problem of recognizing handwritten mathematics notation has been studied for over forty years with little practical success. The poor performance of math recognition systems is due, at least in part, to a lack of realistic data for use in training recognition systems and evaluating their accuracy. In fields for which such data is available, such as face and voice recognition, the data, along with objectively-evaluated recognition contests, has contributed to the rapid advancement of the state of the art. This thesis proposes a method for constructing data corpora not only for hand- written math recognition, but for sketch recognition in general. The method consists of automatically generating template expressions, transcribing these expressions by hand, and automatically labelling them with ground-truth. This approach is motivated by practical considerations and is shown to be more extensible and objective than other potential methods. We introduce a grammar-based approach for the template generation task. In this approach, random derivations in a context-free grammar are controlled so as to generate math expressions for transcription. The generation process may be controlled in terms of expression size and distribution over mathematical semantics. Finally, we present a novel ground-truthing method based on matching terminal symbols in grammar derivations to recognized symbols. The matching is produced by a best-first search through symbol recognition results. Experiments show that this method is highly accurate but rejects many of its inputs

Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art

Procedural content generation (PCG) can be applied to a wide variety of tasks in games, from narratives, levels and sounds, to trees and weapons. A large amount of game content is comprised of graphical assets, such as clouds, buildings or vegetation, that do not require gameplay function considerations. There is also a breadth of literature examining the procedural generation of such elements for purposes outside of games. The body of research, focused on specific methods for generating specific assets, provides a narrow view of the available possibilities. Hence, it is difficult to have a clear picture of all approaches and possibilities, with no guide for interested parties to discover possible methods and approaches for their needs, and no facility to guide them through each technique or approach to map out the process of using them. Therefore, a systematic literature review has been conducted, yielding 200 accepted papers. This paper explores state-of-the-art approaches to graphical asset generation, examining research from a wide range of applications, inside and outside of games. Informed by the literature, a conceptual framework has been derived to address the aforementioned gaps

Modeling and generating moving trees from video

We present a probabilistic approach for the automatic production of tree models with convincing 3D appearance and motion. The only input is a video of a moving tree that provides us an initial dynamic tree model, which is used to generate new individual trees of the same type. Our approach combines global and local constraints to construct a dynamic 3D tree model from a 2D skeleton. Our modeling takes into account factors such as the shape of branches, the overall shape of the tree, and physically plausible motion. Furthermore, we provide a generative model that creates multiple trees in 3D, given a single example model. This means that users no longer have to make each tree individually, or specify rules to make new trees. Results with different species are presented and compared to both reference input data and state of the art alternatives

Landscape generator : method to generate plausible landscape configurations for participatory spatial plan-making

Contemporary regional spatial plan-making in the Netherlands is characterized as a complex process wherein multiple actors, with different levels of interests and demands, try to commonly develop a coherent and comprehensive set of future plan scenarios. The construction of the set of spatial plan scenarios is the core activity of each regional spatial planning process and is often unique and tailored to the specific context and policy objectives formulated for a plan area. Modern collaborative scenario construction is complex due to a variety of participating actors, as public planners, domain experts and non-experts as interest groups and landowners. The level of participation of the non-expert group varies from process to process, but for effective spatial scenarios it is important to ergonomically construct, surprising and plausible scenarios with vivid, proximate and concrete content. The last decades, many attempts have been undertaken to support plan scenario development with digital systems, with strong emphasis on the analytical capabilities of computers. Little attention, however is given to the development of intuitive sketch and design tools and methods, that support the interactive process of large-scale collaborative multi-level plan design, by visualizing and modeling comprehensive landscape scenarios down to the level of cadastral lots. Therefore, the main objective of this research is to develop and evaluate a method, that generates plausible landscape configurations by using user-defined landscape typologies, as a digital support tool for participatory spatial plan-making. To enable the effective design and modeling of vivid and plausible future spatial scenarios, there is a need for a method which supports the two main steps of plan scenario construction in Simlandscape. Simlandscape introduces a rich set of instruments and procedures in order to construct a diverse and coherent scenario set that supports communication and social learning and that facilitate a better informed decision-making process. The central notion in Simlandscape is that actual transformation of the landscape takes place at the ownership lot level. Through construction of strategic spatial scenarios down to the level of individual or clustered lots, comprehensive qualitative and quantitative evaluation becomes possible. Design instruments are proposed, that are intuitive in supporting the funneling creative design process from abstract and general sketches to specific and detailed economic function allocation and landscape layout modeling. The latter activity is supported by the definition and allocation of landscape lot typologies with (non-spatial) attributes. The first step in plan scenario construction in Simlandscape consists of the distribution and allocation of landscape lot typologies to lot geometries. This step poses a complex problem, which can be manually as well as automatically be solved, but is not the core of this research. The second step, assumes that a landscape lot typology is allocated to a lot geometry, and contains generation of a plausible landscape configuration, based on the attributes of the landscape lot typology. This step can also be done manually, but is very time-consuming for a total plan area involved. Therefore, automatic generation of a plausible landscape configuration, based on the properties of the allocated landscape lot typology is important and central subject in this research. The automatic generation of landscape configuration is part of the research field called ‘generative modeling’. In chapter 2, the most of the established existing generative approaches in generative landscape modeling are reviewed for their applicability and relevance as the base for the method to generate plausible landscape configurations from landscape lot typologies. In spatial planning literature, four important more or less distinct fields of research are identified which offer directly or indirectly approaches for developing a generative method: 1) procedural modeling, 2) spatial multi-objective optimization modeling, 3) cellular automata and 4) multi-agent systems. The approaches to generate landscape configurations provide several points of departure. Unfortunately, none of the current approaches is directly applicable for the addressed objective in this research. Procedural modeling techniques as shape or landscape grammars are able to produce, or support the creation of detailed, appealing and realistic landscape visualizations. Due to this level of detail of modeling, the process of inference to identify relevant objects and mutual relations in reality, is complex due to the large number of objects and relations to be modeled. Moreover, the ambiguous character of the relations between objects provides large difficulties in identifying objective and generic rules. Spatial multi-objective optimization modeling in spatial planning problems, as linear integer programming, genetic algorithms and simulated annealing, have a strong theoretical base and are applied frequently in spatial planning literature to provide ‘the most favourable’ landscape and plan layout in terms of minimal development costs. More recently, also general spatial shape objectives are included in the multi-objective functions devised. The research objectives in these studies however, are often restricted to a level of layout planning which is less detailed than the objective stated in this research. A direct consequence is that shape objectives are in general terms of compactness and solely defined at the land-use class level. Furthermore, the number of land-uses to be allocated and the site to be modelled is kept relatively small. These features are enough to provide a proof of principle, but not to deal with realistic planning challenges. Cellular automata and multi-agent systems provide robust frameworks to realistically model subject and object interactions in space and time. However, the non-deterministic behavior and outcomes of the model runs make them less suitable to generate plausible landscape configurations as defined in this research. Chapter 3 describes the (development of the) landscape generator, that is compatible with the regional plan scenario development approach identified in Simlandscape. The landscape generator uses landscape types as building blocks of plan scenarios. A landscape typology describes a proposed future spatial development and contains spatial and (non)spatial (descriptive) attributes. A 2D reference image indirectly provides objective compositional and configurational characteristics of the proposed development. In essence, users allocate a landscape typology to a cadastral lot typology and based on this information, the landscape generator produces a comprehensive landscape configuration. The landscape generator is developed as a multi-objective heuristic optimization modeling approach. In this approach a sequentially updated multi-objective function is optimized for a two-dimensional allocation site. It is assumed that the site is homogeneous in physical characteristics (e.g. height, soil etc.). The multi-objective function is compiled from an available library of single spatial attributes. These spatial attributes and their target values are retrieved from the compositional and configurational characteristics present in the reference image of the landscape typology. Examples from the available spatial attributes are the number of landscape component instances, the relative size of each component or each component instance, compactness and shape of component instances and direct adjacency between two different landscape components. In a hypothetical case study, the capabilities and behavior of the landscape generator are demonstrated. In the case study, the landscape generator generates a variety of landscape configurations for a hypothetical allocation site (20x20 cells) and a rural forest estate as allocated landscape typology. The reference image of the rural forest estate provides detailed information for the compilation of the multi-objective function. The landscape generator contains probabilistic elements (e.g. random starting situation, near-random cell swap), which results in different output, each time it is run with identical input settings. The landscape generator is capable of producing a range of landscape configurations for a variety of situations. A unique situation is defined by the allocation of one landscape typology to one allocation site. Theoretically, since the method is based on the objective measurement of spatial characteristics present in a reference image, each user-defined typology can be used for a selected allocation site. The landscape typologies cannot be allocated to every imaginable dimensioned allocation site, but are bounded by the spatial extent which specifies a valid spatial extent. At the heart of the method lies the compilation of the multi-objective function. Ideally, this compilation can be executed completely objectively and without user-interaction, as the reference image of the landscape typology provides the required information. In the current prototype version of the landscape generator, however, the compilation process is partly (and in advance) controlled by the modeler. The modeler needs to specify which of the available spatial attributes to include, in which sequence to optimize them and what attribute target values to specify. Surely, the modeler is informed by statistics calculated for the reference image. An important task is to define consistent guidelines for the compilation of the multi-objective function from each landscape typology, irrespective of the properties of a valid allocation site. In this research, the modeler has been able to define specific guidelines for each landscape typology. In the current state of the method, a continuous assessment, through iterative testing, needs to be made by the modeler, about which compilation is sufficient in producing plausible configurations and which compilation process produces solutions within reasonable computation times. In chapter 4, a method is presented to obtain insight in the usability of the landscape generator. The produced landscape configurations are extensively evaluated in an extensive internet-based validation experiment. For a broad variety of different situations, landscape configurations are generated by the landscape generator for realistically dimensioned and enclosed sites. The configurations are compared with professional hand-drawn configurations, by a large group of planning professionals. The subjects are provided an interactive, user-friendly web-based inquiry, in which they are requested to (graphically) rank order a random selection out of a total set of landscape configurations (hand-made or computer-generated), from ‘most to least plausible’. The population is not informed about the difference in production process of each landscape configuration. In the experiment a distinction is made between subjective and objective plausibility, representing design quality aspects and representativeness of the landscape typology respectively. Eight different situations (three subjective and five objective) are assessed by the group of respondents and analyzed with a modified version of an approved statistical method, known as ‘the law of comparative judgement’. In addition, to indicate points of interest for further improvements of the methodology, implicit and explicit dimensions of evaluation used by the respondents for each of the objective assessments are identified. The implicit dimensions are identified using linear regression analysis, with single spatial metric properties of the configurations as explanatory variables. To identify explicit dimensions of evaluation the respondents are asked for two of the earlier presented situations, to select five pre-defined used dimensions of evaluation. The current experiment setup provides a robust method as well as reliable results about the capability of the landscape generator to produce plausible landscape configurations. With its modern interactive web interface, its well-balanced data scheme (randomness, several situations) and the use of approved statistical methods, the experiment finds a balance between maximum effective information retrieval and an acceptable level of user workload. In chapter 5, the results of the validation experiment are presented and in chapter 6 these results are analyzed. For each of the three assignments of the design quality test, it is concluded that the whole set of computer-generated configurations is not of comparable design quality as the whole set of professional configurations. Several individual computer-generated landscape configurations have comparable design quality as the professional configurations. The landscape generator is able to produce configurations with landscape components which are with respect to its individual area, shape and relative adjacency plausible. The overall structure is, however, often perceived as near-random. In some situations this is regarded plausible, while in other situations it is regarded implausible. The results of the four analyzed assignments of the representativeness test show a more favorable view on the capabilities of the landscape generator. In half of the cases, the whole set of computer-generated configurations are considered comparable in representativeness to professional onfigurations. In the other half, several individual computer-generated are considered of comparable representativeness. The representativeness test is most important in plausibility validation of the landscape generator, as the primary objective of the research implies that each actor (with different levels of design experience) should be able to provide her development idea (described in the landscape typology) as a comprehensive visualization in an integrated plan scenario. In the initial planning phases of application of the landscape generator, it is more important to obtain a first impression of the impacts (visual and analytical) of a plan scenario than a completely well-modeled and calculated landscape design. Possible non-professional design choices in a landscape typology can be reflected in the generated landscape configurations. Analysis to dimensions of evaluation gives insight into possible explanations for the plausibility ordering of the subjects.A distinction is made between explicit and implicit dimensions of evaluation. Explicit dimensions are directly assembled in the experiment and provide perceived dimensions of evaluations. The implicit dimensions, identified with linear regression analysis are however uncertain in its reliability and ideally should be assembled in relation to explicit dimensions. Results of the linear regression analysis can direct future research with different approaches. First, the attribute target values in the current compilation can be re-specified. Second, non-used but available spatial attributes can be added to the multi-objective function. Third, new spatial attributes may be developed to be included in the optimization process. In light of the main objective in this research, it is important to define consistent guidelines for generating landscape typologies for different situations. In this research, a start is made to identify important choices with respect to the minimal selection of spatial attributes, the influence of its sequence and feasible attribute target value specification. The experiment results further provide detailed directions for improvements of the landscape generator. Other recommendations put forward in this research are related to: 1) the modification of the current heuristic approach (for performance improvement and local trapping avoidance purposes) by hybridization with existing heuristic approaches as simulated annealing and evolutionary algorithms, 2) full-automatic translation from the main characteristics of a landscape typology into the compilation of the multi-objective optimization function; this translation should be as generic as possible and the resulting configurations should be thoroughly validated for plausibility for a variety of possible representative situations (i.e. combination of proposed landscape typology with typical influential allocation site characteristics), 3) extending, if possible, the current library of available spatial attributes with functions that describe more overall organizational properties of landscape typologies or investigation of (parallel or sequential) optimization at different scale levels, 4) the inclusion or extension with representative infrastructure generation and 5) the increase in the effectiveness of the validation experiment by standardizing the acquisition of professional configurations (e.g. designing materials, formats and conditions and automation of conversion to images used in the inquiry) and 6) increase in the reliability of the validation experiment by separating the different parts of the experiment according prioritisation of experiment objectives