Bluefish: A Relational Framework for Graphic Representations

Complex graphic representations -- such as annotated visualizations, molecular structure diagrams, or Euclidean geometry -- convey information through overlapping perceptual relations. To author such representations, users are forced to use rigid, purpose-built tools with limited flexibility and expressiveness. User interface (UI) frameworks provide only limited relief as their tree-based models are a poor fit for expressing overlaps. We present Bluefish, a diagramming framework that extends UI architectures to support overlapping perceptual relations. Bluefish graphics are instantiated as relational scenegraphs: hierarchical data structures augmented with adjacency relations. Authors specify these relations with scoped references to components found elsewhere in the scenegraph. For layout, Bluefish lazily materializes necessary coordinate transformations. We demonstrate that Bluefish enables authoring graphic representations across a diverse range of domains while preserving the compositional and abstractional affordances of traditional UI frameworks. Moreover, we show how relational scenegraphs capture previously latent semantics that can later be retargeted (e.g., for screen reader accessibility).Comment: 27 pages, 14 figure

Concrete syntax definition for modeling languages

Model Driven Engineering (MDE) promotes the use of models as primary artefacts of a software development process, as an attempt to handle complexity through abstraction, e.g. to cope with the evolution of execution platforms. MDE follows a stepwise approach, by prescribing to develop abstract models further improved to integrate little by little details relative to the final deployment platforms. Thus, the application of an MDE process results in various models residing at various levels of abstraction. Each one of these models is expressed in a modeling language, in which one may find appropriate concepts for the abstraction level considered. Many advocate to use the right (modeling) language for the right purpose. This means that it is sometimes better approach to use small languages specific to the considered domain and abstraction level, than to use general purpose languages (e.g. UML) when they do not perfectly fit the (modeling) needs. As a matter of fact, an MDE development process, which involves many different domains and abstraction levels, should also involve a large variety of modeling languages. Project managers who want to apply an MDE process need to deal with this language proliferation to such an extent that, in the long run, one may infer that language engineers can become major actors of software development teams. We believe that comprehensive modeling language management facilities may considerably alleviate that MDE drawback. Such facilities may include modeling language definition, extension, adaptation, or composition. To define a (modeling) language, one needs to define its abstract syntax, its semantics, and one or more concrete syntaxes. This thesis focuses on concrete syntax definition for modeling languages, when the abstract syntax is given in the form of a metamodel. We will provide solutions both for textual and graphical concrete syntaxes. Some of our experiences in building textual languages (as MTL, a model transformation language), and graphical languages (as Netsilon, a web-application modeler) has shown that a lot of work was spent in implementing interface using traditional techniques, be it a text processor generated from a compiler compiler specification, or a modeler making use of modern 2D graphical libraries. Indeed, abstract and concrete syntax were implemented in a disconnected way, and it was then necessary to assemble them, which became rapidly clumsy while abstract syntax evolved. We built our solution to concrete syntax definition as companions of the abstract syntax. The definition of concrete syntax we propose here made it possible to build automatic tools able to analyze or synthesize models from/to text, and to create graphical modelers. We will present a metamodel for textual concrete syntax definition to construct constructive reversible grammars. We will also propose a technique for graphical concrete syntax definition following a two-step process: specification and realization. Specification is a restrictive approach in which a metamodel defines a graphical concrete syntax. Both relations with abstract syntax and spatial relationships are expressed by means of constraints. The realization step proposes a way to provide the concrete syntax tree a meaning, by attributing it a graphical appearance, and by expressing possible user interactions. The structure of the document is the following. After introducing in deeper details the problem and the general structure of the solution we propose, we will take a tour of MDE, text and graph grammars. Then, we will present Netsilon as an example of an MDE tool to MDE development, which required both the definition of a graphical and a textual modeling language. The two following sections will present the solutions we propose for textual and graphical concrete syntax definition, respectively. Final remarks and possible improvements, especially regarding reusability in general of MDE meta-artifacts (like metamodels or model transformations), and of concrete syntax in particular, will conclude the document

Generation of abstract geometric art based on exact aesthetics, gestalt theory and graphic design principles

In this thesis artificial intelligence ideas are applied to the domain of fine arts and especially modern art. First, we take a closer look at avant garde art movements of the late 19th and first half of the 20th century. After that, we make an analysis of the knowledge on which this art movement is partly based by considering the fields of aesthetics, gestalt psychology and graphic design. Having formalised general ideas about what a well-formed painting should consist of, we then look at ways of incorporate these ideas in a model for generating a composition. We design a formal framework to which we map the domain concepts. Based on the framework, we make a top-down knowledge decomposition. To demonstrate how our ideas can be applied in a practical situation, we have implemented a prototype system. Theoretically, this system is split up into the front-end part, in which the actual output is generated, and the back-end part, in which artificial intelligence techniques are applied to the actual concepts of composing an artwork. The front-end is partly based on the multimedia generation system called Cuypers, which was developed at the Centrum voor Wiskunde en Informatica (CWI) in Amsterdam. Cuypers was made to generate dynamic presentations and therefore generates XML, which subsequently is transformed into a desired format (XHTML, TIME, SMIL) using XSLT transformation stylesheets. Our system generates output in the Scalable Vector Graphics (SVG) format, which is an XML based standard for vector graphics and animation. The back-end part is based on the formal ideas about art described above. It is implemented in Eclipse Prolog. Finally, we discuss the artistic significance of our results. We conclude this thesis by discussing the advantages and disadvantages of the conceptual decisions, as well as suggesting directions for future research, including ideas for the evaluation of the generated composition

Configurable nD-visualization for complex Building Information Models

With the ongoing development of building information modelling (BIM) towards a comprehensive coverage of all construction project information in a semantically explicit way, visual representations became decoupled from the building information models. While traditional construction drawings implicitly contained the visual representation besides the information, nowadays they are generated on the fly, hard-coded in software applications dedicated to other tasks such as analysis, simulation, structural design or communication. Due to the abstract nature of information models and the increasing amount of digital information captured during construction projects, visual representations are essential for humans in order to access the information, to understand it, and to engage with it. At the same time digital media open up the new field of interactive visualizations. The full potential of BIM can only be unlocked with customized task-specific visualizations, with engineers and architects actively involved in the design and development process of these visualizations. The visualizations must be reusable and reliably reproducible during communication processes. Further, to support creative problem solving, it must be possible to modify and refine them. This thesis aims at reconnecting building information models and their visual representations: on a theoretic level, on the level of methods and in terms of tool support. First, the research seeks to improve the knowledge about visualization generation in conjunction with current BIM developments such as the multimodel. The approach is based on the reference model of the visualization pipeline and addresses structural as well as quantitative aspects of the visualization generation. Second, based on the theoretic foundation, a method is derived to construct visual representations from given visualization specifications. To this end, the idea of a domain-specific language (DSL) is employed. Finally, a software prototype proofs the concept. Using the visualization framework, visual representations can be generated from a specific building information model and a specific visualization description.Mit der fortschreitenden Entwicklung des Building Information Modelling (BIM) hin zu einer umfassenden Erfassung aller Bauprojektinformationen in einer semantisch expliziten Weise werden Visualisierungen von den Gebäudeinformationen entkoppelt. Während traditionelle Architektur- und Bauzeichnungen die visuellen Reprä̈sentationen implizit als Träger der Informationen enthalten, werden sie heute on-the-fly generiert. Die Details ihrer Generierung sind festgeschrieben in Softwareanwendungen, welche eigentlich für andere Aufgaben wie Analyse, Simulation, Entwurf oder Kommunikation ausgelegt sind. Angesichts der abstrakten Natur von Informationsmodellen und der steigenden Menge digitaler Informationen, die im Verlauf von Bauprojekten erfasst werden, sind visuelle Repräsentationen essentiell, um sich die Information erschließen, sie verstehen, durchdringen und mit ihnen arbeiten zu können. Gleichzeitig entwickelt sich durch die digitalen Medien eine neues Feld der interaktiven Visualisierungen. Das volle Potential von BIM kann nur mit angepassten aufgabenspezifischen Visualisierungen erschlossen werden, bei denen Ingenieur*innen und Architekt*innen aktiv in den Entwurf und die Entwicklung dieser Visualisierungen einbezogen werden. Die Visualisierungen müssen wiederverwendbar sein und in Kommunikationsprozessen zuverlässig reproduziert werden können. Außerdem muss es möglich sein, Visualisierungen zu modifizieren und neu zu definieren, um das kreative Problemlösen zu unterstützen. Die vorliegende Arbeit zielt darauf ab, Gebäudemodelle und ihre visuellen Repräsentationen wieder zu verbinden: auf der theoretischen Ebene, auf der Ebene der Methoden und hinsichtlich der unterstützenden Werkzeuge. Auf der theoretischen Ebene trägt die Arbeit zunächst dazu bei, das Wissen um die Erstellung von Visualisierungen im Kontext von Bauprojekten zu erweitern. Der verfolgte Ansatz basiert auf dem Referenzmodell der Visualisierungspipeline und geht dabei sowohl auf strukturelle als auch auf quantitative Aspekte des Visualisierungsprozesses ein. Zweitens wird eine Methode entwickelt, die visuelle Repräsentationen auf Basis gegebener Visualisierungsspezifikationen generieren kann. Schließlich belegt ein Softwareprototyp die Realisierbarkeit des Konzepts. Mit dem entwickelten Framework können visuelle Repräsentationen aus jeweils einem spezifischen Gebäudemodell und einer spezifischen Visualisierungsbeschreibung generiert werden

Über die Pragmatik der Graphischen Modellierung

Graphical models help to understand complex systems. However, with the user interaction paradigms established today, activities such as creating, maintaining or browsing graphical models can be very tedious. This thesis presents an approach to enhance productivity by focusing on the pragmatics of model-based design. The contribution includes an interpretation of the notion of pragmatics, orthogonal to syntax and semantics in Model-Driven Engineering (MDE). A proposal on pragmatics-aware modeling is given, employing sophisticated automated layout algorithms to close the gap between MDE and graph drawing theory. Thus, a view management logic presents customized views on models. These concepts get illustrated with the open source Kiel Integrated Environment for Layout Eclipse Rich Client (KIELER) with multiple applications including editing and simulation and shows how view management helps to tame complexity

Automatic Analysis and Validation of the Chemical Literature

ThesisMethods to automatically extract and validate data from the chemical literature in legacy formats to machine-understandable forms are examined. The work focuses of three types of data: analytical data reported in articles, computational chemistry output files and crystallographic information files (CIFs). It is shown that machines are capable of reading and extracting analytical data from the current legacy formats with high recall and precision. Regular expressions cannot identify chemical names with high precision or recall but non-deterministic methods perform significantly better. The lack of machine-understandable connection tables in the literature has been identified as the major issue preventing molecule-based data-driven science being performed in the area. The extraction of data from computational chemistry output files using parser-like approaches is shown to be not generally possible although such methods work well for input files. A hierarchical regular expression based approach can parse > 99:9% of the output files correctly although significant human input is required to prepare the templates. CIFs may be parsed with extremely high recall and precision, contain connection tables and the data is of high quality. The comparison of bond lengths calculated by two computational chemistry programs show good agreement in general but structures containing specific moieties cause discrepancies. An initial protocol for the high-throughput geometry optimisation of molecules extracted from the CIFs is presented and the refinement of this protocol is discussed. Differences in bond length between calculated and experimentally determined values from the CIFs of less than 0.03 Angstrom are shown to be expected by random error. The final protocol is used to find high-quality structures from crystallography which can be reused for further science.Unilever Centre for Molecular Science Informatic

Security analyses for detecting deserialisation vulnerabilities : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science at Massey University, Palmerston North, New Zealand

An important task in software security is to identify potential vulnerabilities. Attackers exploit security vulnerabilities in systems to obtain confidential information, to breach system integrity, and to make systems unavailable to legitimate users. In recent years, particularly 2012, there has been a rise in reported Java vulnerabilities. One type of vulnerability involves (de)serialisation, a commonly used feature to store objects or data structures to an external format and restore them. In 2015, a deserialisation vulnerability was reported involving Apache Commons Collections, a popular Java library, which affected numerous Java applications. Another major deserialisation-related vulnerability that affected 55\% of Android devices was reported in 2015. Both of these vulnerabilities allowed arbitrary code execution on vulnerable systems by malicious users, a serious risk, and this came as a call for the Java community to issue patches to fix serialisation related vulnerabilities in both the Java Development Kit and libraries. Despite attention to coding guidelines and defensive strategies, deserialisation remains a risky feature and a potential weakness in object-oriented applications. In fact, deserialisation related vulnerabilities (both denial-of-service and remote code execution) continue to be reported for Java applications. Further, deserialisation is a case of parsing where external data is parsed from their external representation to a program's internal data structures and hence, potentially similar vulnerabilities can be present in parsers for file formats and serialisation languages. The problem is, given a software package, to detect either injection or denial-of-service vulnerabilities and propose strategies to prevent attacks that exploit them. The research reported in this thesis casts detecting deserialisation related vulnerabilities as a program analysis task. The goal is to automatically discover this class of vulnerabilities using program analysis techniques, and to experimentally evaluate the efficiency and effectiveness of the proposed methods on real-world software. We use multiple techniques to detect reachability to sensitive methods and taint analysis to detect if untrusted user-input can result in security violations. Challenges in using program analysis for detecting deserialisation vulnerabilities include addressing soundness issues in analysing dynamic features in Java (e.g., native code). Another hurdle is that available techniques mostly target the analysis of applications rather than library code. In this thesis, we develop techniques to address soundness issues related to analysing Java code that uses serialisation, and we adapt dynamic techniques such as fuzzing to address precision issues in the results of our analysis. We also use the results from our analysis to study libraries in other languages, and check if they are vulnerable to deserialisation-type attacks. We then provide a discussion on mitigation measures for engineers to protect their software against such vulnerabilities. In our experiments, we show that we can find unreported vulnerabilities in Java code; and how these vulnerabilities are also present in widely-used serialisers for popular languages such as JavaScript, PHP and Rust. In our study, we discovered previously unknown denial-of-service security bugs in applications/libraries that parse external data formats such as YAML, PDF and SVG

Model-Driven Development of Interactive Multimedia Applications

The development of highly interactive multimedia applications is still a challenging and complex task. In addition to the application logic, multimedia applications typically provide a sophisticated user interface with integrated media objects. As a consequence, the development process involves different experts for software design, user interface design, and media design. There is still a lack of concepts for a systematic development which integrates these aspects. This thesis provides a model-driven development approach addressing this problem. Therefore it introduces the Multimedia Modeling Language (MML), a visual modeling language supporting a design phase in multimedia application development. The language is oriented on well-established software engineering concepts, like UML 2, and integrates concepts from the areas of multimedia development and model-based user interface development. MML allows the generation of code skeletons from the models. Thereby, the core idea is to generate code skeletons which can be directly processed in multimedia authoring tools. In this way, the strengths of both are combined: Authoring tools are used to perform the creative development tasks while models are used to design the overall application structure and to enable a well-coordinated development process. This is demonstrated using the professional authoring tool Adobe Flash. MML is supported by modeling and code generation tools which have been used to validate the approach over several years in various student projects and teaching courses. Additional prototypes have been developed to demonstrate, e.g., the ability to generate code for different target platforms. Finally, it is discussed how models can contribute in general to a better integration of well-structured software development and creative visual design

Dynamic Network Notation: A Graphical Modeling Language to Support the Visualization and Management of Network Effects in Service Platforms

Service platforms have moved into the center of interest in both academic research and the IT industry due to their economic and technical impact. These multitenant platforms provide own or third party software as metered, on-demand services. Corresponding service offers exhibit network effects. The present work introduces a graphical modeling language to support service platform design with focus on the exploitation of these network effects