Semi-Automated SVG Programming via Direct Manipulation

Direct manipulation interfaces provide intuitive and interactive features to a broad range of users, but they often exhibit two limitations: the built-in features cannot possibly cover all use cases, and the internal representation of the content is not readily exposed. We believe that if direct manipulation interfaces were to (a) use general-purpose programs as the representation format, and (b) expose those programs to the user, then experts could customize these systems in powerful new ways and non-experts could enjoy some of the benefits of programmable systems. In recent work, we presented a prototype SVG editor called Sketch-n-Sketch that offered a step towards this vision. In that system, the user wrote a program in a general-purpose lambda-calculus to generate a graphic design and could then directly manipulate the output to indirectly change design parameters (i.e. constant literals) in the program in real-time during the manipulation. Unfortunately, the burden of programming the desired relationships rested entirely on the user. In this paper, we design and implement new features for Sketch-n-Sketch that assist in the programming process itself. Like typical direct manipulation systems, our extended Sketch-n-Sketch now provides GUI-based tools for drawing shapes, relating shapes to each other, and grouping shapes together. Unlike typical systems, however, each tool carries out the user's intention by transforming their general-purpose program. This novel, semi-automated programming workflow allows the user to rapidly create high-level, reusable abstractions in the program while at the same time retaining direct manipulation capabilities. In future work, our approach may be extended with more graphic design features or realized for other application domains.Comment: In 29th ACM User Interface Software and Technology Symposium (UIST 2016

Deuce: A Lightweight User Interface for Structured Editing

We present a structure-aware code editor, called Deuce, that is equipped with direct manipulation capabilities for invoking automated program transformations. Compared to traditional refactoring environments, Deuce employs a direct manipulation interface that is tightly integrated within a text-based editing workflow. In particular, Deuce draws (i) clickable widgets atop the source code that allow the user to structurally select the unstructured text for subexpressions and other relevant features, and (ii) a lightweight, interactive menu of potential transformations based on the current selections. We implement and evaluate our design with mostly standard transformations in the context of a small functional programming language. A controlled user study with 21 participants demonstrates that structural selection is preferred to a more traditional text-selection interface and may be faster overall once users gain experience with the tool. These results accord with Deuce's aim to provide human-friendly structural interactions on top of familiar text-based editing.Comment: ICSE 2018 Paper + Supplementary Appendice

Open-standards rich media mobile platform & rapid deployment service creation tool

This paper builds upon the work carried out by Brunel University in the field of "Fast Prototyping And Semi-automated User Interface And Application Generation for Converged Broadcast and Cellular Terminals" [1]. This work involved the development of a service creation application for interactive services on mobile devices and methodologies and tools to speed up and deskill the deployment process. This paper aims at further enhancing these tools and presents an enhanced open standards reference platform for mobile digital TV and rich media services. By using a Scalar Vector Graphics (SVG)-driven Java MIDP application (as opposed to bitmapped raster graphics-driven MHP), rich media services can be broadcast to mobile devices running various Java-supported platforms with a user interface scalable to any screen size. Moreover, the Rich Media Mobile Browser is integrated into a service creation tool, therefore enabling rapid testing and deployment of rich mobile media services. The following sections detail the motivation behind the need for a platform which allows for rich media play-out on mobile devices, along with the rich media mobile viewing application and the tools used to create and test rich media with speed and ease

Sketch-n-Sketch: Output-Directed Programming for SVG

For creative tasks, programmers face a choice: Use a GUI and sacrifice flexibility, or write code and sacrifice ergonomics? To obtain both flexibility and ease of use, a number of systems have explored a workflow that we call output-directed programming. In this paradigm, direct manipulation of the program's graphical output corresponds to writing code in a general-purpose programming language, and edits not possible with the mouse can still be enacted through ordinary text edits to the program. Such capabilities provide hope for integrating graphical user interfaces into what are currently text-centric programming environments. To further advance this vision, we present a variety of new output-directed techniques that extend the expressive power of Sketch-n-Sketch, an output-directed programming system for creating programs that generate vector graphics. To enable output-directed interaction at more stages of program construction, we expose intermediate execution products for manipulation and we present a mechanism for contextual drawing. Looking forward to output-directed programming beyond vector graphics, we also offer generic refactorings through the GUI, and our techniques employ a domain-agnostic provenance tracing scheme. To demonstrate the improved expressiveness, we implement a dozen new parametric designs in Sketch-n-Sketch without text-based edits. Among these is the first demonstration of building a recursive function in an output-directed programming setting.Comment: UIST 2019 Paper + Appendi

Common Atlas Format and 3D Brain Atlas Reconstructor: Infrastructure for Constructing 3D Brain Atlases

One of the challenges of modern neuroscience is integrating voluminous data of diferent modalities derived from a variety of specimens. This task requires a common spatial framework that can be provided by brain atlases. The first atlases were limited to two-dimentional presentation of structural data. Recently, attempts at creating 3D atlases have been made to offer navigation within non-standard anatomical planes and improve capability of localization of different types of data within the brain volume. The 3D atlases available so far have been created using frameworks which make it difficult for other researchers to replicate the results. To facilitate reproducible research and data sharing in the field we propose an SVG-based Common Atlas Format (CAF) to store 2D atlas delineations or other compatible data and 3D Brain Atlas Reconstructor (3dBAR), software dedicated to automated reconstruction of three-dimensional brain structures from 2D atlas data. The basic functionality is provided by (1) a set of parsers which translate various atlases from a number of formats into the CAF, and (2) a module generating 3D models from CAF datasets. The whole reconstruction process is reproducible and can easily be configured, tracked and reviewed, which facilitates fixing errors. Manual corrections can be made when automatic reconstruction is not sufficient. The software was designed to simplify interoperability with other neuroinformatics tools by using open file formats. The content can easily be exchanged at any stage of data processing. The framework allows for the addition of new public or proprietary content

Semi-automated mobile television interactive application generation based on XHTML and Java ME

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University, 11/02/2011.Mobile Digital TV (MDTV), the hybrid of Digital Television (DTV) and mobile devices (such as mobile phones), has introduced a new way for people to watch DTV and has brought new opportunities for development in the DTV industry. Nowadays, the development of the next generation MDTV service has progressed in terms of both hardware layers and software, with interactive services/applications becoming one of the future MDTV service trends. However, current MDTV interactive services still lack in terms of attracting the consumers and the service creation and implementation process relies too much on commercial solutions, resulting in most parts of the process being proprietary. In addition, this has increased the technical demands for developers as well as has increased substantially the cost of producing and maintaining MDTV services. In light of the aforementioned situation, the Thesis has contributed to this field, by proposing an innovative MDTV service creation and consumption system based on XHTML and Java ME. On the head-end it introduces a semi-automatic creation mechanism to facilitate a less technical and more efficient interactive service creation process. This enables designers and creative individuals to be actively involved in the MDTV service creation process and to develop interactive-rich MDTV service. On the client-end it employs an open-source software environment as the interactive service MDTV consumption platform, rendering the MDTV service implementation process as less proprietary as possible. Furthermore, the Thesis offers a discussion on the different MDTV interactive application models currently used and based on the proposed software, a novel MDTV service presentation method is further introduced and adopted instead of the Rich Media and ECMAScript based methods. Finally, a series of qualitative testing procedures have been implemented with regards to conducting an essential evaluation on the operability of the proposed software system

Security analyses for detecting deserialisation vulnerabilities : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science at Massey University, Palmerston North, New Zealand

An important task in software security is to identify potential vulnerabilities. Attackers exploit security vulnerabilities in systems to obtain confidential information, to breach system integrity, and to make systems unavailable to legitimate users. In recent years, particularly 2012, there has been a rise in reported Java vulnerabilities. One type of vulnerability involves (de)serialisation, a commonly used feature to store objects or data structures to an external format and restore them. In 2015, a deserialisation vulnerability was reported involving Apache Commons Collections, a popular Java library, which affected numerous Java applications. Another major deserialisation-related vulnerability that affected 55\% of Android devices was reported in 2015. Both of these vulnerabilities allowed arbitrary code execution on vulnerable systems by malicious users, a serious risk, and this came as a call for the Java community to issue patches to fix serialisation related vulnerabilities in both the Java Development Kit and libraries. Despite attention to coding guidelines and defensive strategies, deserialisation remains a risky feature and a potential weakness in object-oriented applications. In fact, deserialisation related vulnerabilities (both denial-of-service and remote code execution) continue to be reported for Java applications. Further, deserialisation is a case of parsing where external data is parsed from their external representation to a program's internal data structures and hence, potentially similar vulnerabilities can be present in parsers for file formats and serialisation languages. The problem is, given a software package, to detect either injection or denial-of-service vulnerabilities and propose strategies to prevent attacks that exploit them. The research reported in this thesis casts detecting deserialisation related vulnerabilities as a program analysis task. The goal is to automatically discover this class of vulnerabilities using program analysis techniques, and to experimentally evaluate the efficiency and effectiveness of the proposed methods on real-world software. We use multiple techniques to detect reachability to sensitive methods and taint analysis to detect if untrusted user-input can result in security violations. Challenges in using program analysis for detecting deserialisation vulnerabilities include addressing soundness issues in analysing dynamic features in Java (e.g., native code). Another hurdle is that available techniques mostly target the analysis of applications rather than library code. In this thesis, we develop techniques to address soundness issues related to analysing Java code that uses serialisation, and we adapt dynamic techniques such as fuzzing to address precision issues in the results of our analysis. We also use the results from our analysis to study libraries in other languages, and check if they are vulnerable to deserialisation-type attacks. We then provide a discussion on mitigation measures for engineers to protect their software against such vulnerabilities. In our experiments, we show that we can find unreported vulnerabilities in Java code; and how these vulnerabilities are also present in widely-used serialisers for popular languages such as JavaScript, PHP and Rust. In our study, we discovered previously unknown denial-of-service security bugs in applications/libraries that parse external data formats such as YAML, PDF and SVG

Autonomously designed free-form 2D DNA origami

Scaffolded DNA origami offers the unique ability to organize molecules in nearly arbitrary spatial patterns at the nanometer scale, with wireframe designs further enabling complex 2D and 3D geometries with irregular boundaries and internal structures. The sequence design of the DNA staple strands needed to fold the long scaffold strand to the target geometry is typically performed manually, limiting the broad application of this materials design paradigm. Here, we present a fully autonomous procedure to design all DNA staple sequences needed to fold any free-form 2D scaffolded DNA origami wireframe object. Our algorithm uses wireframe edges consisting of two parallel DNA duplexes and enables the full autonomy of scaffold routing and staple sequence design with arbitrary network edge lengths and vertex angles. The application of our procedure to geometries with both regular and irregular external boundaries and variable internal structures demonstrates its broad utility for nanoscale materials science and nanotechnology.National Science Foundation (U.S.) (Grant CCF-1564025)National Science Foundation (U.S.) (Grant CMMI-1334109)Office of Naval Research (Grant N000141210621