A Diagram Is Worth A Dozen Images

Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation and reasoning, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for over 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs

Learning with multiple representations: An example of a revision lesson in mechanics

We describe an example of learning with multiple representations in an A-level revision lesson on mechanics. The context of the problem involved the motion of a ball thrown vertically upwards in air and studying how the associated physical quantities changed during its flight. Different groups of students were assigned to look at the ball's motion using various representations: motion diagrams, vector diagrams, free-body diagrams, verbal description, equations and graphs, drawn against time as well as against displacement. Overall, feedback from students about the lesson was positive. We further discuss the benefits of using computer simulation to support and extend student learning.Comment: 10 pages, 5 figures, 2 tables http://iopscience.iop.org/0031-912

Drawing OWL 2 ontologies with Eddy the editor

In this paper we introduce Eddy, a new open-source tool for the graphical editing of OWL~2 ontologies. Eddy is specifically designed for creating ontologies in Graphol, a completely visual ontology language that is equivalent to OWL~2. Thus, in Eddy ontologies are easily drawn as diagrams, rather than written as sets of formulas, as commonly happens in popular ontology design and engineering environments. This makes Eddy particularly suited for usage by people who are more familiar with diagramatic languages for conceptual modeling rather than with typical ontology formalisms, as is often required in non-academic and industrial contexts. Eddy provides intuitive functionalities for specifying Graphol diagrams, guarantees their syntactic correctness, and allows for exporting them in standard OWL 2 syntax. A user evaluation study we conducted shows that Eddy is perceived as an easy and intuitive tool for ontology specification

SoundBar: exploiting multiple views in multimodal graph browsing

In this paper we discuss why access to mathematical graphs is problematic for visually impaired people. By a review of graph understanding theory and interviews with visually impaired users, we explain why current non-visual representations are unlikely to provide effective access to graphs. We propose the use of multiple views of the graph, each providing quick access to specific information as a way to improve graph usability. We then introduce a specific multiple view system to improve access to bar graphs called SoundBar which provides an additional quick audio overview of the graph. An evaluation of SoundBar revealed that additional views significantly increased accuracy and reduced time taken in a question answering task

Neural Diagrammatic Reasoning

Diagrams have been shown to be effective tools for humans to represent and reason about complex concepts. They have been widely used to represent concepts in science teaching, to communicate workflow in industries and to measure human fluid intelligence. Mechanised reasoning systems typically encode diagrams into symbolic representations that can be easily processed with rule-based expert systems. This relies on human experts to define the framework of diagram-to-symbol mapping and the set of rules to reason with the symbols. This means the reasoning systems cannot be easily adapted to other diagrams without a new set of human-defined representation mapping and reasoning rules. Moreover such systems are not able to cope with diagram inputs as raw and possibly noisy images. The need for human input and the lack of robustness to noise significantly limit the applications of mechanised diagrammatic reasoning systems. A key research question then arises: can we develop human-like reasoning systems that learn to reason robustly without predefined reasoning rules? To answer this question, I propose Neural Diagrammatic Reasoning, a new family of diagrammatic reasoning systems which does not have the drawbacks of mechanised reasoning systems. The new systems are based on deep neural networks, a recently popular machine learning method that achieved human-level performance on a range of perception tasks such as object detection, speech recognition and natural language processing. The proposed systems are able to learn both diagram to symbol mapping and implicit reasoning rules only from data, with no prior human input about symbols and rules in the reasoning tasks. Specifically I developed EulerNet, a novel neural network model that solves Euler diagram syllogism tasks with 99.5% accuracy. Experiments show that EulerNet learns useful representations of the diagrams and tasks, and is robust to noise and deformation in the input data. I also developed MXGNet, a novel multiplex graph neural architecture that solves Raven Progressive Matrices (RPM) tasks. MXGNet achieves state-of-the-art accuracies on two popular RPM datasets. In addition, I developed Discrete-AIR, an unsupervised learning architecture that learns semi-symbolic representations of diagrams without any labels. Lastly I designed a novel inductive bias module that can be readily used in today’s deep neural networks to improve their generalisation capability on relational reasoning tasks.EPSRC Studentship and Cambridge Trust Scholarshi

Proceedings of the Graduate Student Symposium of the 7th International Conference on the Theory and Application of Diagrams, July 5 2012

Proceedings of the Graduate Student Symposium held at the 7th International Conference on the Theory and Application of Diagrams, ( Diagrams 2012 ), held at the University of Kent on July 5, 2012. Dr. Nathaniel Miller, professor of in the School of Mathematical Sciences at UNC, served on the symposium organizing committee