Characteristics of diagrammatic reasoning

International audienceDiagrammatic, analogical or iconic representations are often contrasted with linguistic or logical representations, in which the shape of the symbols is arbitrary. Although commonly used, diagrams have long suffered from their reputation as mere tools, as mere support for intuition. We list here the main characteristics of diagrammatic inferential systems, and defend the idea that heterogeneous representation systems, including both linguistic and diagrammatic representations, offer real computational perspectives in knowledge modeling and reasoning

Neural Diagrammatic Reasoning

Diagrams have been shown to be effective tools for humans to represent and reason about complex concepts. They have been widely used to represent concepts in science teaching, to communicate workflow in industries and to measure human fluid intelligence. Mechanised reasoning systems typically encode diagrams into symbolic representations that can be easily processed with rule-based expert systems. This relies on human experts to define the framework of diagram-to-symbol mapping and the set of rules to reason with the symbols. This means the reasoning systems cannot be easily adapted to other diagrams without a new set of human-defined representation mapping and reasoning rules. Moreover such systems are not able to cope with diagram inputs as raw and possibly noisy images. The need for human input and the lack of robustness to noise significantly limit the applications of mechanised diagrammatic reasoning systems. A key research question then arises: can we develop human-like reasoning systems that learn to reason robustly without predefined reasoning rules? To answer this question, I propose Neural Diagrammatic Reasoning, a new family of diagrammatic reasoning systems which does not have the drawbacks of mechanised reasoning systems. The new systems are based on deep neural networks, a recently popular machine learning method that achieved human-level performance on a range of perception tasks such as object detection, speech recognition and natural language processing. The proposed systems are able to learn both diagram to symbol mapping and implicit reasoning rules only from data, with no prior human input about symbols and rules in the reasoning tasks. Specifically I developed EulerNet, a novel neural network model that solves Euler diagram syllogism tasks with 99.5% accuracy. Experiments show that EulerNet learns useful representations of the diagrams and tasks, and is robust to noise and deformation in the input data. I also developed MXGNet, a novel multiplex graph neural architecture that solves Raven Progressive Matrices (RPM) tasks. MXGNet achieves state-of-the-art accuracies on two popular RPM datasets. In addition, I developed Discrete-AIR, an unsupervised learning architecture that learns semi-symbolic representations of diagrams without any labels. Lastly I designed a novel inductive bias module that can be readily used in today’s deep neural networks to improve their generalisation capability on relational reasoning tasks.EPSRC Studentship and Cambridge Trust Scholarshi

Hybrid Reasoning and the Future of Iconic Representations

We give a brief overview of the main characteristics of diagrammatic reasoning, analyze a case of human reasoning in a mastermind game, and explain why hybrid representation systems (HRS) are particularly attractive and promising for Artificial General Intelligence and Computer Science in general.Comment: pp. 299-31

Accessible reasoning with diagrams: From cognition to automation

High-tech systems are ubiquitous and often safety and se- curity critical: reasoning about their correctness is paramount. Thus, precise modelling and formal reasoning are necessary in order to convey knowledge unambiguously and accurately. Whilst mathematical mod- elling adds great rigour, it is opaque to many stakeholders which leads to errors in data handling, delays in product release, for example. This is a major motivation for the development of diagrammatic approaches to formalisation and reasoning about models of knowledge. In this paper, we present an interactive theorem prover, called iCon, for a highly expressive diagrammatic logic that is capable of modelling OWL 2 ontologies and, thus, has practical relevance. Significantly, this work is the first to design diagrammatic inference rules using insights into what humans find accessible. Specifically, we conducted an experiment about relative cognitive benefits of primitive (small step) and derived (big step) inferences, and use the results to guide the implementation of inference rules in iCon

Cognitive Conditions of Diagrammatic Reasoning

Forthcoming in Semiotica (ISSN: 0037-1998), published by Walter de Gruyter & Co.In the first part of this paper, I delineate Peirce's general concept of diagrammatic reasoning from other usages of the term that focus either on diagrammatic systems as developed in logic and AI or on reasoning with mental models. The main function of Peirce's form of diagrammatic reasoning is to facilitate individual or social thinking processes in situations that are too complex to be coped with exclusively by internal cognitive means. I provide a diagrammatic definition of diagrammatic reasoning that emphasizes the construction of, and experimentation with, external representations based on the rules and conventions of a chosen representation system. The second part starts with a summary of empirical research regarding cognitive effects of working with diagrams and a critique of approaches that use 'mental models' to explain those effects. The main focus of this section is, however, to elaborate the idea that diagrammatic reasoning should be conceptualized as a case of 'distributed cognition.' Using the mathematics lesson described by Plato in his Meno, I analyze those cognitive conditions of diagrammatic reasoning that are relevant in this case

Synthesising Graphical Theories

In recent years, diagrammatic languages have been shown to be a powerful and expressive tool for reasoning about physical, logical, and semantic processes represented as morphisms in a monoidal category. In particular, categorical quantum mechanics, or "Quantum Picturalism", aims to turn concrete features of quantum theory into abstract structural properties, expressed in the form of diagrammatic identities. One way we search for these properties is to start with a concrete model (e.g. a set of linear maps or finite relations) and start composing generators into diagrams and looking for graphical identities. Naively, we could automate this procedure by enumerating all diagrams up to a given size and check for equalities, but this is intractable in practice because it produces far too many equations. Luckily, many of these identities are not primitive, but rather derivable from simpler ones. In 2010, Johansson, Dixon, and Bundy developed a technique called conjecture synthesis for automatically generating conjectured term equations to feed into an inductive theorem prover. In this extended abstract, we adapt this technique to diagrammatic theories, expressed as graph rewrite systems, and demonstrate its application by synthesising a graphical theory for studying entangled quantum states.Comment: 10 pages, 22 figures. Shortened and one theorem adde

Abstract Diagrammatic Reasoning with Multiplex Graph Networks

Abstract reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems. In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks. MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks. MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels. MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates. We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM). For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%. For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin

Investigating diagrammatic reasoning with deep neural networks

Diagrams in mechanised reasoning systems are typically en- coded into symbolic representations that can be easily processed with rule-based expert systems. This relies on human experts to define the framework of diagram-to-symbol mapping and the set of rules to reason with the symbols. We present a new method of using Deep artificial Neu- ral Networks (DNN) to learn continuous, vector-form representations of diagrams without any human input, and entirely from datasets of dia- grammatic reasoning problems. Based on this DNN, we developed a novel reasoning system, Euler-Net, to solve syllogisms with Euler diagrams. Euler-Net takes two Euler diagrams representing the premises in a syl- logism as input, and outputs either a categorical (subset, intersection or disjoint) or diagrammatic conclusion (generating an Euler diagram rep- resenting the conclusion) to the syllogism. Euler-Net can achieve 99.5% accuracy for generating syllogism conclusion. We analyse the learned representations of the diagrams, and show that meaningful information can be extracted from such neural representations. We propose that our framework can be applied to other types of diagrams, especially the ones we don’t know how to formalise symbolically. Furthermore, we propose to investigate the relation between our artificial DNN and human neural circuitry when performing diagrammatic reasoning

Euler Diagram Transformations

Euler diagrams are a visual language which are used for purposes such as the presentation of set-based data or as the basis of visual logical languages which can be utilised for software specification and reasoning. Such Euler diagram reasoning systems tend to be defined at an abstract level, and the concrete level is simply a visualisation of an abstract model, thereby capturing some subset of the usual boolean logic. The visualisation process tends to be divorced from the data transformation process thereby affecting the user's mental map and reducing the effectiveness of the diagrammatic notation. Furthermore, geometric and topological constraints, called wellformedness conditions, are often placed on the concrete diagrams to try to reduce human comprehension errors, and the effects of these conditions are not modelled in these systems. We view Euler diagrams as a type of graph, where the faces that are present are the key features that convey information and we provide transformations at the dual graph level that correspond to transformations of Euler diagrams, both in terms of editing moves and logical reasoning moves. This original approach gives a correspondence between manipulations of diagrams at an abstract level (such as logical reasoning steps, or simply an update of information) and the manipulation at a concrete level. Thus we facilitate the presentation of diagram changes in a manner that preserves the mental map. The approach will facilitate the realisation of reasoning systems at the concrete level; this has the potential to provide diagrammatic reasoning systems that are inherently different from symbolic logics due to natural geometric constraints. We provide a particular concrete transformation system which preserves the important criteria of planarity and connectivity, which may form part of a framework encompassing multiple concrete systems each adhering to different sets of wellformedness conditions

Quantum Picturalism

The quantum mechanical formalism doesn't support our intuition, nor does it elucidate the key concepts that govern the behaviour of the entities that are subject to the laws of quantum physics. The arrays of complex numbers are kin to the arrays of 0s and 1s of the early days of computer programming practice. In this review we present steps towards a diagrammatic `high-level' alternative for the Hilbert space formalism, one which appeals to our intuition. It allows for intuitive reasoning about interacting quantum systems, and trivialises many otherwise involved and tedious computations. It clearly exposes limitations such as the no-cloning theorem, and phenomena such as quantum teleportation. As a logic, it supports `automation'. It allows for a wider variety of underlying theories, and can be easily modified, having the potential to provide the required step-stone towards a deeper conceptual understanding of quantum theory, as well as its unification with other physical theories. Specific applications discussed here are purely diagrammatic proofs of several quantum computational schemes, as well as an analysis of the structural origin of quantum non-locality. The underlying mathematical foundation of this high-level diagrammatic formalism relies on so-called monoidal categories, a product of a fairly recent development in mathematics. These monoidal categories do not only provide a natural foundation for physical theories, but also for proof theory, logic, programming languages, biology, cooking, ... The challenge is to discover the necessary additional pieces of structure that allow us to predict genuine quantum phenomena.Comment: Commissioned paper for Contemporary Physics, 31 pages, 84 pictures, some colo