Ein System für die Online-Erkennung handgeschriebener mathematischer Formeln

Title and Table of Contents 1.Introduction 2. Related Work 3. Preprocessing Techniques for On-Line Handwriting 4. Classification of On-Line Handwritten Symbols 5. Structural Analysis of Mathematical Expressions 6. An Editor for On-Line Handwritten Mathematical Expressions 7. Conclusion BibliographyThis work presents a system for the recognition of on-line handwritten mathematical formulas. The system consists of two main stages: Classification of isolated on-line handwritten symbols and the analysis of spatial relationships among them. We propose a system for the recognition of isolated on-line handwritten characters which is based on support vector classification. We also propose a suitable representation for strokes and symbols which is used to improve the classification rates of the classifier. Our experiments show that our classifier achieved better classification rates in comparison to other popular classification techniques. This could be accomplished by extensive preprocessing of the data and by parameter selection for the support vector classification. We propose a new structural analysis method for the recognition of on-line handwritten mathematical expressions based on a minimum spanning tree construction and symbol dominance. Our method addresses important layout problems frequently encountered in on-line handwritten formula-recognition systems. Our method also aims to handle input as naturally as possible, i.e. using the usual mathematical conventions, without restrictions in the order the symbols are written. Our method handles symbols with non-standard layout, like \sideset{^{*}_{*}}{^{*}_{*}}\prod, as well as tabular layouts, e.g. matrices. Our system for the recognition of on- line handwritten mathematical expressions is used in the Electronic Chalkboard (E-Chalk), a multimedia system for distance-teaching.Die vorliegende Arbeit stellt ein System für die Online-Erkennung handgeschriebener mathematischer Formeln vor. Das System besteht aus zwei verschiedenen Komponenten, einem Klassifikator einzelner handgeschriebener Online-Symbole und einem Analysator mathematischer Strukturen. Die Erkennung der einzelnen Symbole erfolgt mittels Support-Vektor-Maschinen. Aus unserer Experimenten ergab sich, dass unser Klassifikator gegenüber den klassischen Techniken bessere Erkennungsraten erreichte. Diese Ergebnisse wurden durch intensive Vorbearbeitung der Symbole und Suche optimaler Parameter ermöglicht. Unsere Experimente lassen den Schluss zu, dass Support-Vektor-Maschinen den Kompromiss zwischen Trainingszeit und Klassifikationsrate optimieren. In der Arbeit wird eine neue Methode für die Online-Strukturanalyse handgeschriebener mathematischer Ausdrücke besprochen, die sich auf der Aufbau eines minimalen spannenden Baums und Symboldominanz basiert. Diese Technik ermöglicht eine natürliche Eingabe der mathematischen Formeln, d.h., die Symbole und Formeln werden ohne Beschränkungen nach der üblichen mathematischen Notation geschrieben. Unsere Methode lässt sich einfach erweitern, um andere mathematische Strukturen zu erkennen, z.B. Matrizen und andere ungewöhnliche Strukturen, wie die in der LaTeX-Sprache definierte Struktur sideset{^{*}_{*}}{^{*}_{*}} Unser Erkennungssystem wurde in der Programmiersprache Java implementiert und ist das Standard- Formelerkennungssystem des E-Kreide Systems

2D Grammar Extension of the CMP Mathematical Formulae On-line Recognition System

Projecte realitzat en col.laboració amb Czech Technical University in PragueIn the last years, the recognition of handwritten mathematical formulae has recieved an increasing amount of attention in pattern recognition research. However, the diversity of approaches to the problem and the lack of a commercially viable system indicate that there is still much research to be done in this area. In this thesis, I will describe the previous work on a system for on-line handwritten mathematical formulae recognition based on the structural construction paradigm and two-dimensional grammars. In general, this approach can be successfully used in the anaylysis of inputs composed of objects that exhibit rich structural relations. An important benefit of the structural construction is in not treating symbols segmentation and structural anaylsis as two separate processes which allows the system to perform segmentation in the context of the whole formula structure, helping to solve arising ambiguities more reliably. We explore the opening provided by the polynomial complexity parsing algorithm and extend the grammar by many new grammar production rules which made the system useful for formulae met in the real world. We propose several grammar extensions to support a wide range of real mathematical formulae, as well as new features implemented in the application. Our current approach can recognize functions, limits, derivatives, binomial coefficients, complex numbers and more

Symbol detection in online handwritten graphics using Faster R-CNN

Symbol detection techniques in online handwritten graphics (e.g. diagrams and mathematical expressions) consist of methods specifically designed for a single graphic type. In this work, we evaluate the Faster R-CNN object detection algorithm as a general method for detection of symbols in handwritten graphics. We evaluate different configurations of the Faster R-CNN method, and point out issues relative to the handwritten nature of the data. Considering the online recognition context, we evaluate efficiency and accuracy trade-offs of using Deep Neural Networks of different complexities as feature extractors. We evaluate the method on publicly available flowchart and mathematical expression (CROHME-2016) datasets. Results show that Faster R-CNN can be effectively used on both datasets, enabling the possibility of developing general methods for symbol detection, and furthermore, general graphic understanding methods that could be built on top of the algorithm.Comment: Submitted to DAS-201

Intelligent Combination of Structural Analysis Algorithms: Application to Mathematical Expression Recognition

Structural analysis is an important step in many document based recognition problem. Structural analysis is performed to associate elements in a document and assign meaning to their association. Handwritten mathematical expression recognition is one such problem which has been studied and researched for long. Many techniques have been researched to build a system that produce high performance mathematical expression recognition. We have presented a novel method to combine multiple structural recognition algorithms in which the combined result shows better performance than each individual recognition algorithms. In our experiment we have applied our method to combine multiple mathematical expression recognition parsers called DRACULAE. We have used Graph Transformation Network (GTN) which is a network of function based systems in which each system takes graphs as input, apply function and produces a graph as output. GTN is used to combine multiple DRACULAE parsers and its parameter are tuned using gradient based learning. It has been shown that such a combination method can be used to accentuate the strength of individual algorithms in combination to produce better combination result which higher recognition performance. In our experiment we were able to obtain a highest recognition rate of 74% as compared to best recognition result of 70% from individual DRACULAE parsers. Our experiment also resulted into a maximum of 20% reduction of parent recognition errors and maximum 37% reduction in relation recognition errors between symbols in expressions

2D Grammar Extension of the CMP Mathematical Formulae On-line Recognition System

Projecte realitzat en col.laboració amb Czech Technical University in PragueIn the last years, the recognition of handwritten mathematical formulae has recieved an increasing amount of attention in pattern recognition research. However, the diversity of approaches to the problem and the lack of a commercially viable system indicate that there is still much research to be done in this area. In this thesis, I will describe the previous work on a system for on-line handwritten mathematical formulae recognition based on the structural construction paradigm and two-dimensional grammars. In general, this approach can be successfully used in the anaylysis of inputs composed of objects that exhibit rich structural relations. An important benefit of the structural construction is in not treating symbols segmentation and structural anaylsis as two separate processes which allows the system to perform segmentation in the context of the whole formula structure, helping to solve arising ambiguities more reliably. We explore the opening provided by the polynomial complexity parsing algorithm and extend the grammar by many new grammar production rules which made the system useful for formulae met in the real world. We propose several grammar extensions to support a wide range of real mathematical formulae, as well as new features implemented in the application. Our current approach can recognize functions, limits, derivatives, binomial coefficients, complex numbers and more

Features and Algorithms for Visual Parsing of Handwritten Mathematical Expressions

Math expressions are an essential part of scientific documents. Handwritten math expressions recognition can benefit human-computer interaction especially in the education domain and is a critical part of document recognition and analysis. Parsing the spatial arrangement of symbols is an essential part of math expression recognition. A variety of parsing techniques have been developed during the past three decades, and fall into two groups. The first group is graph-based parsing. It selects a path or sub-graph which obeys some rule to form a possible interpretation for the given expression. The second group is grammar driven parsing. Grammars and related parameters are defined manually for different tasks. The time complexity of these two groups parsing is high, and they often impose some strict constraints to reduce the computation. The aim of this thesis is working towards building a straightforward and effective parser with as few constraints as possible. First, we propose using a line of sight graph for representing the layout of strokes and symbols in math expressions. It achieves higher F-score than other graph representations and reduces search space for parsing. Second, we modify the shape context feature with Parzen window density estimation. This feature set works well for symbol segmentation, symbol classification and symbol layout analysis. We get a higher symbol segmentation F-score than other systems on CROHME 2014 dataset. Finally, we develop a Maximum Spanning Tree (MST) based parser using Edmonds\u27 algorithm, which extracts an MST from the directed line of sight graph in two passes: first symbols are segmented, and then symbols and spatial relationship are labeled. The time complexity of our MST-based parsing is lower than the time complexity of CYK parsing with context-free grammars. Also, our MST-based parsing obtains higher structure rate and expression rate than CYK parsing when symbol segmentation is accurate. Correct structure means we get the structure of the symbol layout tree correct, even though the label of the edge in the symbol layout tree might be wrong. The performance of our math expression recognition system with MST-based parsing is competitive on CROHME 2012 and 2014 datasets. For future work, how to incorporate symbol classifier result and correct segmentation error in MST-based parsing needs more research

Handwriten mathematical expression recognition using graph grammars

This thesis presents a graph grammar approach for the recognition of handwritten mathematical expressions. Pen based interfaces provide a natural human computer interaction; interfaces for entering mathematical expressions are no exception to that. The problem is challenging, as it includes the sub-problems of character recognition (OCR) and 2-dimensional structure understanding. Thus, on top of the problems of the standard OCR systems, such as high variation in character shapes, the two dimensional nature of a mathematical expression brings further ambiguity. We use graph grammars for structural understanding of the expressions in order to represent as much information as possible in the parse process. Representing input expression as a graph protects the geometrical relations among the symbols of the input, while alternatives include methods for linearization of the input which may introduce critical errors into the parse process. Also graph grammars have the advantage of flexibility over procedurally coded parse systems. Another important aspect of our system is the fact that all alternative parses are evaluated and the one with maximum likelihood is selected as the intended expression. The likelihoods are estimated according to OCR confidence scores and structural relationships statistics. The segmentation step precedes the parse process, and segments and groups strokes collected from the Tablet input, according to timestamps and distance in space respectively. Then, the segmented symbols are recognized by the OCR engine which uses offline (image) features to allow for flexibility in time dimension, such as adding extra strokes and symbols anytime during the equation. The extracted features are used in an ANN and SVM combination engine returning top-3 character alternatives and confidence values. The parse process expands the graph by generating new tokens with repeated application of grammar rules. At the end, one or more tokens contain the full expression, along with a confidence value based on the 2-dimensional layout of the symbols in the expression and the associated statistics of geometrical relations between symbols. These and the OCR confidence scores are used in disambiguating alternative parses. Our approach is more powerful compared to graph re-writing systems in that all alternative parses are evaluated, rather than selecting the most likely rule application at a particular step, in an irreversible fashion. This also eliminates the need for specifying rule precedences, making system development or use of alternate grammars easier. The only limitation of our system is that segmentation errors are irreversible. That is, the parse process does not handle alternate segmentations, in order to keep the complexity of the parse process down. We alleviate this problem by providing feedback to the user as the segmentation proceeds, in real time. Our user interface gives error correction tools to the user to correct OCR errors and it can generate LATEX code, and MathML codes and graphical rendering of the input handwritten mathematical expression. An extensive collection of mathematical expression and isolated symbols are collected from 15 users for 57 different expressions from a 70-character alphabet. There are, in total, 1710 mathematical expressions and 10500 isolated characters. All samples are in the natural writing styles of the users