Online handwritten mathematical expression recognition

This thesis presents a system for online handwritten mathematical expression recognition that involves integrals, summation notation, superscripts and subscripts, square-roots, fractions, trigonometric and logarithmic functions; together with a user-interface for writing scientific article. The aim of this study is to utilize the most convenient man-machine-interface, a pen, for input of mathematical expressions. In pen-enabled devices, handwriting sequences are collected by the digitization of pen movements which outputs an array of coordinates called strokes. A neural network is trained for recognizing each stroke and a recursive algorithm parses the expression by combining neural network output and structure of the expression. The interface associated with the proposed system integrates the built-in recognition capabilities of the Microsoft's Tablet PC-API for recognizing textual input and also supports conversion of hand-drawn figures into PNG format, which enable the user to enter text, mathematics and draw figures in a single interface. After the recognition, all output is combined into one LATEX code and compiled into a PDF file.s The system presented in this thesis provides a natural interface, hence enables easyinput of mathematical expressions in all pen-enabled devices such as tablet PCs, PDAs, external tablet pads, electronic pen-boards etc

Symbol detection in online handwritten graphics using Faster R-CNN

Symbol detection techniques in online handwritten graphics (e.g. diagrams and mathematical expressions) consist of methods specifically designed for a single graphic type. In this work, we evaluate the Faster R-CNN object detection algorithm as a general method for detection of symbols in handwritten graphics. We evaluate different configurations of the Faster R-CNN method, and point out issues relative to the handwritten nature of the data. Considering the online recognition context, we evaluate efficiency and accuracy trade-offs of using Deep Neural Networks of different complexities as feature extractors. We evaluate the method on publicly available flowchart and mathematical expression (CROHME-2016) datasets. Results show that Faster R-CNN can be effectively used on both datasets, enabling the possibility of developing general methods for symbol detection, and furthermore, general graphic understanding methods that could be built on top of the algorithm.Comment: Submitted to DAS-201

Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

Handwritten mathematical expression recognition (HMER) has attracted extensive attention recently. However, current methods cannot explicitly study the interactions between different symbols, which may fail when faced similar symbols. To alleviate this issue, we propose a simple but efficient method to enhance semantic interaction learning (SIL). Specifically, we firstly construct a semantic graph based on the statistical symbol co-occurrence probabilities. Then we design a semantic aware module (SAM), which projects the visual and classification feature into semantic space. The cosine distance between different projected vectors indicates the correlation between symbols. And jointly optimizing HMER and SIL can explicitly enhances the model's understanding of symbol relationships. In addition, SAM can be easily plugged into existing attention-based models for HMER and consistently bring improvement. Extensive experiments on public benchmark datasets demonstrate that our proposed module can effectively enhance the recognition performance. Our method achieves better recognition performance than prior arts on both CROHME and HME100K datasets.Comment: 12 Page

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset