Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval

We summarize math search engines and search interfaces produced by the Document and Pattern Recognition Lab in recent years, and in particular the min math search interface and the Tangent search engine. Source code for both systems are publicly available. "The Masses" refers to our emphasis on creating systems for mathematical non-experts, who may be looking to define unfamiliar notation, or browse documents based on the visual appearance of formulae rather than their mathematical semantics.Comment: Paper for Invited Talk at 2015 Conference on Intelligent Computer Mathematics (July, Washington DC

European Digital Mathematics Library

The aim of this paper is to survey the European Digital Mathematics Library project goals and achievements as well as an outlook for sustainable development. “Making mathematics literature published in Europe available online” www.eudml.or

Mathematical Expression Recognition based on Probabilistic Grammars

[EN] Mathematical notation is well-known and used all over the world. Humankind has evolved from simple methods representing countings to current well-defined math notation able to account for complex problems. Furthermore, mathematical expressions constitute a universal language in scientific fields, and many information resources containing mathematics have been created during the last decades. However, in order to efficiently access all that information, scientific documents have to be digitized or produced directly in electronic formats. Although most people is able to understand and produce mathematical information, introducing math expressions into electronic devices requires learning specific notations or using editors. Automatic recognition of mathematical expressions aims at filling this gap between the knowledge of a person and the input accepted by computers. This way, printed documents containing math expressions could be automatically digitized, and handwriting could be used for direct input of math notation into electronic devices. This thesis is devoted to develop an approach for mathematical expression recognition. In this document we propose an approach for recognizing any type of mathematical expression (printed or handwritten) based on probabilistic grammars. In order to do so, we develop the formal statistical framework such that derives several probability distributions. Along the document, we deal with the definition and estimation of all these probabilistic sources of information. Finally, we define the parsing algorithm that globally computes the most probable mathematical expression for a given input according to the statistical framework. An important point in this study is to provide objective performance evaluation and report results using public data and standard metrics. We inspected the problems of automatic evaluation in this field and looked for the best solutions. We also report several experiments using public databases and we participated in several international competitions. Furthermore, we have released most of the software developed in this thesis as open source. We also explore some of the applications of mathematical expression recognition. In addition to the direct applications of transcription and digitization, we report two important proposals. First, we developed mucaptcha, a method to tell humans and computers apart by means of math handwriting input, which represents a novel application of math expression recognition. Second, we tackled the problem of layout analysis of structured documents using the statistical framework developed in this thesis, because both are two-dimensional problems that can be modeled with probabilistic grammars. The approach developed in this thesis for mathematical expression recognition has obtained good results at different levels. It has produced several scientific publications in international conferences and journals, and has been awarded in international competitions.[ES] La notación matemática es bien conocida y se utiliza en todo el mundo. La humanidad ha evolucionado desde simples métodos para representar cuentas hasta la notación formal actual capaz de modelar problemas complejos. Además, las expresiones matemáticas constituyen un idioma universal en el mundo científico, y se han creado muchos recursos que contienen matemáticas durante las últimas décadas. Sin embargo, para acceder de forma eficiente a toda esa información, los documentos científicos han de ser digitalizados o producidos directamente en formatos electrónicos. Aunque la mayoría de personas es capaz de entender y producir información matemática, introducir expresiones matemáticas en dispositivos electrónicos requiere aprender notaciones especiales o usar editores. El reconocimiento automático de expresiones matemáticas tiene como objetivo llenar ese espacio existente entre el conocimiento de una persona y la entrada que aceptan los ordenadores. De este modo, documentos impresos que contienen fórmulas podrían digitalizarse automáticamente, y la escritura se podría utilizar para introducir directamente notación matemática en dispositivos electrónicos. Esta tesis está centrada en desarrollar un método para reconocer expresiones matemáticas. En este documento proponemos un método para reconocer cualquier tipo de fórmula (impresa o manuscrita) basado en gramáticas probabilísticas. Para ello, desarrollamos el marco estadístico formal que deriva varias distribuciones de probabilidad. A lo largo del documento, abordamos la definición y estimación de todas estas fuentes de información probabilística. Finalmente, definimos el algoritmo que, dada cierta entrada, calcula globalmente la expresión matemática más probable de acuerdo al marco estadístico. Un aspecto importante de este trabajo es proporcionar una evaluación objetiva de los resultados y presentarlos usando datos públicos y medidas estándar. Por ello, estudiamos los problemas de la evaluación automática en este campo y buscamos las mejores soluciones. Asimismo, presentamos diversos experimentos usando bases de datos públicas y hemos participado en varias competiciones internacionales. Además, hemos publicado como código abierto la mayoría del software desarrollado en esta tesis. También hemos explorado algunas de las aplicaciones del reconocimiento de expresiones matemáticas. Además de las aplicaciones directas de transcripción y digitalización, presentamos dos propuestas importantes. En primer lugar, desarrollamos mucaptcha, un método para discriminar entre humanos y ordenadores mediante la escritura de expresiones matemáticas, el cual representa una novedosa aplicación del reconocimiento de fórmulas. En segundo lugar, abordamos el problema de detectar y segmentar la estructura de documentos utilizando el marco estadístico formal desarrollado en esta tesis, dado que ambos son problemas bidimensionales que pueden modelarse con gramáticas probabilísticas. El método desarrollado en esta tesis para reconocer expresiones matemáticas ha obtenido buenos resultados a diferentes niveles. Este trabajo ha producido varias publicaciones en conferencias internacionales y revistas, y ha sido premiado en competiciones internacionales.[CA] La notació matemàtica és ben coneguda i s'utilitza a tot el món. La humanitat ha evolucionat des de simples mètodes per representar comptes fins a la notació formal actual capaç de modelar problemes complexos. A més, les expressions matemàtiques constitueixen un idioma universal al món científic, i s'han creat molts recursos que contenen matemàtiques durant les últimes dècades. No obstant això, per accedir de forma eficient a tota aquesta informació, els documents científics han de ser digitalitzats o produïts directament en formats electrònics. Encara que la majoria de persones és capaç d'entendre i produir informació matemàtica, introduir expressions matemàtiques en dispositius electrònics requereix aprendre notacions especials o usar editors. El reconeixement automàtic d'expressions matemàtiques té per objectiu omplir aquest espai existent entre el coneixement d'una persona i l'entrada que accepten els ordinadors. D'aquesta manera, documents impresos que contenen fórmules podrien digitalitzar-se automàticament, i l'escriptura es podria utilitzar per introduir directament notació matemàtica en dispositius electrònics. Aquesta tesi està centrada en desenvolupar un mètode per reconèixer expressions matemàtiques. En aquest document proposem un mètode per reconèixer qualsevol tipus de fórmula (impresa o manuscrita) basat en gramàtiques probabilístiques. Amb aquesta finalitat, desenvolupem el marc estadístic formal que deriva diverses distribucions de probabilitat. Al llarg del document, abordem la definició i estimació de totes aquestes fonts d'informació probabilística. Finalment, definim l'algorisme que, donada certa entrada, calcula globalment l'expressió matemàtica més probable d'acord al marc estadístic. Un aspecte important d'aquest treball és proporcionar una avaluació objectiva dels resultats i presentar-los usant dades públiques i mesures estàndard. Per això, estudiem els problemes de l'avaluació automàtica en aquest camp i busquem les millors solucions. Així mateix, presentem diversos experiments usant bases de dades públiques i hem participat en diverses competicions internacionals. A més, hem publicat com a codi obert la majoria del software desenvolupat en aquesta tesi. També hem explorat algunes de les aplicacions del reconeixement d'expressions matemàtiques. A més de les aplicacions directes de transcripció i digitalització, presentem dues propostes importants. En primer lloc, desenvolupem mucaptcha, un mètode per discriminar entre humans i ordinadors mitjançant l'escriptura d'expressions matemàtiques, el qual representa una nova aplicació del reconeixement de fórmules. En segon lloc, abordem el problema de detectar i segmentar l'estructura de documents utilitzant el marc estadístic formal desenvolupat en aquesta tesi, donat que ambdós són problemes bidimensionals que poden modelar-se amb gramàtiques probabilístiques. El mètode desenvolupat en aquesta tesi per reconèixer expressions matemàtiques ha obtingut bons resultats a diferents nivells. Aquest treball ha produït diverses publicacions en conferències internacionals i revistes, i ha sigut premiat en competicions internacionals.Álvaro Muñoz, F. (2015). Mathematical Expression Recognition based on Probabilistic Grammars [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/51665TESI

Non-Visual Representation of Complex Documents for Use in Digital Talking Books

Essential written information such as text books, bills, and catalogues needs to be accessible by everyone. However, access is not always available to vision-impaired people. As they require electronic documents to be available in specific formats. In order to address the accessibility issues of electronic documents, this research aims to design an affordable, portable, standalone and simple to use complete reading system that will convert and describe complex components in electronic documents to print disabled users

Math in the Dark: Tools for Expressing Mathematical Content by Visually Impaired Students

Blind and visually impaired students are under-represented in the science, technology, engineering, and mathematics disciplines of higher education and the workforce. This is due primarily to the difficulties they encounter in trying to succeed in mathematics courses. While there are sufficient tools available to create Braille content, including the special Nemeth Braille used in the U.S. for mathematics constructs, there are very few tools to allow a blind or visually impaired student to create his/her own mathematical content in a manner that sighted individuals can use. The software tools that are available are isolated, do not interface well with other common software, and may be priced for institutional use instead of individual use. Instructors are unprepared or unable to interact with these students in a real-time manner. All of these factors combine to isolate the blind or visually impaired student in the study of mathematics. Nemeth Braille is a complete mathematical markup system in Braille, containing everything that is needed to produce quality math content at all levels of complexity. Blind and visually impaired students should not have to learn any additional markup languages in order to produce math content. This work addressed the needs of the individual blind or visually impaired student who must be able to produce mathematical content for course assignments, and who wishes to interact with peers and instructors on a real-time basis to share mathematical content. Two tools were created to facilitate mathematical interaction: a Nemeth Braille editor, and a real-time instant messenger chat capability that supports Nemeth Braille and MathML constructs. In the Visually Impaired view, the editor accepts Nemeth Braille input, displays the math expressions in a tree structure which will allow sub-expressions to be expanded or collapsed. The Braille constructs can be translated to MathML for display within MathType. Similarly, in the Sighted view, math constructs entered in MathType can be translated into Nemeth Braille. Mathematical content can then be shared between sighted and visually impaired users via the instant messenger chat capability. Using Math in the Dark software, blind and visually impaired students can work math problems fully in Nemeth Braille and can seamlessly convert their work into MathML for viewing by sighted instructors. The converted output has the quality of professionally produced math content. Blind and VI students can also communicate and share math constructs with a sighted partner via a real-time chat feature, with automatic translation in both directions, allowing VI students to obtain help in real-time from a sighted instructor or tutor. By eliminating the burden of translation, this software will help to remove the barriers faced by blind and VI students who wish to excel in the STEM fields of study

Mathspeak: An Audio Method for Presenting Mathematical Formulae to Blind Students

This paper describes the problems involved with learning and understanding math for vision impaired students and developing a computer system approach for rendering mathematical formulae into audio form. Access to mathematics is an obstacle for blind students. The lack of easy access to mathematical resources is a barrier to higher education for many blind students and puts them at an unfair disadvantage in school, academia, and industry [1]. Results from the National Assessment of Educational Progress show that there is great disparity between the math skills of students with disabilities and students without disabilities [2]. A methodology for rendering technical documents, in particular, complex mathematical formula, in an audio descriptive form (Mathspeak) is presented in this paper

MathBrush web application: Design and implementation of an online pen-input interface for computer algebra systems

Several pen-math systems have been developed for mobile and tablet platforms, most notably by the MathBrush project. With the increasing variety of available devices and platforms used by students, this thesis aims to design and implement a version of MathBrush for the web, such that it can be accessible from any device with a web browser. First, a formative study is conducted to gain a current understanding of the common processes used by post-secondary math students for completing assignments, such as: discussing the reliance of using paper, and identifying benefits/limitations of current tools used. Second, the MathBrush web application is implemented which requires creating a new architecture to support the web-based features of the application. Finally, a user study is performed to gain feedback from current math students. This feedback will highlight students' opinions on the application, and will relate back to discussions from the formative study to determine the overall usability of the MathBrush web application

Technical Document Accessibility

Electrical and Electronic Engineerin