Integrating Multiple Sketch Recognition Methods to Improve Accuracy and Speed

Sketch recognition is the computer understanding of hand drawn diagrams. Recognizing sketches instantaneously is necessary to build beautiful interfaces with real time feedback. There are various techniques to quickly recognize sketches into ten or twenty classes. However for much larger datasets of sketches from a large number of classes, these existing techniques can take an extended period of time to accurately classify an incoming sketch and require significant computational overhead. Thus, to make classification of large datasets feasible, we propose using multiple stages of recognition. In the initial stage, gesture-based feature values are calculated and the trained model is used to classify the incoming sketch. Sketches with an accuracy less than a threshold value, go through a second stage of geometric recognition techniques. In the second geometric stage, the sketch is segmented, and sent to shape-specific recognizers. The sketches are matched against predefined shape descriptions, and confidence values are calculated. The system outputs a list of classes that the sketch could be classified as, along with the accuracy, and precision for each sketch. This process both significantly reduces the time taken to classify such huge datasets of sketches, and increases both the accuracy and precision of the recognition

Integrating Multiple Sketch Recognition Methods to Improve Accuracy and Speed

Sketch recognition is the computer understanding of hand drawn diagrams. Recognizing sketches instantaneously is necessary to build beautiful interfaces with real time feedback. There are various techniques to quickly recognize sketches into ten or twenty classes. However for much larger datasets of sketches from a large number of classes, these existing techniques can take an extended period of time to accurately classify an incoming sketch and require significant computational overhead. Thus, to make classification of large datasets feasible, we propose using multiple stages of recognition. In the initial stage, gesture-based feature values are calculated and the trained model is used to classify the incoming sketch. Sketches with an accuracy less than a threshold value, go through a second stage of geometric recognition techniques. In the second geometric stage, the sketch is segmented, and sent to shape-specific recognizers. The sketches are matched against predefined shape descriptions, and confidence values are calculated. The system outputs a list of classes that the sketch could be classified as, along with the accuracy, and precision for each sketch. This process both significantly reduces the time taken to classify such huge datasets of sketches, and increases both the accuracy and precision of the recognition

Mechanix: An Intelligent Web Interface for Automatic Grading of Sketched Free-Body Diagrams

Sketching free body diagrams is an essential skill that students learn in introductory physics and engineering classes; however, university class sizes are growing and often have hundreds of students in a single class. This situation creates a grading challenge for instructors as there is simply not enough time nor resources to provide adequate feedback on every problem. We have developed a web-based application called Mechanix to provide automated real-time feedback on hand-drawn free body diagrams for students. The system is driven by novel sketch recognition algorithms developed for recognizing and comparing trusses, general shapes, and arrows in diagrams. We have discovered students perform as well as paper homework or other online homework systems which only check the final answer through deployment to five universities with 450 students completing homework on the system over the 2018 and 2019 school years. Mechanix has reduced the amount of manual grading required for instructors in those courses while ensuring students can correctly draw the free body diagram

PerSketchTivity: Recognition System and Progressive Learning Analysis

PerSketchTivity is a sketch-based tutoring system for design sketching that allows students to hone their skills in design sketching and self-regulated learning through real-time feedback. Students learn design-sketching fundamentals through drawing exercises of reference shapes starting from basic to complex shapes in all dimensions and subsequently receive real-time feedback assessing their performance. PerSketchTivity consists of a recognition system that evaluates the correctness of a student's sketch and provides real-time feedback, evaluating the sketch based on error (accuracy), smoothness, and speed. The focus of this thesis is to evaluate the performance of the system in terms of the recognition accuracy (does the system correctly understand what the student intended to draw) as well as the educational impact on the sketching abilities of the students practicing with this system. Each student's increase in sketching ability is measured in terms of the accuracy, smoothness, and the speed at which the strokes. Data analysis comparing the early to late sketches showed a statistically significant increase in sketching ability

Multi-domain sketch understanding

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2004.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 121-128).by Christine J. Alvarado.Ph.D

Sketchography - Automatic Grading of Map Sketches for Geography Education

Geography is a vital classroom subject that teaches students about the physical features of the planet we live on. Despite the importance of geographic knowledge, almost 75% of 8th graders scored below proficient in geography on the 2014 National Assessment of Educational Progress. Sketchography is a pen-based intelligent tutoring system that provides real-time feedback to students learning the locations, directions, and topography of rivers around the world. Sketchography uses sketch recognition and artificial intelligence to understand the user’s sketched intentions. As sketches are inherently messy, and even the most expert geographer will draw only a close approximation of the river’s flow, data has been gathered from both novice and expert sketchers. This data, in combination with professors’ grading rubrics and statistically driving AI-algorithms, provide real-time automatic grading that is similar to a human grader’s score. Results show the system to be 94.64% accurate compared to human grading

Rethinking Pen Input Interaction: Enabling Freehand Sketching Through Improved Primitive Recognition

Online sketch recognition uses machine learning and artificial intelligence techniques to interpret markings made by users via an electronic stylus or pen. The goal of sketch recognition is to understand the intention and meaning of a particular user's drawing. Diagramming applications have been the primary beneficiaries of sketch recognition technology, as it is commonplace for the users of these tools to rst create a rough sketch of a diagram on paper before translating it into a machine understandable model, using computer-aided design tools, which can then be used to perform simulations or other meaningful tasks. Traditional methods for performing sketch recognition can be broken down into three distinct categories: appearance-based, gesture-based, and geometric-based. Although each approach has its advantages and disadvantages, geometric-based methods have proven to be the most generalizable for multi-domain recognition. Tools, such as the LADDER symbol description language, have shown to be capable of recognizing sketches from over 30 different domains using generalizable, geometric techniques. The LADDER system is limited, however, in the fact that it uses a low-level recognizer that supports only a few primitive shapes, the building blocks for describing higher-level symbols. Systems which support a larger number of primitive shapes have been shown to have questionable accuracies as the number of primitives increase, or they place constraints on how users must input shapes (e.g. circles can only be drawn in a clockwise motion; rectangles must be drawn starting at the top-left corner). This dissertation allows for a significant growth in the possibility of free-sketch recognition systems, those which place little to no drawing constraints on users. In this dissertation, we describe multiple techniques to recognize upwards of 18 primitive shapes while maintaining high accuracy. We also provide methods for producing confidence values and generating multiple interpretations, and explore the difficulties of recognizing multi-stroke primitives. In addition, we show the need for a standardized data repository for sketch recognition algorithm testing and propose SOUSA (sketch-based online user study application), our online system for performing and sharing user study sketch data. Finally, we will show how the principles we have learned through our work extend to other domains, including activity recognition using trained hand posture cues

A Sketch-Based Educational System for Learning Chinese Handwriting

Learning Chinese as a Second Language (CSL) is a difficult task for students in English-speaking countries due to the large symbol set and complicated writing techniques. Traditional classroom methods of teaching Chinese handwriting have major drawbacks due to human experts’ bias and the lack of assessment on writing techniques. In this work, we propose a sketch-based educational system to help CSL students learn Chinese handwriting faster and better in a novel way. Our system allows students to draw freehand symbols to answer questions, and uses sketch recognition and AI techniques to recognize, assess, and provide feedback in real time. Results have shown that the system reaches a recognition accuracy of 86% on novice learners’ inputs, higher than 95% detection rate for mistakes in writing techniques, and 80.3% F-measure on the classification between expert and novice handwriting inputs

Combining representations for improved sketch recognition

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 89-96).Sketching is a common means of conveying, representing, and preserving information, and it has become a subject of research as a method for human-computer interaction, specifically in the area of computer-aided design. Digitally collected sketches contain both spatial and temporal information; additionally, they may contain a conceptual structure of shapes and sub shapes. These multiple aspects suggest several ways of representing sketches, each with advantages and disadvantages for recognition. Most existing sketch recognitions systems are based on a single representation and do not use all available information. We propose combining several representations and systems as a way to improve recognition accuracy. This thesis presents two methods for combining recognition systems. The first improves recognition by improving segmentation, while the second seeks to predict how well systems will recognize a given domain or symbol and combine their outputs accordingly. We show that combining several recognition systems based on different representations can improve the accuracy of existing recognition methods.by Sonya J. Cates.Ph.D