Creating the Perception-based LADDER sketch recognition language

Sketch recognition is automated understanding of hand-drawn diagrams. Current sketch recognition systems exist for only a handful of domains, which contain on the order of 10--20 shapes. Our goal was to create a generalized method for recognition that could work for many domains, increasing the number of shapes that could be recognized in real-time, while maintaining a high accuracy. In an effort to effectively recognize shapes while allowing drawing freedom (both drawing-style freedom and perceptually-valid variations), we created the shape description language modeled after the way people naturally describe shapes to 1) create an intuitive and easy to understand description, providing transparency to the underlying recognition process, and 2) to improve recognition by providing recognition flexibility (drawing freedom) that is aligned with how humans perceive shapes. This paper describes the results of a study performed to see how users naturally describe shapes. A sample of 35 subjects described or drew approximately 16 shapes each. Results show a common vocabulary related to Gestalt grouping and singularities. Results also show that perception, similarity, and context play an important role in how people describe shapes. This study resulted in a language (LADDER) that allows shape recognizers for any domain to be automatically generated from a single hand-drawn example of each shape. Sketch systems for over 30 different domains have been automatically generated based on this language. The largest domain contained 923 distinct shapes, and achieved a recognition accuracy of 83% (and a top-3 accuracy of 87%) on a corpus of over 11,000 sketches, which recognizes almost two orders of magnitude more shapes than any other existing system.National Science Foundation (U.S.) (grant 0757557)National Science Foundation (U.S.) (grant 0943499

Integrating Multiple Sketch Recognition Methods to Improve Accuracy and Speed

Sketch recognition is the computer understanding of hand drawn diagrams. Recognizing sketches instantaneously is necessary to build beautiful interfaces with real time feedback. There are various techniques to quickly recognize sketches into ten or twenty classes. However for much larger datasets of sketches from a large number of classes, these existing techniques can take an extended period of time to accurately classify an incoming sketch and require significant computational overhead. Thus, to make classification of large datasets feasible, we propose using multiple stages of recognition. In the initial stage, gesture-based feature values are calculated and the trained model is used to classify the incoming sketch. Sketches with an accuracy less than a threshold value, go through a second stage of geometric recognition techniques. In the second geometric stage, the sketch is segmented, and sent to shape-specific recognizers. The sketches are matched against predefined shape descriptions, and confidence values are calculated. The system outputs a list of classes that the sketch could be classified as, along with the accuracy, and precision for each sketch. This process both significantly reduces the time taken to classify such huge datasets of sketches, and increases both the accuracy and precision of the recognition

Integrating Multiple Sketch Recognition Methods to Improve Accuracy and Speed

Sketch recognition is the computer understanding of hand drawn diagrams. Recognizing sketches instantaneously is necessary to build beautiful interfaces with real time feedback. There are various techniques to quickly recognize sketches into ten or twenty classes. However for much larger datasets of sketches from a large number of classes, these existing techniques can take an extended period of time to accurately classify an incoming sketch and require significant computational overhead. Thus, to make classification of large datasets feasible, we propose using multiple stages of recognition. In the initial stage, gesture-based feature values are calculated and the trained model is used to classify the incoming sketch. Sketches with an accuracy less than a threshold value, go through a second stage of geometric recognition techniques. In the second geometric stage, the sketch is segmented, and sent to shape-specific recognizers. The sketches are matched against predefined shape descriptions, and confidence values are calculated. The system outputs a list of classes that the sketch could be classified as, along with the accuracy, and precision for each sketch. This process both significantly reduces the time taken to classify such huge datasets of sketches, and increases both the accuracy and precision of the recognition

A natural interaction reasoning system for electronic circuit analysis in an educational setting

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 42).This thesis presents a sketch-based interaction system that can be used to illustrate the process of reasoning about an electrical circuit in an educational setting. Recognition of hand-drawn shapes is accomplished in a two stage process where strokes are first processed into primitives like lines or ellipses, then combined into the appropriate circuit device symbols using a shape description language called LADDER. The circuit is then solved by a constraint-propagation reasoning component. The solution is shown to the user along with the justifications that support each deduction. The level of detail and the speed of the solution playback can be customized to tailor to a student's particular learning pace. A small user study was conducted to test the performance of the recognition component, which revealed several recognition problems common to almost all of the users' experiences with the system. Suggestions for dealing with these problems are also presented.by Chang She.M.Eng

Evaluation of Conceptual Sketches on Stylus-Based Devices

Design Sketching is an important tool for designers and creative professionals to express their ideas and thoughts onto visual medium. Being a very critical and versatile skill for engineering students, this course is often taught in universities on pen and paper. However, this traditional pedagogy is limited by the availability of human instructors for their feedback. Also, students having low self-efficacy do not learn efficiently in traditional learning environment. Using intelligent interfaces this problem can be solved where we try to mimic the feedback given by an instructor and assess the student drawn sketches to give them insight of the areas they need to improve on. PerSketchTivity is an intelligent tutoring system which allows students to practice their drawing fundamentals and gives them real-time assessment and feedback. This research deals with finding the evaluation metrics that will enable us to grade students from their sketch data. There are seven metrics that we will work with to analyse how each of them contribute in deciding the quality of the sketches. The main contribution of this research is to identify the features of the sketch that can distinguish a good quality sketch from a poor one and design a grading metric for the sketches that can give a final score between 0 and 1 to the user sketches. Using these obtained features and our grading metric method, we grade all the sketches of students and experts

Hybrid sketching : a new middle ground between 2- and 3-D

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Architecture, 2005.Includes bibliographical references (leaves 124-133).This thesis investigates the geometric representation of ideas during the early stages of design. When a designer's ideas are still in gestation, the exploration of form is more important than its precise specification. Digital modelers facilitate such exploration, but only for forms built with discrete collections of high-level geometric primitives; we introduce techniques that operate on designers' medium of choice, 2-D sketches. Designers' explorations also shift between 2-D and 3-D, yet 3-D form must also be specified with these high-level primitives, requiring an entirely different mindset from 2-D sketching. We introduce a new approach to transform existing 2-D sketches directly into a new kind of sketch-like 3-D model. Finally, we present a novel sketching technique that removes the distinction between 2-D and 3-D altogether. This thesis makes five contributions: point-dragging and curve-drawing techniques for editing sketches; two techniques to help designers bring 2-D sketches to 3-D; and a sketching interface that dissolves the boundaries between 2-D and 3-D representation. The first two contributions of this thesis introduce smooth exploration techniques that work on sketched form composed of strokes, in 2-D or 3-D. First, we present a technique, inspired by classical painting practices, whereby the designer can explore a range of curves with a single stroke. As the user draws near an existing curve, our technique automatically and interactively replaces sections of the old curve with the new one. Second, we present a method to enable smooth exploration of sketched form by point-dragging. The user constructs a high-level "proxy" description that can be used, somewhat like a skeleton, to deform a sketch independent of(cont.) the internal stroke description. Next, we leverage the proxy deformation capability to help the designer move directly from existing 2-D sketches to 3-D models. Our reconstruction techniques generate a novel kind of 3-D model which maintains the appearance and stroke structure of the original 2-D sketch. One technique transforms a single sketch with help from annotations by the designer; the other combines two sketches. Since these interfaces are user-guided, they can operate on ambiguous sketches, relying on the designer to choose an interpretation. Finally, we present an interface to build an even sparser, more suggestive, type of 3-D model, either from existing sketches or from scratch. "Camera planes" provide a complex 3-D scaffolding on which to hang sketches, which can still be drawn as rapidly and freely as before. A sparse set of 2-D sketches placed on planes provides a novel visualization of 3-D form, with enough information present to suggest 3-D shape, but enough missing that the designer can 'read into' the form, seeing multiple possibilities. This unspecified information--this empty space--can spur the designer on to new ideas.by John Alex.Ph.D

Multi-domain sketch understanding

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2004.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 121-128).by Christine J. Alvarado.Ph.D

Eye Tracking Methods for Analysis of Visuo-Cognitive Behavior in Medical Imaging

Predictive modeling of human visual search behavior and the underlying metacognitive processes is now possible thanks to significant advances in bio-sensing device technology and machine intelligence. Eye tracking bio-sensors, for example, can measure psycho-physiological response through change events in configuration of the human eye. These events include positional changes such as visual fixation, saccadic movements, and scanpath, and non-positional changes such as blinks and pupil dilation and constriction. Using data from eye-tracking sensors, we can model human perception, cognitive processes, and responses to external stimuli. In this study, we investigated the visuo-cognitive behavior of clinicians during the diagnostic decision process for breast cancer screening under clinically equivalent experimental conditions involving multiple monitors and breast projection views. Using a head-mounted eye tracking device and a customized user interface, we recorded eye change events and diagnostic decisions from 10 clinicians (three breast-imaging radiologists and seven Radiology residents) for a corpus of 100 screening mammograms (comprising cases of varied pathology and breast parenchyma density). We proposed novel features and gaze analysis techniques, which help to encode discriminative pattern changes in positional and non-positional measures of eye events. These changes were shown to correlate with individual image readers' identity and experience level, mammographic case pathology and breast parenchyma density, and diagnostic decision. Furthermore, our results suggest that a combination of machine intelligence and bio-sensing modalities can provide adequate predictive capability for the characterization of a mammographic case and image readers diagnostic performance. Lastly, features characterizing eye movements can be utilized for biometric identification purposes. These findings are impactful in real-time performance monitoring and personalized intelligent training and evaluation systems in screening mammography. Further, the developed algorithms are applicable in other application domains involving high-risk visual tasks