A words-of-interest model of sketch representation for image retrieval

In this paper we propose a method for sketch-based image retrieval. Sketch is a magical medium which is capable of conveying semantic messages for user. It’s in accordance with user’s cognitive psychology to retrieve images with sketch. In order to narrow down the semantic gap between the user and the images in database, we preprocess all the images into sketches by the coherent line drawing algorithm. During the process of sketches extraction, saliency maps are used to filter out the redundant background information, while preserve the important semantic information. We use a variant of Words-of-Interest model to retrieve relevant images for the user according to the query. Words-of-Interest (WoI) model is based on Bag-ofvisual Words (BoW) model, which has been proven successfully for information retrieval. Bag-of-Words ignores the spatial relationships among visual words, which are important for sketch representation. Our method takes advantage of the spatial information of the query to select words of interest. Experimental results demonstrate that our sketch-based retrieval method achieves a good tradeoff between retrieval accuracy and semantic representation of users’ query

How human schematization and systematic errors take effect on sketch map formalizations

Dissertation submitted in partial fulfilment of the requirements for the Degree of Master of Science in Geospatial TechnologiesSketch map is an important way to represent spatial information used in many geospatial reasoning tasks (Forbus, K., Usher, J., & Chapman, V. 2004). Compared with verbal or textual language, sketch map is a more interactive mode that more directly supports human spatial thinking and thus is a more natural way to reflect how people perceive properties of spatial objects and their spatial relations. One challenging application of sketch maps is called Spatial-Query-by-Sketch proposed by Egenhofer. Being a design of query language for geographic information systems (GISs), it allows a user to formulate a spatial query by drawing the desired spatial configuration with a pen on a touch-sensitive computer screen and get it translated into a symbolic representation to be processed against a geographic database (Egenhofer, M. 1997). During the period of sketch map drawing, errors due to human spatial cognition in mind may occur. A ready example is as follows: distance judgments for route are judged longer when the route has many turns or landmarks or intersections (Tversky, B. 2002). Direction get straightened up in memory. When Parisians were asked to sketch maps of their city, the Seine was drawn as a curve, but straighter than it actually is (Milgram, S. and Jodelet, D. 1976). Similarly, buildings and streets with different shapes are often simply depicted as schematic figures like blobs and lines. These errors are neither random nor due solely to ignorance; rather they appear to be a consequence of ordinary perceptual and cognitive processes (Tversky, 2003). Therefore, when processing sketch map analysis and representing it in a formal way, like Egenhofer's analysis approach for Spatial-Query-by-Sketch, the resulting formalization must necessarily be wrong if it does not account for the fact that some spatial information is distorted or omitted by humans. Therefore, when sketch map analysis is processed and represented in a formal way same as Egenhofer’s analytical approach to Spatial-Query-by-Sketch, the resulting formalization is simply erroneous since it never takes into account the fact that some spatial information is distorted or neglected in human perceptions. Though Spatial-Query-by-Sketch overcomes the limitations of conventional spatial query language by taking into consideration those alternative interaction methods between users and data, it is still not always true that accuracy of its query results is reliable.(...

A Multilevel Road Alignment Model for Spatial-Query-by-Sketch

A sketch map represents an individual’s perception of a specific location. However, the information in sketch maps is often distorted and incomplete. Nevertheless, the main roads of a given location often exhibit considerable similarities between the sketch maps and metric maps. In this work, a shape-based approach was outlined to align roads in the sketch maps and metric maps. Specifically, the shapes of main roads were compared and analyzed quantitatively and qualitatively in three levels pertaining to an individual road, composite road, and road scene. An experiment was performed in which for eight out of nine maps sketched by our participants, accurate road maps could be obtained automatically taking as input the sketch and the metric map. The experimental results indicate that accurate matches can be obtained when the proposed road alignment approach Shape-based Spatial-Query-by-Sketch (SSQbS) is applied to incomplete or distorted roads present in sketch maps and even to roads with an inconsistent spatial relationship with the roads in the metric maps. Moreover, highly similar matches can be obtained for sketches involving fewer roads

A Content-based search engine on medical images for telemedicine

Retrieving images by content and forming visual queries are important functionality of an image database system. Using textual descriptions to specify queries on image content is another important component of content-based search. The authors describe a medical image database system MIQS which supports visual queries such as query by example and query by sketch. In addition, it supports textual queries on spatial relationships between the objects of an image. MIQS is designed as a client-server application in which the client accesses the database and its images via the WWW.published_or_final_versio

A qualitive reasoning approach for improving query results for sketch based queries by topological analysis of spatial aggregation

Dissertation submitted in partial fulfillment of the requirements for the Degree of Master of Science in Geospatial Technologies.Sketch-based spatial query systems provide an intuitive method of user interaction for spatial databases. These systems must be capable of interpreting user sketches in a way that matches the information that the user intended to provide. One challenge that must be overcome is that humans always simplify the environments they have experienced and this is reflected in the sketches they draw. One such simplification is manifested as aggregation or combination of spatial objects into conceptually or spatially related groups. In this thesis I develop a system that uses reasoning tools of the RCC-8 to evaluate sketchbased queries and provide a method for minimizing the effects of aggregation by determining whether a solution to a query can be expanded if some groups of regions are assumed to be parts of a larger aggregate region. If such a group of regions is found, then this group must be included in the solution. The solution is approximate because the approach taken only verifies that assumed parts of an aggregate are not inconsistent with the configuration of the whole solution. Only cases where the size of the solution equals the size of the query minus one are analysed. It is observed that correctly identifying aggregated regions leads to solutions that are more similar to the original query sketch when the size of every other solution is smaller than the size of the query or when a lower limit is placed on the acceptable size of a solution because the new, expanded or refined solution becomes more complete with respect to the sketch of the query

Learning Cross-Modal Deep Embeddings for Multi-Object Image Retrieval using Text and Sketch

In this work we introduce a cross modal image retrieval system that allows both text and sketch as input modalities for the query. A cross-modal deep network architecture is formulated to jointly model the sketch and text input modalities as well as the the image output modality, learning a common embedding between text and images and between sketches and images. In addition, an attention model is used to selectively focus the attention on the different objects of the image, allowing for retrieval with multiple objects in the query. Experiments show that the proposed method performs the best in both single and multiple object image retrieval in standard datasets.Comment: Accepted at ICPR 201

Asymmetric Feature Maps with Application to Sketch Based Retrieval

We propose a novel concept of asymmetric feature maps (AFM), which allows to evaluate multiple kernels between a query and database entries without increasing the memory requirements. To demonstrate the advantages of the AFM method, we derive a short vector image representation that, due to asymmetric feature maps, supports efficient scale and translation invariant sketch-based image retrieval. Unlike most of the short-code based retrieval systems, the proposed method provides the query localization in the retrieved image. The efficiency of the search is boosted by approximating a 2D translation search via trigonometric polynomial of scores by 1D projections. The projections are a special case of AFM. An order of magnitude speed-up is achieved compared to traditional trigonometric polynomials. The results are boosted by an image-based average query expansion, exceeding significantly the state of the art on standard benchmarks.Comment: CVPR 201