User queries over image collections, based on semantic similarity,
can be processed in several ways. In this paper, we propose to reuse the rules
produced by rule-based classifiers in their recognition models as query pattern
definitions for searching image collections

Ivanova, Krassimira

Mitov, Iliya

Spyratos, Nicolas

Stanchev, Peter

Bulgarian Digital Mathematics Library at IMI-BAS

Digital Presentation and Preservation of Cultural and Scientific Heritage, Vol.3, 2013, ISSN: 1314-4006 Query Enrichment for Image Collections by Reuse of Classification Rules  Nicolas Spyratos1, Peter Stanchev2,3, Krassimira Ivanova2, Iliya Mitov2 1: Université Paris-Sud  2: Institute of Mathematics and Informatics, Bulgarian Academy of Sciences, Sofia, Bulgaria 3: Kettering University, Flint, USA  Nicolas.Spyratos@lri.fr, pstanche@kettering.edu, kivanova@math.bas.bg, imitov@math.bas.bg Abstract. User queries over image collections, based on semantic similarity, can be processed in several ways. In this paper, we propose to reuse the rules produced by rule-based classifiers in their recognition models as query pattern definitions for searching image collections. Keywords: Query Reuse, Rule-Based Classification, Image Collections 1.  Introduction Reuse is often described as "not reinventing the wheel" [6]. The process of reusing is inherent to informatics. The classical software reuse has grown out of the macros and subroutines libraries of the 1960s when the assemblers started to offer the possibilities for predefined macros to generate the call and return sequences, using the support for both in-line and separately assembled sequences of codes that could be linked together. The development of this idea has crystallized as a main principle of today's object-oriented programming. Current languages, based on this principle, are building vast collections of reusable software objects and components. Source code, components, development artifacts, patterns, templates all are possible to reuse [1]. In recent years, the process of information reuse has enlarged its scope from program code to data content, using existing content components to create new documents [8] and user interaction.  2.  Features of Rule-based Classifier  In the family of rule-based classifiers fall groups of decision trees, decision rules, association rules. One of the distinctive features of the rule-based classifiers is that they form a human comprehensive recognition model [9]. As a result of the learning phase of such classifiers we receive a set of rules that characterize class labels.  In spite of the decision trees specifics their recognition model, based on split-and-conquer techniques, can easily be transformed into a set of rules. In the decision rules 122 the learned model is represented as a set of IF-THEN rules, produced on the basis of a depth-first induction strategy. Association rules show stable relations between attribute-value pairs that occur frequently in a given dataset, strong associations between frequent patterns (conjunctions of attribute-value pairs) and class labels. In many cases these classifiers give good recognition results and the produced rules can be used as profiles of corresponding class-labels. 3.  Using the Rules from Recognition Model as Query Patterns for Searching Image Collections Image retrieval is an extension to traditional information retrieval. Approaches to image retrieval are somehow derived from conventional information retrieval and are designed to manage the more versatile and enormous amount of visual data which exist. The distinctive characteristic of image collections is that the images can be searched not only by textual metadata, but also on the basis of theirs content. The problem with Content-based Image Retrieval is the so called Semantic gap – the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data has for a user in a given situation [7]. The semantic gap is larger in visual arts images than in natural images since artworks are most often not perfectly realistic. In simple terms, the semantic gap in content-based retrieval stems from the fact that multimedia data is captured by devices in a format, which is optimized for storage and very simple retrieval, and cannot be used to understand what the object "mean". In addition to this, user queries are based on semantic similarity, but the computer usually processes similarity based on low-level feature similarity. To bridge this gap, text annotations by humans were used in conjunction with low-level features of the objects. Another way is to try to build higher level concepts that are comprehensive by humans, on the one hand, and based on the processing of low level features, on the other hand. In this process great importance have the categorization algorithms that allow the system "learning" how to make these decisions. The classification on a test dataset in an image collection using low-level attribute space using rule-based classifiers can produce quite good recognition results for some high-level semantic class-labels. Using MPEG-7 descriptors [4] one can achieve good recognition accuracy for indoors-outdoors, scene types, artists' practices, etc.  In some image collection, we can use a set of produced rules in the recognition model as semantic profiles of corresponding class-labels and include these sets as patterns in the query module, using the set of rules as disjunctive-conjunctive sequence of conditions, and naming them with the name of class-label. In this way the user operates with well-known high-level concepts and this saves him the trouble of understanding and analysing the low-level features, captured by the image analysis. In practice, the low-level features tightly correspond to the "abstract space" of the image content [3], which is connected with the aspects that are specific to art images and reflect cultural influences, specific techniques as well as emotional responses evoked by an image [5]. This fact gives the possibilities to search patterns that define 123 more abstract concepts such as "like/dislike", "exciting/boring", or "relaxing/irritating". Though emotions can be affected by various factors like gender, age, culture, background, etc. and are considered as high-level cognitive processes, they still have certain stability and generality across different people and cultures, which enable researchers to generalize their proposed methodologies from limited samples given a sufficiently large number of observers [10]. If we have quite enough representative part of the observers that are tagged learning set of the images, the produced rules from recognition process can be used to form pattern's queries for the emotional concepts that can be searched as expression from the images. 4.  Example The experiments are made for a small collection of 600 images representing different movements in West-European fine arts – Renaissance, Baroque, Romanticism and Impressionism. The MPEG-7 descriptors for each image are calculated. We have used Dominant Color, Scalable Color, Color Layout, Color Structure, Edge Histogram and Homogeneous Texture MPEG-7 descriptors. The low-level visual information consists of 339 values named with A1 to A339.  A part of the images are also labeled with different high level semantic information, such as “indoor/outdoor”, scene type, artists' name, movement. In the observed case 120 images are labeled with the movement in which their techniques belong. We provide 10-fold cross-validation over this learning dataset using BFTree Classifier [2] and as a result we receive 86.67% classification accuracy.   The recognition model consists of the following tree:  A64 < 9.5 |  A4 < -25.0: Romanticism |  A4 >= -25.0: Baroque A64 >= 9.5 |  A88 < 0.5 |  |  A23 < 2.5 |  |  |  A114 < 3.0: Romanticism |  |  |  A114 >= 3.0: Impressionism |  |  A23 >= 2.5 |  |  |  A206 < 1.5: Romanticism |  |  |  A206 >= 1.5: Renaissance |  A88 >= 0.5 |  |  A11 < -7.5: Renaissance |  |  A11 >= -7.5: Impressionism  As query patterns we can put the following sets of rules with the assumption that the result can contain about 14% false answers, because of the fuzziness of the induction algorithms (Table 1). 124  Table 1. Set of rules using in our example Query Name Search Pattern  Renaissance like (A64>=9.5) and (A88<0.5) and (A23>=2.5) and (A206>=1.5) or (A64>=9.5) and (A88>=0.5) and (A11<-7.5) Baroque like (A64<9.5) and (A4>=-25.0) Romanticism like (A64<9.5) and (A4<-25.0) or (A64>=9.5) and (A88<0.5) and (A23<2.5) and (A114<3.0) or (A64>=9.5) and (A88<0.5) and (A23>=2.5) and (A206<1.5) Impressionism like (A64>=9.5) and (A88<0.5) and (A23<2.5) and (A114>=3.0) or (A64>=9.5) and (A88>=0.5) and (A11>=-7.5)  The user can operate with the queries in a high semantic level, for instance to ask "Which of the images in the collections seem to belong in the Renaissance period", choosing by the names of the input query patterns "Renaissance like". The system will process the low-level visual MPEG-7 features to decide which of the paintings answer the rules conditions, written on the right side of the corresponding query pattern. 5. Conclusion Current digital spaces contain huge amounts of information, with only 1% of the Web data being in textual form, while the rest is of multimedia/streaming nature. There is a clear need to extend the next-generation search tools to accommodate these heterogeneous media. The new search engines must combine search according to textual information or other attributes associated with the files, with the ability of extracting information from the content, which is the scope of action of Content-Based Image Retrieval (CBIR).  The satisfaction of user queries, which are are based on semantic similarity can be achieved in at least three ways:  (1) by supplying text annotations of the digital items by humans, which is a long and hard process and cannot be done for the huge amount of image information available on web;  (2) by trying to annotate automatically with concepts that are comprehensive by humans, based on the processing of low level features using different categorization algorithms; or  (3) by using some advantages of the previous step dynamically, i.e. by not making an annotation in advance and storing metadata, which are not sure that will be used, but by using the query patterns that are formed as a result of previous test annotation that shows enough accurate recognition and use them only when the user query affects the defined concept.  125 In this article we have discussed a variant of this last way showing the possibility to reuse the rules produced by rule-based classifiers in their recognition models as query pattern definitions for searching image collections. Acknowledgment This work was supported in part by the project "VISUAL: Semantic Retrieval in Art collections", No: 01/14 from 21.06.2013, funded by Bulgarian-French programme for scientific cooperation RILA. References [1] Ambler, S.: A Realistic Look at Object-Oriented Reuse. Jan., 1998, http://www.drdobbs.com/a-realistic-look-at-object-oriented-reus/184415594 [2] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I: The WEKA Data Mining Software: An Update; SIGKDD Explorations, 11 (1), 2009 [3] Hurtut, T.: 2D Artistic Images Analysis, a Content-based Survey, 2010, http://hal.archives-ouvertes.fr/hal-00459401_v2/  [4] International Standard ISO/IEC 15938-3 Multimedia Content Description Interface – Part 3: Visual,   http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=34230   [5] Ivanova, K.: A Novel Method for Content-Based Image Retrieval in Art Image Collections Utilizing Colour Semantics. PhD Thesis, Hasselt University, Belgium, 2011. [6] Raymond, E.: The Art of UNIX Programming. Chapter 16 – Reuse. Pearson Education, Inc., 2003, on-line version under the CCA available at:  http://homepage.cs.uri.edu/~thenry/resources/unix_art/index.html  [7] Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content based image retrieval at the end of the early years. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(12), 2000, pp. 1349-1380 [8] Tuan, L.A.: Accessing and Using Complex Multimedia Documents in a Digital Library, Ph.D. thesis the Laboratoire de Recherche en Informatique, Université Paris-Sud, July 2013 [9] Zaiane, O., Antonie, M.-L.: On pruning and tuning rules for associative classifiers. In Proc. of Int. Conf. on Knowledge-Based Intelligence Information & Engineering Systems, LNCS, Vol. 3683, 2005, pp.966-973 [10] Zhang, H., Augilius, E., Honkela, T., Laaksonen J., Gamper H., Alene H.: Analyzing Emotional Semantics of Abstract Art Using Low-Level Image Features. Advances in Intelligent Data Analysis X, Springer LNCS 7014, 2011, pp.413-423 

Query Enrichment for Image Collections by Reuse of Classification Rules

http://sci-gems.math.bas.bg/jspui/bitstream/10525/2080/1/121-125-s1-DiPP2013-5-Spyratos-et-al.pdf

Query Enrichment for Image Collections by Reuse of Classification Rules

Abstract

Similar works

Full text

Available Versions

Bulgarian Digital Mathematics Library at IMI-BAS