AbstractPatent information and images play important roles to describe the novelty of an invention. However, current patent collections do not support image retrieval and patent images are become almost unsearchable. This paper presents a short review of the existing research work and challenges in patent image retrieval domain. From the review, the image feature extraction step is found to be an important step to match the query and database images successfully. In order to improve the current feature extraction step in image patent retrieval, we propose a patent image retrieval approach based on Affine-SIFT technique. Comparison discussions between the existing feature extraction techniques are presented to assess the potential of this proposed approach

Aghamohammadi, Amirhossein

Ang, Mei Choo

Mogharrebi, Marzieh

Ng, Kok Weng

Prabuwono, Anton Satria

Elsevier - Publisher Connector 

 Procedia Technology  11 ( 2013 )  912 – 918 
2212-0173 © 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and peer-review under responsibility of the Faculty of Information Science & Technology, Universiti Kebangsaan Malaysia.
doi: 10.1016/j.protcy.2013.12.275 
The 4th International Conference on Electrical Engineering and Informatics (ICEEI 2013)
Retrieval System for Patent Images
Marzieh Mogharrebia,*, Mei Choo Anga, Anton Satria Prabuwonob,
Amirhossein Aghamohammadia, Kok Weng Ngc
a Institute of Visual Informatics (IVI), Universiti Kebangsaan Malaysia, 43600 UKM Bangi, Selangor, Malaysia
b Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia,43600 UKM Bangi, Selangor, Malaysia
c Design Engineering Section, Product Design and Engineering Centre, SIRIM Berhad Bukit Jalil, Kuala Lumpur, Malaysia
Abstract
Patent information and images play important roles to describe the novelty of an invention. However, current patent collections 
do not support image retrieval and patent images are become almost unsearchable. This paper presents a short review of the 
existing research work and challenges in patent image retrieval domain. From the review, the image feature extraction step is
found to be an important step to match the query and database images successfully. In order to improve the current feature 
extraction step in image patent retrieval, we propose a patent image retrieval approach based on Affine-SIFT technique. 
Comparison discussions between the existing feature extraction techniques are presented to assess the potential of this proposed 
approach.
© 2013 The Authors. Published by Elsevier B.V.
Selection and peer-review under responsibility of the Faculty of Information Science and Technology, Universiti Kebangsaan 
Malaysia.
Keywords:Image retrieval, patent, search engine, feature extraction;
1. Introduction
The patent office has a huge amount of patent information that describes ideas and solutions to a variety design 
problems ranging from mechanical, chemical, electronics and many fields. These descriptions are usually 
* Corresponding author. Tel.: +6014-2378922.
E-mail address: Mogharrebi.marzieh@gmail.com
Available online at www.sciencedirect.com
   uthors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
i  and peer-review unde  respons bility of the Faculty o  Information S ience & Technology, Universiti Keb ngsaan 
i .
ScienceDirect
913 Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
documented both in text and in graphical/imagery forms. However, the current patent documentation is arranged in a 
poorly structured manner and these graphical information is almost unsearchable [1]. In other word, although the 
patent information in the patent documentation is consisted of texts and graphics/images describing the patent, the 
graphics/images information is ignored in the search. Therefore, it is common in any particular patent search, a large 
amount of data is retrieved and a significant amount of the data retrieved is irrelevant or not useful to the user. For a 
user, the search for patent information is a very time-consuming and a very tedious process which inevitably does 
not help the users to perform their work better. Research has shown that "graphical representations or diagrams are 
sometimes worth more than ten thousand words". This reflects the importance of the graphical information and this 
also means graphics/images retrieval is very important [2].
Table, block diagram, flowchart, plot, time chart, line drawing and pictorial information are different classes of 
figures in patent documents and the categorisation of these figures is illustrated in Fig 1[3]. Figures in patent 
documentation are usually binary images. They have no colour and texture [4]. Patent image retrieval (PIR) 
techniques are used to search similar patent images for query image in image database by analyzing the image 
contents. The techniques support users and patent experts to understand the content in the patent images and to find 
the related patent images. In a common patent image retrieval process, a user submits a patent image as an image 
query for finding similar images in patent database. The system extracts features either in a text or shape form or 
both of these forms from the submitted image (query image) and compares the features with all the images in patent 
database. Similarity is calculated based on the distance of the two images using their features vector. A common 
content-based image retrieval system is depicted in Fig 2.
Fig 1. The patent technical drawing [3] .
However, the literature review in Section 2 shows that there is very little published work in the field of patent 
image retrieval [3, 5] and most of the published works are focused on trademark search [6-9]. Therefore, there is a 
need to improve the current patent image retrieval (PIR) techniques and to meet users requirements [3, 4].
Figure
Technical 
drawing Flow Chart
PhotoDiagram Graph
WaveForm
TimechartResponse
Plane view Side view
Button view Top view
elavational view Sectional view
State Circuit
Block 
Diagram
Trellis
914   Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
Fig 2. A common content-based image retrieval.
This remaining paper is organised as follows: Section 2 presents the related work and discuss on potential, 
benefits and challenges existing Patent Image Retrieval (PIR) approaches; Section 3 presents the architecture of the 
proposed approach, Section 4 presents the discussions and comparisons and Section 5 concludes this paper.
2. Related work
In this section, we present the reviews and discussions on the recent patent image retrieval systems which are 
applicable to our research domain. 
Huet [10] experimented a patent image retrieval system in 2001. The experiments were performed on European 
Patent Office (EPO) database. They used relational skeletons also known as the Voronoi skeletonization as a main 
method to capture the geometric structure of the line patterns in images. The two main relational features: relational 
angle and position were calculated in addition to the calculation on the pairwise attributes. In their experiments, two 
retrieval algorithms were investigated: the Histogram and Graph based retrieval. In the Histogram retrieval 2-
dimentional histogram were employed to capture the local visual information. Angle attribute was captured by one 
dimension for 36 bins and position attribute was presented by other dimension for 12 bins. The Bhattacharya 
correlation was used for calculating the similarity between query and database images. In Graph retrieval, 6 nearest 
neighbor graphs were employed to increase the authority local visual information. The Hausdorff distance was used 
for calculating the similarity. The recall measure was employed for performance evaluation. 57 queries from 180 
patents were presented. The processing time was depending on size and complexity of the images but it was not 
reported in the paper. 
Tiwari [11] presented a content-based patent image retrieval system. The experimental was done on United State 
(US) database. The system consisted of two subsystems: The first subsystem created the feature vector and another 
Database Image
Query Image
Feature Extraction Feature Extraction 
Feature Vector Feature Vector
Similarity Measured 
Retrieve Result
User
D
at
ab
as
e
Su
bm
it
te
d 
im
ag
e
915 Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
one retrieved similarity between query image and existing images of database. The graphical interface was provided 
for user to search by keyword and image. Some pages contained more than one figure. Connected components were 
used to segment the figures as individually. The images were stored in database with separated patent number and 
page number. The edge orientation autocorrelogram (EOAC) was used for feature extraction step because it was 
independent of translation but computationally inexpensive. The rotation angle range for query image was between 
0 to 180 degrees scaling. 200 images were used for evaluation. Recall rate was 100% for 15 queries. The number of 
returned image was not reported. A total of 90 seconds processing time was reported for a query image in this 
system.
Vrochidis [12] experimented a content-based image retrieval framework in 2009. The experiments were 
performed on EPO patent database. The system provided a hybrid query: textual and visual query. In their work, the 
authors segmented the figures in each page manually and each segmented figure would be stored in database. The 
adaptive hierarchical density histogram (AHDH) was used as the main approach for extracting the features in query 
image and database. After noise removal and image normalization, the centroid of the image was calculated and 
then divided into four regions. The distribution of the black pixels in white region was calculated to form a feature 
vector. Finally feature vector was sent to a supervised classifier known as the support vector machine for 
classification. 110 query images were chosen for implementing the framework. Recall and prediction were 
calculated for performance evaluation. 77.4 % recall and 49 % prediction was reported. The processing time below 
10 seconds was reported.
Zhiyuan [13] presented a patent image retrieval system based on contour description matrix in 2007. The 
experiments were carried out on US patent database. The patent figures were converted to gray scale and then canny 
edge detection was applied to extract the counter information. The center of the shape was found to form a polar 
axis and edge points were converted using polar coordinate system. This approach divided a patent figure into grids 
ZLWK Ȗ DQGș UDQJH7KH UDWLo between edge points to all the points was calculated and formed the elements in a 
contour description matrix. The performance of the recalls and prediction rates were reported but the processing 
time and the number of images returned was not mentioned.
Csurka [14] presented a patent image retrieval system using Fisher vector in 2011. This system was implemented 
on flowchart, table, graph, drawing, etc. They used SIFT-like histograms and Gaussian mixture model (GMM) to 
build a visual vocabulary for patent images. Linear classification was employed to train the Fisher vector. Two 
strategies based on mean and max were used to compare the query image and images in database. The mean was the 
average distance of two images and max was the maximum distance of similarity on two images. The patent image 
retrieval system was evaluated on CLEF-IP 2011 track. The max strategy was found better than the mean strategy 
on 211 queries. The authors also proposed a text based patent retrieval system based on weighted late fusion 
technique to combine with the visual approach. The best patent retrieval system was obtained from the combination 
of these two approaches.
We summarized PIR system in Table 1 according to the existing approaches in patent image retrieval domain. The 
relational skeleton approach proposed by Huet [10] was sensitive to the image translation, rotation and scaling 
because the use of the line segments. The approach by Tiwari [11] was not sensitive to image scaling and translation 
but the processing time was 90 seconds for a query image which was a major problem of the system. The drawback 
of the AHDH which was used by Vrochidis [12] for patent image retrieval on EPO database is geometrical 
invariance. The approach presented by Zhiyum [13] was not sensitive to rotation and scaling but the processing time 
was not reported. According to the review in [5], the contour description matrix had higher processing time when 
high resolution images were used. The run time of the system again was not reported in Csurka [14] but the fisher 
vector achieved 91% accuracy and thus it was the best method among the existing PIR systems. SIFT was only used 
in the Fisher vector among all the existing approaches [5]. Since there were only a limited number of feature 
extraction approaches being experimented in the patent image retrieval problem, there is a need to further 
investigate other advanced feature extraction techniques to improve the current image retrieval system. Thus, we 
proposed an improved approach in the area of patent image retrieval which would be presented in the next section of 
this paper. 
916   Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
Table 1.  The comparison table of existing patent image retrieval system.
Author Date Dataset
Feature Extraction 
Methodology
Result
Time
(sec)
Huet 2001 EPO Relational skeleton
Histogram 36.46
Graph 52.16
-
Tiwari 2004 US
edge orientation 
autocorrelogram
Recall rate 32% 90
Vrochidis 2009 EPO
adaptive hierarchical 
density histogram
Prec:90.69%
Recall:71.80
F-score:78.95%
-
Zhiyum 2007 US
contour description 
matrix precision-recall curve -
Csurka 2001 - SIFT 91% accuracy -
3. Proposed patent image retrieval approach
From the review in Section 2, SIFT algorithm was found to be the best feature extraction method. However, SIFT 
algorithm was only limited to four transformation parameters namely rotation, translation, expansion and 
contraction [15] and there was an improved Affine-SIFT being introduced. This Affine-SIFT (ASIFT) was 
introduced by [15] to perform full invariant objects matching in images. ASIFT treats the camera axis orientation 
and angle defining (latitude, longitude) left over by the SIFT algorithm [15, 16]. Hence, we propose Affine-SIFT 
(ASIFT) algorithm to find matched objects in patent images. When a user submits a patent image query, the 
proposed approach extracts features based on the Affine-SIFT algorithm. The extracted features are stored in a 
feature vector and a similarity measure is applied to find relevant images. The proposed approach is depicted in Fig 
3.
Query image Database image
Feature vector
ASIFT (affine SIFT)
feature extraction
Feature Extraction
ASIFT (affine SIFT)
feature extraction
Similarity Measured
Feature vector
Feature distance 
between query and 
database images
Feature Extraction
Retrieve Result 
Fig 3. The proposed approach for patent image retrieval.
4. Discussion and comparison 
Patent image search is normally accomplished manually to find any image matches. This manual approach is 
time-consuming and it has poor rate of finding the right information. Image processing-based technique is a better 
way to overcome this limitation and thus content-based patent image retrieval (CBPIR) is used to identify and 
finding matches in patent images. From literature, image matching techniques are found to be extensively utilised
for CBIPR systems. Some of the techniques which known as the affine transformation algorithms are popularly used 
917 Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
in image matching techniques due to the invariant features capabilities. The typical features include scale, rotation 
transformations invariants and they are effective to find similarity between images in patent content. Among these 
affine transformation algorithms, Affine-SIFT (ASIFT) is the latest improved version and it is more robust 
compared to other algorithms. Table 2 shows the comparison of SIFT and ASIFT to verify the efficiency of ASIFT 
algorithm based on [15, 17]. Since Affine-SIFT image matching technique has the potential to enhance the matching 
process, we proposed a content-based patent image retrieval (CBPIR) based on ASIFT algorithm in this paper.
Table 2. Comparison of image matching techniques.
Evaluation parameter SIFT-based algorithm
Affine-SIFT algorithm (in 
proposed system)
Processing time
Relatively longer time for 
feature extraction process 
Faster in feature extraction process
Performance
Efficient but partial invariant 
for scaling, rotation and 
translation
Efficient and full invariant for 
scaling, rotation and translation
5. Conclusion
Patent Image Retrieval (PIR) is used to search similar patent images from image database based on an image 
query. However, there are shortages of published works in patent image retrieval system and thus there is a need for 
more research in this patent image retrieval domain. This paper is intended to propose a better patent image retrieval 
system with the feature extraction step as the main focus in our system. We proposed a patent image retrieval 
approach using a fully invariant affine image comparison technique which is known as ASIFT.  This algorithm is 
not sensitive to scaling, rotation and transformation and thus the proposed patent image retrieval approach has the 
potential to get promising results. 
Acknowledgements
The authors would like to thank Universiti Kebangsaan Malaysia for sponsoring this research by using the 
funding of the research project ERGS/1/2011/STG/UKM/02/44.
References
[1] Lupu, M., et al. Patent images-a glass-encased tool: opening the case. in Proceedings of the 12th 
International Conference on Knowledge Management and Knowledge Technologies. ACM; 2012.
[2] Zhou, Y. and C.L. Tan. Chart analysis and recognition in document images. in Document Analysis and 
Recognition. Proceedings. Sixth International Conference on IEEE;  2001.
[3] Vrochidis, S., et al., Towards content-based patent image retrieval: A framework perspective. World Patent 
Information 2010; 32(2): p. 94-106.
[4] Hanbury, A., et al. Patent image retrieval: a survey. in Proceedings of the 4th workshop on Patent 
information retrieval. ACM;  2011.
[5] Naeem Bhatti, A.H., Image Search in Patents-a Review. International journal on document analysis and 
recognition; 2012.
[6] Lew, M.S., Principles of visual information retrieval. Springer Verlag; 2001.
[7] Jain, A.K. and A. Vailaya, Shape-based retrieval: A case study with trademark image databases. Pattern 
918   Marzieh Mogharrebi et al. /  Procedia Technology  11 ( 2013 )  912 – 918 
recognition 1998; 31(9): p. 1369-1390.
[8] Kim, Y.-S. and W.-Y. Kim, Content-based trademark retrieval system using a visually salient feature.
Image and Vision Computing 1998; 16(12): p. 931-939.
[9] Schietse, J., J.P. Eakins, and R.C. Veltkamp. Practice and challenges in trademark image retrieval. in 
Proceedings of the 6th ACM international conference on Image and video retrieval. ACM;  2007.
[10] Huet, B., et al. Relational skeletons for retrieval in patent drawings. in Image Processing, 2001. 
Proceedings. 2001 International Conference on. IEEE; 2001
[11] Tiwari, A. and V. Bansal. Patseek: Content based image retrieval system for patent database. in 
Proceedings of International Conference on Electronic Business, Beijing, China; 2004.
[12] Vrochidis, S., A. Moumtzidou, and I. Kompatsiaris, Concept-based patent image retrieval. World Patent 
Information; 2012.
[13] Zhiyuan, Z., Z. Juan, and X. Bin. An outward-appearance patent-image retrieval approach based on the 
contour-description matrix. in Frontier of Computer Science and Technology, 2007. FCST 2007. Japan-
China Joint Workshop. IEEE; 2007.
[14] Csurka, G., J.-M. Renders, and G. Jacquet. XRCE’s participation at patent image classification and image-
based patent retrieval tasks of the Clef-IP. In: CLEF (Notebook Papers/Labs/Workshop); 2011.
[15] Yu, G. and J.-M. Morel. A fully affine invariant image comparison method. In: Acoustics, Speech and 
Signal Processing. ICASSP. IEEE International Conference; 2009. 
[16] Morel, J.-M. and G. Yu, ASIFT: A new framework for fully affine invariant image comparison. SIAM 
Journal on Imaging Sciences 2009; 2(2): p. 438-469.
[17] Oji, R., An Automatic Algorithm for Object Recognition and Detection Based on ASIFT Keypoints. arXiv 
preprint arXiv:1211.5829; 2012.


Retrieval System for Patent Images 

file:///data/core-remote/dit/data/elsevier/pdf/d79/aHR0cDovL2FwaS5lbHNldmllci5jb20vY29udGVudC9hcnRpY2xlL3BpaS9zMjIxMjAxNzMxMzAwNDI5NQ==.pdf

Retrieval System for Patent Images

Abstract

Similar works

Full text

Available Versions

Elsevier - Publisher Connector

Elsevier - Publisher Connector