ON-LINE CHINESE CHARACTER RECOGNITION B Y Jian-Zhuang Liu A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DIVISION OF ELECTRONIC ENGINEERING T H E CHINESE UNIVERSITY OF HONG KONG JULY 1 9 9 7 Number: 9908127 LfMI Microform 9908127 Copyright 1998, by UMI Company. All rights reserved. This microform edition is protected against unautliorized copying under Title 17 United States Code. Copyright © 1997 by Jian-Zhuang Liu All right reserved. To my wife and my son, and to the memory of my parents 111 Acknowledgement It is my pleasure to have the opportunity to thank all the people who have given me help during the course of this thesis. First, I would like to express my sincere gratitude to my supervisor, Associate Professor Wai Kuen Cham, for his informal guidance, useful suggestions, and enthusiastic discussions. I am also very grateful to Professor Wei Xin Xie, the President of Shenzhen University, for introducing me to Professor Cham. He was ray advisor and gave me much help in my research and teaching when I was working in Xidian University. Thanks to Professor Hung Tat Tsui for his kindly and useful suggestions in my current and future research. Thanks to Mr. Ding Wah Chan for his very good technical support in the laboratory of image processing and computer vision. Thanks to Professor Chorkin Chan of Hong Kong University, Associate Pro-fessor Michael Ming Yuen Chang, and Associate Professor Kin Hong Wong, for serving as the examination committee members of my thesis. Many thanks to my colleagues, Dr. Yingli Tian, Dr. Yibing Yang, Wai Tat Fu, Fu Wing Tse, Man Ching Auyeung, Hau Lai Ho, Guoliang Fan, Po Ming Wong, and Shu Yuk Yeung, who provided handwritten Chinese characters for vii testing my on-line Chinese character recognition methods. I am deeply indebted to my parents who always encouraged me in my studies when they were alive. Last but not least, I would like to give my special thanks to my wife, Qing Meng for her patience, constant encouragement and support. She brought up our son in Xian, China while I was studying in Hong Kong. Part of this research was supported by the Hong Kong RGC Earmarked Research Grant CUHK67/92E. Abstract The existing methods and commercial products for on-line Chinese character recognition (OLCCR) are not satisfactory when there are stroke order and stroke number variations. This thesis presents several methods for achieving better per-formance of OLCCR. We address three aspects: preprocessing of input hand-writing, representations of Chinese characters and recognition methods. First, we deal with the preprocessing problem. To facilitate the recognition of the types of strokes and segments, an input stroke is represented with a polyline. A method for recognizing the types of strokes with more than two segments is proposed by stroke chain code string matching. Some rules are presented to detect most of frequently-occurred connected strokes and then delete the extra segments in such strokes. Next, we formally define complete relational graphs and distances for mea-suring the similarity between two graphs. With such graphs, we propose stroke-based and segment-based spatially-temporally relational representations for Chi-nese characters, using novel "don't care", "should" and "must" relational fea-tures. Recognition methods are the key to OLCCR. We develop three methods in this thesis. The first one is a state space search method. We formulate the graph vii matching as a state space search problem. To obtain good search efficiency, we use the A* algorithm to perform heuristic search and propose three schemes to speed up the A*: utilize a heuristic function to make the A* expand fewer nodes in a search tree; employ a tree pruning operation to let the A* avoid searching the nodes that have very little chance to be located in the optimal path in a tree; introduce two new criteria, together with the original one, to stop the A* by using the monotone of the evaluation function of the A*. In the two-layer assignment method, finding segment correspondences be-tween two characters is formulated as an assignment problem (in layer 2)’ which can be solved by the Hungarian method. The cost matrix of this assignment problem is derived by the assignment problems in layer 1. To save computa-tional time, a lower bound estimate and the geometric position features of model characters are used to reduce the complexity of the method from O(n^) to 0{n^). The third method is a fast string matching one, which incorporates the ge-ometric position constraints of strokes (or segments) of Chinese characters into Wagner and Fischer's string matching algorithm. To allow more stroke order de-viations for some characters, using two or more strings to represent one of these model characters is a feasible way. In this case, we present a scheme to save com-putational time, by combining two or more separate networks into one and em-ploying a dynamic-programming procedure to solve the shortest path problem. We also make comparisons of the first method with several other methods published recently, and find that our method is very promising. When segments of Chinese characters serve as primitives, the first two methods are stroke order and number free. The last one is stroke number free and runs much faster but is not stroke order free. v i i List of Figures 1.1 The major contributions of the thesis. A line connecting two boxes indicates that there are some relationships between them. 8 2.1 Three input strokes 14 2.2 Recursive procedure of stroke fitting 15 2.3 Stroke approximation, (a), (d) Input strokes, (b), (e) Polylines after fitting the input strokes in step 1. (c), (f) Polylines after the merging processing of the polylines in (b) and (e) 19 2.4 Angle intervals for strokes 1-5 21 2.5 (a) Two input strokes, (b) Corresponding polylines 21 2.6 Stroke type recognition structure 22 2.7 (a) Input stroke, (b) Polyline of (a), (c) Normalized polyline of (b). (d) Chain code string of (c) 23 2.8 8 directions of chain codes 24 2.9 (a) Recognition rate i? as a function of (^1,^2)- (b) The surface obtaining from another viewpoint 29 2.10 A curve and its fitting line. The curve presents the track where Riki, ^2) takes the maximum value 30 Vl l l 2.11 Stroke recognition experiments. On the left of the arrows are a set of input strokes. Their classification results: corresponding standard strokes and stroke types, are indicated on the right. . . 31 2.12 Four examples of misclassification 32 2.13 Several examples where bold segments “ \ “ appear in connected strokes 34 2.14 (a) Examples of connected strokes leading to extra segments, (b) Examples of connected strokes not leading to extra segments. . . 35 2.15 Examples of segment processing. Columns (a) and (d) are input Chinese characters. Columns (b) and (e) are the segment process-ing results. Columns (c) and (f) show the corresponding model characters 38 3.1 4-layer structure of a Chinese character 41 3.2 An example of relational graph representation of a scene, (a) A scene, (b) The corresponding relational graph 43 3.3 (a) A Chinese character, (b) Corresponding complete relational graph, (c) Relation matrix of the graph 44 3.4 Two simplified forms of Fig. 3.3(b) 45 3.5 Examples of edit operations, (a) Gi => G^ via 721 — n[ and a 4 d. (b) Gi Gz via ni —> A,a -> A and 6 — A. (c) Gi G4 via A ^ 714, A d, A e and X — f 47 3.6 Two node mappings, (a) fi^{ni) = A, = ns, f ^ i r i z ) = = rij. (b) / V(7ll) = A, /iV(n2) = fN(jl3) = "5’ /jv(A) = rzg, /yv(A) = 717 51 vii 3.7 A set of characters with their strings of decomposed strokes. A number near a stroke indicates the order of the stroke when the character is written 57 3.8 Stroke-based spatially relational representation, (a) A Chinese character, (b) Complete relational graph representing the char-acter. (c) Corresponding spatial relation matrix. A point on or near a stroke indicates the geometric center of the stroke. Dif-ferent strokes are labeled with different numbers. A node of the graph represents a stroke by containing its number and type (the lower number) 61 3.9 Two model characters and their handwritten styles 63 3.10 Segment-based spatially relational representation, (a) A Chinese character, (b) Complete relational graph representing the char-acter. (c) Corresponding spatial relation matrix 64 3.11 Stroke-based spatially-temporally relational representation, (a) A Chinese character, (b) Complete relational graph representing the character, (c) Corresponding spatial-temporal relation matrix. 66 3.12 Segment-based spatially-temporally relational representation, (a) A Chinese character, (b) Complete relational graph representing the character, (c) Corresponding spatial-temporal relation matrix. 67 4.1 A state space tree for matching between Gi and Gj. Symbols “ ” “•” and “•” denote the initial state, middle states and goal states, respectively. 82 vii 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 A search tree, u is a middle node and t; is a goal node that can be reached from u 89 A search tree, u is a middle node, w x and y are successors of u. 90 (a) A bipartite graph 5 = (Vi U Vn^E), where Vi = N'; U A“ I Ail = \N'Jl V2 = JV U Aj, and \Aj\ = (b) A matching in the bipartite graph 94 A model character (a) and its handwritten version (b) 97 A partial tree for matching from the character in Fig. 4.5(a) to the character in Fig. 4.5(b) 98 (a) A Chinese character and the smallest rectangle ABCD sur-rounding the character, (b) Geometric illustration of ^0-3(2). (c) 8 directions 99 A partial tree obtained by pruning the tree in Fig. 4.6. All possible nodes at depths 1 and 2 are shown 102 Example 1 for showing the nondecreasing estimated / values. (a) A model character and one of its handwritten characters. (b) Estimated f values. The node numbers denote the sequence of the nodes expanded by the A• when the A* searches a tree for the optimal matching between the two characters. The A* termi-nates after expanding 14 nodes. The matching distance between the two characters is 7 106 vii 4.10 Example 2 for showing the nondecreasing estimated / values. (a) A model character and one of its handwritten characters. (b) Estimated f values. The node numbers denote the sequence of the nodes expanded by the A*, when the A* searches a tree for the optimal matching between the two characters. The A* termi-nates after expanding 16 nodes. The matching distance between the two characters is 15 107 4.11 Example 3 for showing the nondecreasing estimated f values. (a) A model character and the input character in Fig. 4.10(a). (b) Estimated f values. The node numbers denote the sequence of the nodes expanded by the A•’ when the A* searches a tree for the optimal matching between the two characters. The A* termi-nates after expanding 36 nodes. The matching distance between the two characters is 57 108 4.12 Some model Chinese characters for testing the stroke-based recog-nition method 113 4.13 Some test characters having correct stroke numbers, together with their corresponding models 114 4.14 Some test characters having one or two connected strokes, to-gether with their corresponding models 115 4.15 Some test characters written each having 4 to 7 strokes, together with their corresponding models 118 4.16 Some test characters written each having 1 to 3 strokes, together with their corresponding models 119 vii 4.17 (a) A Chinese character whose strokes are labeled with the num-bers that indicate their standard order of writing, (b) A hand-written version of (a) which has many stroke order deviations. . 120 4.18 Some handwritten characters that the segment-based method can-not recognize. Their corresponding model characters are also shown. 121 5.1 (a) A complete bipartite graph, (b) A complete matching of a bipartite graph corresponding to the assignment in (5.5) 129 5.2 A model character (a) and its handwritten character (b) 131 5.3 A bipartite graph formulation of segment correspondences be-tween the two characters in Fig. 5.2 131 5.4 A Chinese character (a) and its very similar handwritten style (b). 134 5.5 The structure for obtaining segment correspondences between character 1 and character 2 137 5.6 Some test data in the experiments 143 6.1 A trace (T, Si, S2) 149 6.2 A network for calculation of the distance between a string of length m and a string of length n 152 6.3 A network for calculation of the distance between two strings in Fig. 6.1. The bold path P corresponds to the trace T = {(1,1),(2,3).(5,4),(7,5)} 153 6.4 Some test characters each, having less than three connected strokes. 159 6.5 A model character (a) and a set of its input handwritten charac-ters (b)-(j) having different stroke orders 160 Xl l l 6.6 Some test characters written freely, all of which are recognized correctly 163 6.7 (a) A model character with the numbers labeling its standard stroke order, (b) The same character as (a) but with different stroke order 164 6.8 (a) Combining strings S[ and S'2 together, (b) An input string R. 165 6.9 (a) A network for calculating R). (b) A network for cal-culating 6{S2, R). (C) A network obtained by combining (a) and (b) 166 6.10 (a) Combining strings 5i, S2, S3 and S4 together, (b) An input string R 167 6.11 Four networks (a)—(d) for calculating 6(SI, R), 5(82, R), S(S3, H), aud S(S4,Ii), respectively. 168 6.12 A network obtained by combining networks (a)-(d) in Fig. 6.11. 169 xvii List of Tables 2.1 Standard strokes 13 2.2 18 model strokes 20 2.3 14 Multi-segment strokes used to determine whether an input stroke with more than two segments belongs to one of them. . . 36 3.1 18 model strokes 70 3.2 Costs associated with stroke type correspondences 71 3.3 Definition of the function rj 72 3.4 Costs associated with segment type correspondences 74 4.1 Numbers of nodes generated by the A* for three matching exam-ples each in three cases 116 5.1 Matrix [ yl9x9 for estimating the cost matrix [cij]9x9 136 5.2 Cost matrix [c‘j]9x9 for finding the segment correspondences be-tween two characters in Fig. 5.4 136 5.3 Performance comparison of three methods 144 vii Contents 1 Introduction 1 1.1 Importance of on-Line Chinese Character Recognition 1 1.2 Review of Recent Studies of the Subject 3 1.3 Outline of the Thesis 7 2 Preprocessing 11 2.1 Introduction 11 2.2 Stroke Approximation with Polylines 13 2.3 Stroke Type Recognition 20 2.3.1 Normalization and Chain, code extraction 22 2.3.2 Chain Code String Matching 24 2.4 Segment Extraction and Processing 33 2.5 Summary 37 3 Relational Graph Representations of Chinese Characters 40 3.1 Introduction 40 3.2 Relational Graphs and Distance Measures 42 3.2.1 Complete Relational Graphs 42 vii 3.2.2 Edit Operations on Graphs 45 3.2.3 Distances between Two Graphs 48 3.3 Representations of Chinese Characters 56 3.3.1 Stroke-Based Spatially Relational Representation . . . . 56 3.3.2 Segment-Based Spatially Relational Representation . . . 63 3.3.3 Spatially-Temporally Relational Representations 64 3.4 Assigning Costs to Node and Arc Correspondences 68 3.4.1 Assigning Costs for Stroke-Based Relational Graph Match-ing 69 3.4.2 Assigning Costs for Segment-Based Relational Graph Match-ing 74 3.5 Summary 75 A State Space Search Method 78 4.1 Introduction 78 4.2 State Space Formulation of the Graph Matching 80 4.3 The A' Algorithm 84 4.4 Schemes for Speeding up the A" Algorithm 87 4.4.1 A Lower Bound Estimate 87 4.4.2 A Tree Pruning Strategy 97 4-4.3 Criteria for Stopping the A' Algorithm 103 4.5 Experimental Results I l l 4.5.1 Stroke-Based Recognition I l l 4.5.2 Segment-Based Recognition 116 x v i i 4.6 Comparisons of the Segment-Based Recognition Method with Sev-eral Other Studies 121 4.7 Summary 125 5 A Two-Layer Assignment Method 127 5.1 Introduction 127 5.2 The Assignment Problem 128 5.3 A Two-Layer Assignment Formulation of on-Line Chinese Char-acter Recognition 130 5.3.1 Finding Segment Correspondences between Two Characters 130 5.3.2 Calculating the Similarity of Two Characters 138 5.4 Two Complexity Reduction Schemes 139 5.4.1 A Lower Bound Estimate 140 5.4.2 Geometric Position Constraints 141 5.5 Experimental Results 142 5.6 Summary 145 6 A Fast String Matching Method 146 6.1 Introduction 146 6.2 The WFSM Algorithm 147 6.3 Application of the WFSM Algorithm to on-Line Chinese Charac-ter Recognition 152 6.4 Experimental Results 157 6.4.1 Stroke-Based Recognition 157 6.4.2 Segment-Based Recognition 161 6.5 Extension of the String Matching Method 164 XVlll 6.6 Summary 170 7 Conclusions and Suggestions 172 7.1 Contributions of this Thesis 172 7.2 Suggestions for Further Research 178 Bibliography 183 vii Chapter Introduction 1.1 Importance of on-Line Chinese Character Recognition Recently, rapid development of computer techniques has made personal com-puters (PCs) cheap enough for family use. To enter text into a computer, using a keyboard is faster than handwriting for small-alphabet languages such as En-glish, but it is cumbersome for large-alphabet Chinese. Hundreds of millions who use Chinese in their daily life are bothered all along by the input of Chinese characters into computers, except those who have taken a lot of time to learn by rote some input methods that encode Chinese characters. Therefore, a good on-line Chinese character recognition (OLCCR) system will provide a friendly-interface for the use of Chinese and popularize PCs in China and some other areas. In addition to computers, some products, such as portable electronic diaries, electronic Chinese-English dictionaries, multi-functional telephones and Chapter 1 Introduction simple Chinese typewriters, may also require to be able to recognize on-line inputted Chinese characters. There are two research fields of handwritten Chinese character recognition: on-line recognition and off-line recognition. In this thesis we only discuss the former study. On-line recognition means that a machine recognizes the input characters while one writes on a digitizer with a stylus pen. Off-line recognition is performed after the handwriting is completed, generally with a scanner con-verting the image of handwriting on paper into a bit pattern. On-line devices can capture the temporal information of handwriting, such as the number, order and direction change of strokes of a Chinese character. Thus, for recognizing the Chinese characters written in a similar degree of distortion, on-line recogni-tion is easier than off-line recognition. By the way, another advantage of on-line recognition is that a user may participate in the recognition process after the computer selects a small set of possible candidates for an input character. Recognition of handwritten Chinese characters is considered as a hard prob-lem because of large categories, complex structure, and widely variable and many similar shapes of Chinese characters. Although great progress has been made in OLCCR since the 1970's [36 88, 97), a number of researchers are still involved in this topic for achieving better performance of OLCCR. From the review in the next section, we can see that the existing methods are not satis-factory. Researchers hope to develop better algorithms which are stroke order and stroke number free, and can run on general computers (e.g. PCs) within an acceptable computational time. Now commercial products for OLCCR are available but their performance still needs improving, because they require that input Chinese characters should be written both in the block (not cursive) style 2 Chapter 1 Introduction and basically according to their standard stroke orders (in other words, they allow only very few and common stroke order deviations). 1.2 Review of Recent Studies of the Subject A lot of approaches to OLCCR have been proposed since 1970's, most of which may be classified into one of the techniques: transform, decision tree, string matching, syntactic-semantic analysis, radical (component) decomposition and graph matching [21, 36, 60, 88 97]. There are so many approaches that we cannot mention them one by one. Thus, in the following we just give the review of recent studies, which in general, have better performance than the methods published earlier. For the reader who wants to know more about this topic, the survey papers [36 88, 97] are recommended. To evaluate an OLCCR approach, we consider its three aspects: tolerance of stroke order variations, tolerance of stroke number variations, and running time. In [56], Lin et al. proposed a deviation-expansion model to represent Chi-nese characters, and dynamic programming is used to carry out the character matching. Their approach is stroke-based and in essence a string matching one. It requires that an input character should not have more than one stroke num-ber variation and more than two connected strokes. The running time of their algorithm is 0(2"), where n is the stroke number of an input character. Chou et al. [21] extended the above model to a segment-based deviation tree, which is also a string matching one and is not stroke order free. It cannot tolerate more than two stroke order deviations. Their stroke-segment preprocessing scheme makes the approach allow more stroke number deviations. The computational 3 Chapter 1 Introduction complexity is also 0(2"). In [17], Chen et ad. developed a stroke-sequence decision tree to represent Chinese characters and employed stroke positions to calculate the similarity between two characters. The approach is not stroke order and number free. It cannot handle the input characters with stroke number deviations more than one. Tsay and Tsai [92] used attributed string matching by split-and-merge for on-line Chinese character recognition. The proposed method can recognize cursive characters but imposes the constraint of correct stroke orders on them. The authors suggested that their approach could be used to design a writer-dependent system. In [22], Chou and Tsai proposed a discrete iteration scheme to solve the OLCCR problem. The features used to measure the similarity between two characters include lengths, orientations and locations of segments. Their method is not stroke order and number free. The provided test characters are in block style and almost have no connected strokes. In [19], a hierarchical deformation model is proposed to describe the defor-mation of on-line cursive Chinese characters. An elastic matching algorithm and a constrained parabola transformation are used to find the correspondences of strokes between two characters. The method requires that input characters should be written in correct stroke orders. The algorithm is very computation-ally intensive. The time for recognizing a character is 4.2 seconds on a Sun 4 SPARC workstation when there are only 20 model characters. In [40], Hsieh et al. employed a greedy algorithm for bipartite matching to carry out the recognition. The method is segment-based and stroke order free. The provided test characters are neatly written, some of which have one or two connected strokes. Their algorithm needs large amounts of computation. Its 4 Chapter 1 Introduction running time is 0(max{n®, m®}), where n and m are the segment numbers of two characters under matching. The average time for recognizing a character is 39 seconds on a Sun workstation when there are 452 model characters. Chen and Lee [16] proposed a fuzzy attribute graph representation for Chi-nese characters. They used a set of segment intersection features to describe only the relations between segments within the same components. A maximum clique finding algorithm is employed to perform the graph matching. Two prob-lems exist if the maximum clique finding algorithm is used: (1) it is NP-complete and so is time consuming; (2) the thresholds, which are utilized to build an asso-ciation graph describing all possible compatible mappings between two graphs, may eliminate any possibility of including a given pair of nodes in the final clique, resulting in incorrect recognition [26]. The average time for recognizing a character is 2 seconds on a Sun SPARC-II workstation under the conditions: (1) there are 650 model characters each with a stroke number between 1 and 12, and (2) a preclassification is employed. Obviously, when their model base is enlarged, the algorithm is too slow to use. The method is stroke order free, but its tolerance of stroke number variations is unknown since no test data or characters are provided. In summary, the above methods are not good enough. Only the last two are not stroke order free, but they require large amounts of computational time. In addition, the method in [40] can only recognize neatly written characters, some of which have one or two connected strokes. The tolerance of stroke number variations in [16] is unknown since no test characters are given. The last method 16] uses graphs to represent Chinese characters. Besides it and our work, there ire several off-line recognition methods that also use graphs to represent Chinese 5 Chapter 1 Introduction characters [13’ 18 68]. Because they are related to part of our work in Chapters 3 and 4 we also give a brief review of them here. In the application of graph representations and graph matching to Chinese character recognition, a computational problem arises due to the large cate-gories of Chinese characters and the inherent combinatorial explosion of graph matching. In order to save computational time, Chen et ai. [18] and Lu et al. 68) used a two-layer graph to represent a Chinese character. In the first layer, nodes describe components (or radicals) of a Chinese character and arcs describe the relations among these components. In the second layer, each component of the first layer is represented by a graph, in which nodes and arcs represent the strokes and the relations among these strokes of a component, respectively. This strategy results in several smaller graphs for each Chinese character, so match-ing time can be reduced. However, a new problem of how to correctly group the strokes of a Chinese character into its components arises. The wide handwriting variations and connected strokes make it very difficult to extract components of Chinese characters at a high rate of success. In [18], a relaxation matching algorithm is used to carry out the graph matching but it is still time consuming. Although there are only 24 models (10 numerals and 14 Chinese characters), the method needs 30 seconds to recognize an input character on a PC/AT compat-ible. In [68], exhaustive search with some search rules are employed to perform the graph matching, but the recognition time required is not reported. Chan and Cheung [13] used character graphs to represent handwritten Chi-nese characters and radical graphs to represent model radicals. The recognition of a character is completed from the radicals found by matching radical graphs with the character graph. A NP-complete maximum clique finding algorithm is 6 Chapter 1 Introduction employed to perform the graph matching, which is expected to be computation-ally intensive (no recognition time is reported). As we have mentioned above, reliably extracting radicals from the characters with stroke type distortion or connected strokes is a difficult work. Therefore, these three methods are only suitable for recognizing neatly written characters. The test characters provided in [13 18, 68] support this conclusion. In Chapter 3 we propose complete relational graph representations of model and input Chinese characters. In a graph, nodes denote primitives (strokes or segments) and their types, and arcs describe the relations between any two primitives. Different from the methods in [13, 18 68], we do not need to extract the radicals of Chinese characters. Also different from the method in [16], we use the relations between any two primitives in our representations, which provide much information that is very beneficial to the graph matching procedures (see Section 4.4.3). The features to describe the relations between primitives in our method are also different from those in [13 16, 18, 68]. In addition, we use the state space search and the A* algorithm to carry out the graph matching. With the proposed pruning strategy and stopping rules, our matching method is fast enough for practical application. 1.3 Outline of the Thesis In the present thesis, we will address three aspects of OLCCR: preprocessing of input handwriting, representations of Chinese characters and recognition meth-ods. The major contributions are shown in Fig. 1.1. The thesis is organized as follows. Chapter 1 Introduction Figure 1.1: The major contributions of the thesis. A line connecting two boxes indicates that there are some relationships between them. In Chapter 2 we develop several preprocessing approaches to OLCCR. First, we approximate input strokes with polylines by using an efficient splitting and merging algorithm, to facilitate the recognition of strokes and segments. Sec-ondly, we propose a method for identify the types of strokes each with more than two lines, which consists of three procedures: normalization of strokes, ex-traction. of stroke chain code strings, and matching between input code strings and model code strings. The method can be used not only in stroke-based OLCCR but also in segment-based OLCCR. Thirdly, we present some rules to detect frequently-occurred connected strokes and then delete the extra segments in such strokes. In Chapter 3, we formally define the complete relational graphs and the 8 Chapter 1 Introduction distances for measuring the similarity between two graphs. With such graphs, we propose several relational representations for OLCCR. The representations incorporate the human knowledge of Chinese characters and can reflect their features well (except some very similar character pairs). The novel "don't care", "should" and "must" relational features allow us to represent unstable, stable and very stable primitive relations conveniently. We also deal with assigning costs to node and arc correspondences for calculating the graph matching dis-tances. In Chapter 4, we formulate the graph matching as a state space search prob-lem. The optimal matching between two graphs is equivalent to finding the best goal node in a search tree. To obtain good search efficiency, we use the A* algorithm to perform heuristic search and propose three schemes to speed up the A*: (1) a heuristic function is defined to make the A* expand fewer nodes in a search tree; (2) a tree pruning strategy, which employs the geometric position features of strokes (or segments) of Chinese characters to prune a search tree, is presented to let the A* avoid searching the nodes that have very little chance to be located in the optimal path from the initial node to the best goal node in a tree; (3) two new criteria are proposed to stop the A* by utilizing the mono-tone of the evaluation function of the A*. To demonstrate the performance of the method, we also give the experimental results and make some comparisons between our method and other studies published recently. In Chapter 5, we propose a two-layer assignment method for OLCCR. Find-ing segment correspondences between two characters is formulated as a weighted bipartite graph minimum cost complete matching problem, which corresponds to an assignment problem (in layer 2) and can be solved by the Hungarian method. 9 Chapter 1 Introduction The cost matrix of this assignment problem is derived by the assignment prob-lems in layer 1. To save the computational time, a lower bound estimate and the geometric position features of model characters are used to reduce the com-plexity of the method from 0(n^) to O(n^), where n = max{r2i, 712} and ni and 712 are two segment numbers of two characters under matching. We also present some experimental results to show the performance of the method. In Chapter 6, we propose a fast string matching method, which incorpo-rates the geometric position constraints of primitives of Chinese characters into Wagner and Fischer's string matching algorithm. Some experimental results are given. In order to allow more stroke order deviations for some characters, we suggest to use two or more strings to represent one of these model characters, and present a scheme to save computational time. It combines two or more separate networks into one and employs a dynamic-programming procedure to solve the shortest path problem. Finally, in Chapter 7 we summarize the contributions of the thesis and discuss the directions for future research. 10 Chapter 2 Preprocessing 2.1 Introduction A pattern recognition method, in general, requires a set of features of objects to represent and recognize the objects. The procedures to obtain the features prior the classification phase are called preprocessing. For on-line Chinese character recognition, some of these features — stroke (segment) numbers, orders, coordi-nates, lengths and directions, radical relations, stroke (segment) relations, and so on — are employed to do the recognition job. A tablet digitizer can capture on-line input data while a user writes on it with a stylus pen. These data may contain different types of noise, arising from the limited accuracy of the tablet, digitizing process, erratic hand motion, etc. Therefore, a few common techniques have been used to reduce the noise [88 . Smoothing averages a point with its neighbors [3’ 4 42 43, 87, 102]. Filtering is utilized to eliminate duplicate points and to reduce the number of points 3, 4, 37, 42 52]. Wi ld point correction can replace or eliminate occasional Chapter 2 Pre-processing spurious points [37, 72’ 87]. Dehooking eliminates hooks that occur at the beginning and the end of strokes [66, 87). In addition to these, normalization that adjusts the character size to a standard is required in many methods [3 4 16’ 40 52, 72 94, 95 97]. Not all these preprocessing approaches are necessary for a recognition sys-tem, and some new preprocessing techniques may be more suitable for different methods. In this thesis, we propose two graph representations of Chinese char-acters. The former is stroke-based, i.e., the primitives are strokes, and the latter is segment-based, i.e., the primitives are segments. The nodes of a graph denote the strokes (or segments) of a character, and the arcs indicate the relations be-tween any two strokes (or segments) of the character. Before recognition phase, we have to first extract the strokes (or segments) of an input Chinese character as efficiently and reliably as possible, and then construct its graph representation. A stroke is defined as the writing from pen down to pen up when one writes on a digitizer with a stylus pen. A Chinese character consists of a set of standard strokes, and each standard stroke consists of from one to four segments, as shown in Table 2.1.1 Segments are the smallest units that compose a Chinese character. On-line devices can capture the temporal information of the writing, such as the number, order and direction change of strokes. To conveniently identify strokes and segments of an input character, we represent each stroke with straight lines. A line splitting and merging method for reaching this goal is presented in Section 2.2. The stroke type recognition method is proposed in Section 2.3. Section 2.4 gives some schemes for obtaining the segments that ^The standard stroke “ J “ is not given in the table. It is diflficult to define the segment number of this stroke. However, since it is very similar to the standard stroke “ J “ in handwriting, we consider they belong to the same type in stroke identification. 12 Chapter 2 Pre-processing Table 2.1 Standard strokes. One segment strokes z \ / Two segment strokes 7 < \ J Three segment strokes V " I ^ Four segment strokes b Z T I ^ are used to represent Chinese characters. This chapter is concluded with the summary in Section 2.5. 2.2 Stroke Approximation with Polylines A polyline is a concatenation of straight lines and can be used to approximate an object's stroke boundary in computer vision [6]. Here we use it to represent an input stroke. A polyline can fit a stroke to any desired degree of accuracy. The problem is how to find corners or breakpoints that yield a polyline we desire. Fig. 2.1 shows three input strokes that have one, two and four segments, respectively. The arrow on each stroke indicates the direction of the stroke writing. We hope to obtain the results of one line representing stroke (a), two lines representing stroke (b) and four lines representing stroke (c). A two-step approach is proposed to approximate a stroke with a polyline. Step 1 called a line splitting procedure, uses the iterated endpoint fit method 27] to recursively find a polyline fitting of a stroke. Step 2 is a line merging 13 Chapter 2 Pre-processing t (a) (b) (c) Figure 2.1: Three input strokes. procedure that merges some connected lines according to a rule. To explain step 1 clearly, consider an input stroke shown in Fig. 2.2(a). The initial polyline is a line between the first and the end points of the stroke, marked by A and B (Fig. 2.2(b)). Suppose the point in the stroke that is farthest from the line is C. If the distance from C to the line is above a predefined threshold, then the line AB is split into two lines AC and CB (Fig. 2.2(c)). This procedure is recursively applied to lines and the points of the stroke. Note that these points are now partitioned into two groups corresponding to the two lines. A point D in the first group that is farthest from its corresponding line AC is found, and the line will be split again if the point is too far from the line (Figs. 2.2(c) and (d)). The procedure terminates when the distances, from all points of the stroke to their corresponding lines of the polyline, are all below the threshold. Fig. 2.2(f) shows the final polyline for the fitting of the stroke. In the following, we give an algorithm for the implementation of this recursive procedure. 14 Figure 2.2: Recursive procedure of stroke fitting. Polyline fitting algorithm Input: An n-point stroke, represented by two arrays- x[nl and t/[nj, where (x[/J,7/[/]) is the 2D coordinate of the Ith point of the stroke, 0 < / < n — 1. Output: An m-line polyline, represented by an array key^p[m + 1), where key^p[r] and key.p[r + 1] denote the beginning and the end points of the (r + l)-th line of the polyline, 0 < r < m - 1, keyjp[Qi\ = 0 < keyjp[l\ < < keyjp m =71 — begin -Throughout this thesis, if an array x[n] is given then it is meant that the size of the array is n and the index of the array is from 0 to n - 1. 15 Chapter 2 Pre-processing C B A B (b) D C 3 (a) 7A D B Chapter 2 Pre-processing line^num := 1; key.p[Q] := 0; key.p[l] := n — 1; (comment; the initial line of the polyline) for i = 0,1,… linejnum — 1 do begin Loop the point set {{x[k€y.p[i\], j/[Aret/-p[z]]), (a:[fcei/_p(tl + 1], y[key-p[i] + 1]), • • • ’ {x[key-p[i + 1]], y[key.pli + Ij]), find a point (x[Tnaxjp], y[Tnaxjp]) that is farthest from a line with two endpoints {(x[keyjp[i\[, y[key.p[i\[) and {(x[keyjp[i1]], y[keyjp[i + 1]]); if the distance of the point from the line is larger than a prede-fined threshold T^ then begin linejnum := linejnum + 1; for j = linejnum, linejnum — 1,..., i + 2 do key.p[j] := key.p[j — 1]; (comment: rearrange the lines of the polyline) key.p[i + Ij := max.p; go to Loop; end end end 16 Chapter 2 Pre-processing Algorithm Effectiveness. The main effort of the algorithm is the calculation of distances from the points of a stroke to their corresponding lines of the polyline in each iteration. Let n be the number of points of the stroke. If the last resulting polyline has I lines, then the algorithm will terminates after I iterations. In each iteration, the number of points to be visited is at most n. Thus the upper bound on the computational time of the algorithm is 0(ln). In general, I is much less than n (less than 10 in stroke fitting), so the algorithm is very efficient. For an input stroke, the value of the threshold Td in the algorithm determines the number of lines of the resulting polyline. The smaller the threshold is, the more lines the algorithm yields. It is desired that a stroke be fit by a polyline just as what we want. For example, we consider the stroke shown in Fig. 2.2(a) is a 4-seginent stroke and wish the algorithm had yielded a 4-line polyline. If T^ is larger, we can obtain such a polyline. However, too large threshold may make the algorithm ignore some segments of a stroke when it is written on a small area. In order to obtain more desirable polylines, we employ a simple and fast line merging procedure after the stroke fitting. Line merging algorithm Input: A polyline. Output: A modified polyline. begin calculate the angles of lines of the polyline; 17 Chapter 2 Pre-processing while the change between angles of two connected lines of the polyline is less than a predefined threshold Ta do begin merge the two lines into one; calculate the angle of the new line; end end The thresholds Td and Ta in the above two algorithms are determined by experiments. In our application, an input character is written on a 6cm x 6cm area of the digitizer and is normalized on a 100-pixel x 100-pixel image. Tj and Ta are chosen as 6 pixels and 50 degree, respectively. Some stroke approximations by polylines are presented in Fig. 2.3 in which columns (a) and (d) are input strokes, columns (b) and (e) give their fitting results by the polyline fitting algorithm (step 1), and columns (c) and (f) are the polylines after the line merging processing (step 2) of the polylines in (b) and (e). From the examples, we can see that in most cases, step 2 does not change the results of step 1, while for the strokes in rows 4-8 of column (a), step 2 obtains improved polylines. In practical handwriting, erratic hand motion is easy to generate some wild points and hooks at the beginning or end of strokes such as those in rows 1-3 of Fig. 2.3(a). The polyline fitting algorithm can handle these kinds of noise. 18 Chapter 2 Pre-processing Figure 2.3: Stroke approximation, (a), (d) Input strokes, (b), (e) Polylines after fitting the input strokes ia step 1. (c), (f) Polylines after the merging processing of the polylines in (b) and (e). 19 Chapter 2 Pre-processing Table 2.2: 18 model strokes. Type ‘ Strokes 1 Type Strokes } 1 ~ - 10 * Z 2 : r II y/ 3 I Z 12 b 4 \ 13 5 Z 14 ^ 6 i \ 15 T ^ ? 7 i J 16 C 8 < 17 9 ! L 18 2.3 Stroke Type Recognition Strokes with different shapes provide very useful information for us to distinguish a character from the others. In the proposed stroke-based Chinese character recognition methods, the primitives are strokes. Therefore, recognition of types of input strokes is one of the important steps. The approximation of input strokes by polylines benefits the stroke recognition task. The standard strokes 1-15 and three connected strokes occurring often are listed in Table 2.2. They are called model strokes. A connected stroke is a stroke that concatenates two or more standard strokes. We group some strokes together since they are similar to one another. Strokes of types 1-5 appear most frequently in Chinese characters. By analyzing these strokes in Chinese character handwriting, we define the writing angle intervals of (-20°, 30°], (250 ’290 j (180 250 1’ 290° 340 ] and (30 75 ] as shown in Fig. 2.4 for strokes 1-5, respectively. If the polyline approximation of an input stroke has one or two lines, we can 20 Chapter 2 Pre-processing 250 270° 290° Figure 2.4: Angle intervals for strokes 1-5. lines with those of for a polyline with identify it easily by comparing the direction changes of the the standard strokes having one or two segments. However, more than two lines, its stroke type recognition becomes complicated because (1) wide variations exist in handwriting, and (2) it is impossible that the polyline approximation of a stroke always produces a result we desire. Look at the two input strokes shown in Fig. 2.5(a). Comparing their fitting polylines in 1 Figure 2.5: (a) Two input strokes, (b) Corresponding polylines. Fig. 2.5(b) with their corresponding model strokes in Table 2.2 we find the differences between, the line numbers of the polylines and the segment numbers of the model strokes. In order to identify these kinds of input strokes reliably, we propose a stroke recognition approach using chain code string matching in the following. 21 (b) Chapter 2 Pre-processing (^ ^^ T^nput stroke^ polyline fining line-&-segmem direction comparison 1 1 CjlS^ied strokeiy^ Yes normalization 1 chain code extraction chain code siring matching chain code base of model strokes C^d^ificd strokejyp^ Figure 2.6: Stroke type recognition structure. 2.3.1 Normalization and Chain code extraction The structure of stroke type recognition is shown in Fig. 2,6. For an input stroke approximated by a polyline with one or two lines, its type identification is an easy task as mentioned above, so we only consider the recognition of a stroke fit by a polyline with more than two lines. Strokes belonging to the same type may be written as having significantly different sizes. Thus normalization of polylines of strokes before recognizing them is reasonable. Let (Xmax, Vmax) and (xmin, Vmin) be the upper-right and the lower-left corner points of the smallest rectangle that surrounds a polyline. Let (r, y) be one of 22 Chapter 2 Pre-processing the vert exes that represent the polyline. The equations = — (X - Xniin) (2.1) max min yi = — ~ ( y - ymm) (2.2) Vmax — 2/min will transform (x, y) to a new vertex (xi, yi) of the corresponding normalized polyline that is located in a square of size 100-by-100. Fig. 2.7 gives an example of normalization. - \ 5555566600000 000066666444 (a) (b) (c) (d) Figure 2.7: (a) Input stroke, (b) Polyline of (a), (c) Normalized polyline of (b). (d) Chain code string of (c). A polyline consists of several lines, and the directions and lengths of the lines represent the feature of the polyline. If we divide each line into a set of shorter lines (called line units), each with the same length, then chain codes become a very suitable tool for the representation of a polyline. Chain codes, in our application, are a notation for recording a string of line units along a polyline. A code specifies the direction of a line unit. There are eight quantized directions as shown in Fig. 2.8. Starting at the first line unit and ending at the last line unit of a (normalized) polyline, a string of chain codes is not difficult to obtain by investigating the direction and length of its every line. Fig. 2.7(d) presents such an example. 23 Chapter 2 Pre-processing ~ > 0 5 6 7 Figure 2.8: 8 directions of chain codes. The chain code base contains a set of chain code strings of the model strokes having two or more segments. The procedure of constructing the base is just like that of finding the chain code string of an input stroke to be recognized, including (1) fitting strokes with polylines, (2) normalizing the polylines, and (3) obtaining the chain code strings from the polylines. 2.3.2 Chain Code String Matching A critical step in stroke type recognition is to measure the similarity between two strokes. Now we represent normalized strokes with chain code strings. The string edit operations and the string edit distance proposed by Wagner and Fisher [96] can be used to reach this goal. Here we introduce only the basic concepts and the string matching algorithm following the work of Wagner and Fisher. More detailed description will be given in Section 6.2. Let A denote a null chain code. An edit operation is a pair (a b) written as a —y 6, where a or 6 may be a code of a string but if a and 6 ^ A, both a and b must be two codes. The three edit operations on a string are code substitution, code insertion and code deletion, denoted by a 6, A ^ a and a —A, respectively. Obviously, there are infinite sequences of edit operations 24 Chapter 2 Preprocessing that can transform a string to another string. Let 7 be a cost function that assigns to each edit operation a —6 a nonnegative real number 7(a —> 6), and let the cost of a sequence of edit operations be the sum of all the edit operation costs. Then the edit distance 6{Si, S2) between two strings Si and S2 is defined as the minimum cost of a sequence of edit operations that transforms Si to S2. Wager and Fisher provided the following efficient algorithm with the com-plexity 0 ( m n ) for computing the distance between a string of length m and a string of length n. String matching algorithm [96 Input: String Si -=S1S2-. Sm and string S2 =s\s\ ‘ 0 (positivity); (b) 7(a — 6) = 0 if and only if a = 6 (definiteness); (c) 7(a — 6 ) = 7 ( 6 — a) (symmetry); (d) 7(a 6 ) + 7 ( 6 —)• c) > 7 ( a — c) (triangle inequality). For our chain code string matching problem, we also have to choose reason-able cost values with respect to different edit operations. Let s‘ Sj € { 0 , 1 , 7 } be two chain codes. We define 7(5. sj) = 7(sj — Si) = min{A:i|si - sjl,ki(8 - js‘ Sj\)} (2.3) as code substitution costs and 7(5. — A) 4 5,) = k2 (2.4) as code deletion and code insertion costs, where ki and k are two positive values. Theorem 2.1 The cost function 7 defined in (2.3) and (2.4) ” a metric if k2 > 2ki. 26 Chapter 2 Pre-processing Proof. It is obvious that 7 fulfills the conditions of positivity, definiteness and symmetry. To proof the triangle inequality 7(5, — s ) + 7(sj — Sfc) > ’(Si — sjk), let us consider two cases. Case 1. Suppose s,, s , Sk G {0,1,.... 7}. From (2.3) and the figure below, ^ 2 H a 5 we see that 7(5^ — Sk) is directly proportional to the angle 6 < 180° from 5, to Sk. If Sj is located in this interval, then the angle 1 from 5, to s j and the angle 02 from Sj to Sk satisfy the relation + 0 and thus 7(Si — Sj) + 7( —Sit) = 7{si — Sk). Otherwise, + > and 7(Si — Sj) + 7(sj Sfc) > 7(Si — Sfc). Case 2. Recall that a 6 A ^ A. If 5 = A, then Sj, s^ € {0,1,… 7}. We have 7(A — Sj) + 7(Sj. — Sk) = k2 + 7(5 -> Sk) > k2 = 7(A Sk). If Sk = A, we also have l{si — Sj) + 7 (s j — A) = 7(Si — Sj) -f fc2 > fcz = 7(s,- A). 27 Chapter 2 Pre-processing If Sj = A, since 7(5,• Sk) < 4 ,1 (see (2.3)), we further have 7 ( S i A ) + 7 ( A ^ S k ) = 2 k 2 > 4 k i > 7 ( 5 , — s t ) . Therefore the triangle inequality holds. • Let Pi, P2, Pq be q model strings and S be an input string to be classified. The solution to the problem of recognizing 5 is first to find a model string Pk 6 {Pi,P2,...’ Pq) such that S{Pk. S) = mm{6{PuS), ^(Ps, 5), ... 5)}, and then to classify S into the class which Pk belongs to if 5[Pk, S) is less than a predefined threshold; otherwise to classify it into an unknown-stroke-type class. Note that the number of model strings may be greater than the number of string classes. We use three to stand for one stroke class in our recognition scheme. This is beneficial to tolerating more handwriting variations. Before applying the string matching algorithm to Chinese character recogni-tion, we have to determine the two parameters ki in (2.3) and ko in (2.4). In our learning procedure, for each class, 20 strokes written by 4 people were collected as the training data. The prototypes of a class consist of three strokes. One was written in its standard style and the other two are its generally handwritten deformed versions. The aim of the learning procedure is to find the optimal parameters ki and fc2 that maximize the following recognition rate ^ . __ The number of strokes classified correctly 1 ‘ 2 The total number of the training strokes ‘ ‘ Fig. 2.9(a) shows the relation between R and (^i, ^2)- From another view-point we can obtain the projected surface on the ki-R plane (Fig. 2.9(b)). It is 28 Chapter 2 Pre-processing (a OS zs (b) Figure 2.9: (a) Recognition rate as a function of (iti A:2). (b) The surface obtaining from another viewpoint. 29 Chapter 2 Pre-processing Figure 2.10 A curve and its fitting line. The curve presents the track where R{ki, takes the maximum value. clear that there is an area where R(ki,k2) takes the maximal value. The curve in Fig. 2.10 is generated by connecting the discrete points (k[, k'^ys satisfying ki^ ) = max{i?(A:i ^2)}. Approximating the curve with a straight line we have k2 = 2.5Ski. (2.6) The figure indicates that a point [ki, k!"} that fulfills ki > 1.5 and k2 ^ 2.58ki has more neighbors {k", k'^Ys, where R{k'(, k'^) = max{i?(A:i, ^2)}- Therefore we choose ki = 2 and k2 = 5.16 as the parameters of the string matching algorithm. Some experiments have been carried out to test the performance of the pro-posed stroke recognition approach. The test data contain more than 1000 strokes written by 5 people. Fig. 2.11 shows some of the input strokes and the classifica-tion results, together with their corresponding stroke types. There are a variety 30 Figure 2.11: Stroke recognition experiments. On the left of the arrows are a set of input strokes. Their classification results: corresponding standard strokes and stroke types, are indicated on the right. 31 Chapter 2 Pre-processing 11 13 10 8 9 16 14 7 / L : : => => => => => => => u 0 4 V M V n C ( O J / Figure 2.12: Four examples of misciassification. of handwriting stroke sizes and styles in the data. Our approach achieves a cor-rect recognition rate of 96.2%. Fig. 2.12 gives four strokes recognized incorrectly in the sense that the classified stroke type of an input stroke is different from the type that the subject expects the stroke should be. However, the input stroke in Fig. 2.12(a) is similar to both type 8 stroke and type 9 stroke, and the input stroke in Fig. 2.12(c) is similar to both type 6 stroke and type 12 stroke. The stroke recognition experiments also show what pairs of strokes are easily confused. This information is very useful for the design of stroke-based Chinese character recognition methods in which assigning different costs for stroke type comparisons is necessary. By the way, it is not definite that misciassification of some strokes of a character leads to incorrect recognition of the character. 32 Chapter 2 Pre-processing => (b) ( Chapter 2 Pre-processing 2.4 Segment Extraction and Processing In out segment-based Chinese character recognition methods, the primitives are segments of strokes. Segments are the smallest units that construct Chinese characters. Each standard stroke consists of one to four segments, (see Table 2.1) A connected stroke may have more than four segments. Six segment types are defined as the primitives: type 1 “ — ” -20°’30 type 2 " i " (250°,290°], type 3 “ Z ” (180°,250°], type 4 “ \ ” (290°,340 type 5 " y " (30° ,75°] and type 0 denoting an unstable short 1-segment stroke that is easy to be written as one of the segment types 1-4 such as the top stroke of the character “ ”. Actually, segment types 1-5 are just the same as stroke types 1-5, respectively. After approximating an input stroke by a polyline, if we consider a line of the polyline is equivalent to a segment, obtaining the polyline means that we have finished the segment extraction. It is important to note that segment “ ' x ” with directions ranging from 75 ° to 180 is not included in the set of segment primitives while they exist in the standard strokes. By analyzing handwritten Chinese characters, we can find such two facts: (1) ignoring this kind of segments in Chinese characters does not make us be confused when we recognize them, and (2) they exist in many-connected strokes of handwritten Chinese characters, as shown in Fig. 2.13. Therefore, we delete these segments before the processing described below. This scheme has also been used in several other methods [17 21, 64. From the next chapters, we can see that segment-based Chinese character recognition methods can deal with the problem of recognizing Chinese charac-ters with more connected strokes better than stroke-based methods. Connected 33 Chapter 2 Pre-processing Z \ X Figure 2.13: Several examples where bold segments “ \ ” appear in connected strokes. strokes often lead to extra segments. In order to facilitate our segment-based recognition, we use some rules to determine which segments of an input character should be employed for later recognition. Besides the frequently occurring connected strokes led by the segment “ \ ” there are many connected strokes in natural Chinese character handwriting. Some of them yield extra segments but some do not. Figs. 2.14(a) and (b) show-two sets of characters or components of characters corresponding to the former and the latter cases, respectively. The segments of both standard strokes and the connected strokes that have no extra segments should be used to represent Chinese characters. However, some of these strokes, such as “ z ” and “ ” also appear in Fig. 2.14(a). Thus a trade-off must be made. We adopt such a scheme that all the segments of these strokes will remain. To summarize, we give the following rules that are used in the segment preprocessing. 34 Figure 2.14: (a) Examples of connected strokes leading to extra segments, (b) Examples of connected strokes not leading to extra segments. 35 Chapter 2 Pre-processing = i E = ly : i z z = M \ : I : : Chapter 2 Pre-processing Table 2.3: 14 Multi-segment strokes used to determine whether with more than two segments belongs to one of them. input stroke Stroke Stroke Stroke I 6 11 2 7 w 12 M 3 8 a 13 £ 4 9 14 5 10 Rule 1. The segments “ \ “ existing in all input strokes are deleted. Rule 2. If the segment number of an input stroke is less than 3 all the segments of the stroke remain. Rule 3. If the segment number of an input stroke Si is greater than 2, the stroke recognition method presented in the last section is employed to find the minimum distance (5(5,-, Sj) = Si), S{Si, S2), • S{Si, 5i4)}, where Sj, j = 1 2 , 1 4 is one of the strokes shown in Table 2.3. • If S{Si, Sj) < Tr and j < 11 then all the segments of the stroke remain. • If (5(5i, Sj) < Tr and j = 11 or 12 then the segments “ I “ remain and the others are deleted. • If 6(Si, Sj) < Tr and j = 13 or 14 then the segments “ —> “ remain and the others are deleted. • If Sj) > Tr, then Rule 4 is used. Rule 4. Suppose a stroke (after the processing of deleting segment " \ ") has m segments. If m is odd, then the 2th, 4th, ... m — l)-th segments are deleted 36 Chapter 2 Pre-processing and the others remain. If m is even, then the 2th, 4th, ... , (m — 2)-th segments are deleted and the others remain. Here TV is a predefined threshold and Rule 4 is borrowed from [21]. Clearly, it is impossible to delete all extra segments or to obtain all the segments that should remain for later character recognition. However, these rules do provide us satisfactory processing results, as illustrated in Fig. 2.15. The model characters having 9 to 11 strokes were written as their input versions having 2 to 6 strokes. It can been seen that the input characters, after processing, are more recogniz-able. To break the connected strokes at proper positions, Rule 1 contributes the most because lots of connected strokes lead to the segments " \ Rule 1 is so effective that it, after modification, is also used in the stroke-based representation and recognition of Chinese characters. The modified rule is delete the segments “ \ “ that exist in input strokes and are not the I cist segments in these strokes. Finally, we have to state that some segment processing errors of an input character (such as deleting a segment that should remain to represent the char-acter) do not mean that a misclassification of the character must take place. To design robust recognition methods that can tolerate more handwriting variations and preprocessing errors is the aim of the next several chapters. 2.5 Summary We have introduced several preprocessing approaches to on-line Chinese charac-ter recognition in this chapter. First, we have approximated input strokes with 37 Chapter 2 Pre-processing 7 CM^ ^^ at, f 1 4 \ \ V H S ^ (a) (b) (d) (e) Figure 2.15: Examples of segment processing. Columns (a) and (d) are input Chinese characters. Columns (b) and (e) are the segment processing results. Columns (c) and (f) show the corresponding model characters. 38 Chapter 2 Pre-processing polylines by using the efficient polyline fitting and line merging algorithms, to facilitate the recognition of strokes and segments. Secondly, we have proposed a method for identify strokes each with more than two lines. It consists of three procedures: normalization of strokes, extraction of stroke chain code strings, and matching between input code strings and model code strings. The method works well and can be used not only in stroke-based but also in segment-based on-line recognition of Chinese characters. Thirdly, in the section of segment extraction and processing, some rules are presented to detect most of frequently-occurred connected strokes and then delete the extra segments in such strokes. These rules make our recognition methods have the ability to recognize more freely-written Chinese characters. Parts of the results presented in this chapter have been published in [58 59 60, 61 62’ 631. 39 Chapter 3 Relational Graph Representations of Chinese Characters 3.1 Introduction What kinds of features of Chinese characters to be chosen and how to repre-sent Chinese characters using these features are two important issues on Chinese character recognition. Human beings have the best ability in recognizing com-plicated objects. Development of on-line Chinese character recognition systems that approach the human ability is the goal of researchers working on this field. Chinese characters are two-dimensional (2D) pictographic characters. In gen-eral, the structure of a Chinese character having more than five strokes can be decomposed into four levels as shown in Fig. 3.1. The most basic elements con-structing Chinese characters are strokes but the segment level is, in my opinion, 40 Chapter 3 Relational Graph Representations of Chinese Characters character I n -I components strokes segments Figure 3.1: 4-layer structure of a Chinese character. most useful for computer recognition of freely handwritten Chinese characters. Let us consider how a human being distinguishes a Chinese character from the others. If he/she is familiar with Chinese characters, he/she recognizes a printed or neatly written character by identifying its each component and the 2D arrangement of the components. His/Her ability of quickly finding a component comes from his/her understanding of how the component is formed by the 2D arrangement of some strokes, the smaller elements. If a character is freely written and has several connected strokes, in order to recognize it, he/she also needs the knowledge of general freely handwritten styles of Chinese characters. Obviously, human beings use 2D relational (structural) features of Chinese characters to conduct the recognition activity. The structural methods that can capture human knowledge of Chinese characters very well should have the best performance. Relational graphs are a powerful tool for the representation of relational structures of a pattern. They have been used for 2D or 3D scene analyses [11 28 33, 45, 54 80 82, 84, 85 90 100 101] as well as on-line and 41 Chapter 3 Relational Graph Representations of Chinese Characters off-line Chinese character recognition [13 16 18 58 59 60 61 62 63 68 • In Section 3.2 we formally define the complete relational graphs and the distance measures for comparing the similarity between two graphs. Then, we propose several graph representations for on-line Chinese character recognition in Section 3.3, including stroke-based and segment-based spatially relational representations, as well as stroke-based and segment-based spatially-temporally relational representations. The assignments of costs to node and arc correspon-dences for calculating distances between two graphs are presented in Section 3.4. The chapter ends with the summary, 3.2 Relational Graphs and Distance Measures 3.2.1 Complete Relational Graphs Definition 3.1 Let Vn be a set of node labels and Va a set of arc labels. A relational graph over V = V^\jVa is a 4-tuple G — {N, where • N is a finite nonempty set of nodes; • A C N X N is a set of distinct directed pairs of distinct elements in N called arcs; • fi : N — Vn is a function for labeling the node; • £ : A Va is a function for labeling the arcs. Relational graphs can be used to describe the structural information of pat-terns. Fig. 3.2 shows a simple example of the representation of a scene, where nodes indicate object primitives and arcs describe spatial relations between ob-jects. In this thesis, we use complete relational graphs to represent Chinese 42 Chapter 3 Relational Graph Representations of Chinese Characters (a) V)v={book cup board, clock ... y^ ={ above, right of left of , . . .} Node set={n 1, AZ2, n-^.n^] A r c s e t = { f l | fl2 “ 3 <24 5} |I(a2 i)=board, \yin 2)=cup, pi(/2 3)=fX(A2 4)=:bOOk £(a 1)= z{a 2)= z{a 5)= above £(«3)=e(«4)=right of. Figure 3.2: An example of relational graph representation of a scene, (a) A scene, (b) The corresponding relational graph. characters. Definition 3.2 A complete relational graph is a relational graph such that for any two distinct nodes rii and 112, there are two arcs: one from n\ to n2 and the other from no to rii, denoted by (ni, no) and (n2,ni), respectively. A possible complete relational graph representation of the Chinese character in Fig. 3.3(a) is illustrated in Fig. 3.3(b). In this example, the primitives are strokes of the character and their types are represented by the nodes of the graph; 43 Chapter 3 Relational Graph Representations of Chinese Characters 1 I n 2 "3 " 4 "5 " 6 (1,2.0) (2,1,0) (2.1.0) (1,1,0) (2.1.0) I'J. — (0.2.0) (2,1.0) (2,1.0) (2,1.0) (0,1,0) ” 3 (2,0,0) (2,0.0) (2,2,1) (1,2,0) (2,1.0) r 4 (2.0,0) (2,0,0) (2,2,1) (2,1,0) (0,1.0) "5 (0.0,0) (2,0.0) (0,2.0) (2,0.0) (0,2,0) "6 (2,0.0) (1,0.0) (2,0.0) (1,0.0) (1.2.0) (C) Figure 3.3: (a) A Chinese character, (b) Corresponding complete relational graph, (c) Relation matrix of the graph. the arcs describe the relations between any two nodes (strokes). Three relation types are used: (1) "below" (denoted by "0"), "above" (denoted by “1 or "don't care" (denoted by “2” 2) “right of" (denoted by “0” ’ "left o f (denoted by “1”)’ or "don't care" (denoted by “2”) (3) "uncrossed" (denoted by "0"), "crossed" (denoted by T ) , o r "don't care" (denoted by "2"). The relations of an arc (rii, Uj) are described by a vector (aj: a j af,) where a? , af^ G {0,1, 2}. The superscripts 1 2 and 3 on respective af^, afj and a^j represent the three types of relations. For example, the relations of arc (ni’n2) are (1, 2,0), which suggest that in general handwriting, the geometric center of stroke 1 is always 44 Chapter 3 Relational Graph Representations of Chinese Characters above that of stroke 2; it is sometimes on the left of and sometimes on the right of that of stroke 2 these two strokes are uncrossed each other. All the relations among strokes of the character are shown in. Fig. 3.3(c). The reason why we use complete relational graphs to represent Chinese characters and the more general representation will be discussed in Section 3.3. Since we always deal with complete relational graphs in most parts of this thesis, we will just speak of graphs unless otherwise stated, and we use G or G iV", >1) to denote G = (iV, A, fi, £•) for short. In addition, the graph in Fig. 3.3(b) may be drawn in its simplified forms of Fig. 3.4(a) or (b). (a) (b) Figure 3.4: Two simplified forms of Fig. 3.3(b). Definition 3.3 An induced subgraph G' = {N\ A') of G = (N, A) is a graph whose node set C N and whose arc set comprises exactly the arcs of G which join nodes in N’. 3.2.2 Edit Operations on Graphs Edit operations are commonly used to transform a string, a tree or a graph to another [11, 28 30 69, 79 81 90 91, 96 98]. The concept of edit distances is easily understood in comparison of the similarity between two strings, two 45 Chapter 3 Relational Graph Representations of Chinese Characters trees or two graphs. In this section, several edit operations on complete rela-tional graphs are formally defined. The edit distance and the matching distance between two graphs are presented in the next section. Let A denote a null node or arc. An edit operation is written as a —> 6, where a — 6 # A —A, and a or 6 may be a node or an arc of a graph but if a \ and 6 ^ A, both a and b must be two nodes or two arcs. The following six kinds of edit operations are used • node insertion: X ^ a {a is a node) • node substitution: a — b (a and b are nodes) • node deletion: a — A {a is a node) • arc insertion: A —> a (a is an arc) • arc substitution: a — b (a and b are arcs) • arc deletion: a —> A (a is an arc) These edit operations are used to transform a graph to another. In our appli-cation, the graphs under study are required to be complete. So some constraints on these operations are necessary: • If a node of a graph is inserted, arcs that join this node and all the existing nodes of the graph are also be inserted. • If a node of a graph is deleted, arcs that join this node and all the other nodes of the graph are also be deleted. • An arc is deleted only when one or two of its end nodes are deleted. • An arc is inserted only when one or two of its end nodes are inserted. The application of an edit operation a —> 6 to graph Ga results in Gg, which is written as Ga Gb via a ^ 6. Let E he a. sequence ei, 6 2 , S m of edit 46 Figure 3.5: Examples of edit operations, (a) Gi G2 via rii —> n[ and a — d. (b) G3 via rii ^ A, a A and 6 —)• A. (c) Gi => G4 via A -> 774,A —> rf, A —> e and A — / . operations. An edit transformation of graph G^ to graph Ge is a sequence of Go, G i , G t n such that Ga = Go, Gb = Gm and C?‘_i => G, via e, for I < i < m. The transformation is also denoted by Ga => Gb- Several edit operations and transformations on graphs are shown in Fig. 3.5. Note that for an edit transformation, in order to fulfill the constraints men-tioned above, Ga and Gb need to be complete relational graphs, but Gi, C?2 … Gm-i may not. For example, if ei is a node deletion operation, then Gi, which has a set of arcs each with only one node at its end, is not even any kind of graph defined in graph theory. However, sometimes we stiil call them graphs if there is no confusion. 47 Chapter 3 Relational Graph Representations of Chinese Characters © 0 V © 0 0 Y © d © (b (a) 0 V © © 0 T V © / @ ® V © / 0 Chapter 3 Relational Graph Representations of Chinese Characters 3.2.3 Distances between Two Graphs In practical recognition problems, objects belonging to the same class may have different degrees of distortion compared with their model object. As a conse-quence, the graphs representing them may also be different. To measure the similarity (or distance) between two graphs, costs associated with these edit operations are necessary. Let 7 be a cost function that assigns to each edit operation a — b a nonnegative real number —> 6). 7 can also be extended to a sequence of edit operations E = ei, 6 2 , e ^ by setting 7( = l i ^ i ) -If m = 0, i.e., no edit operation is applied, we define j(E) = 0. Definition 3.4 Let Gi and Gj be two graphs. The edit distance from Gi to Gj is defined as 5{Gi, Gj) = min{7( )| is a sequence of edit operations that transforms Gi to Gj}. (3.1) Theorem 3.1 S(Gi, Gj) is a metric on So if the following conditions are ful-filled, where So is the set of all (complete relational) graphs. (a) 7(a —> a) = 0; (b) — 6 ) > 0 if a (c) 7(a —> 6 ) = 7 ( 6 — a). Proof. (1) (Positivity) SiGi^Gj) > 0 holds for all Gi, Gj 6 So since 7(0 - r 6) > 0. (2) (Definiteness) On the one hand, if Gi = Gj, we have 5(Gi, Gj) = 0 from the definition of S{Gi, Gj). On the other hand, if Gi + Gj, in order to 48 Chapter 3 Relational Graph Representations of Chinese Characters transform G to Gj by a sequence of edit operations, at least one edit operation y{a —6) > 0 (a 6) in the sequence must be applied, so 5{Gi, Gj) > 0. This implies that if 5(Gi, Gj) = 0, Gi = Gj. (3) (Symmetry) Suppose <7, => Gj by a sequence of edit operations Ei = e i ,e2 , - ,em and 5{Gi,Gj) = 7(^1)- A sequence E2 = 6 e -1’..• ei will transform Gj to Gi and the transformation cost 7(^2) = 7( 1) since 7(0 — b) = 7(6 — a ) . Assume S{Gj, Gi) 7(«£"2). By assumption, there must ex-ist a sequence of edit operations E3 = e'„ e'„_i . ’ e'l such that Gj => Gi and 7( 3) < 7(>E"2). With the application of the sequence E4 = ei eS,... e“ to Gi, Gi => Gj will result and we have = 7( 3) < 7( i) which is in contradic-tion to S{Gi, Gj) = 7( 1). Therefore, 5{Gj, Gi) = Gj). (4) (Triangle inequality) We uow show that S(Gi, Gj) < S{Gi, Gk)-i-S{Gk, Gj) for all Gi, Gj, Gk G Sg- Let Ex = ei, e ’ …’ ti transform G, to Gk. and Gk)== 7(^1), and let E2 = e^ 1, ef+2’ -.-’ transform Gk to Gj and S{Gk, Gj) = 7(£"2). Obviously, E3 = 61,62, e/, e^n transforms G, to Gj. Hence yiE^)= y{Ei) + 7(£ 2) = Gk) + S(Gk,Gj). By the definition of the edit distance 6{Gi, Gj), it is immediate that Gj) < 7 (^3 )= G“ Gk) + ^(Gjk, Gj). • Note that the edit distance fulfills the triangle inequality even if such a property does not hold for the cost function 7(0 b). In other words, 7(0 —> 6) < 7(a — c) + 7(c —> b) is not required. There are infinite ways to transform a graph to another. For example, the substitution of a for b may be done not only by a —r 6, but also by a — c and then c b. To simplify the problem of finding the distance between two graphs, a concept mapping is introduced in the following. 49 Chapter 3 Relational Graph Representations of Chinese Characters Definition 3.5 Let A, = {A} and Aj = {A} be two sets of null nodes. Let Gi = [Ni, Ai) and Gj — {Nj,Aj) be two graphs. A node mapping from Gi to Gj is a function In • Ni U At Nj U Aj satisfying the folloiving conditions: f s W + A; (b) If Ui ^ TRi then /iv( • /"(""^) for all “ G iV, and fN{ni),Mmi) e Nj (c) For a node rii e Ni, there exists a node rij € Nj U Aj such that fNirii) = rij (d) For a node rij € Nj, there exists a node rii e N] U A: such that /ivH^j) = "i-In this definition, for n^ € Ni and Uj € Nj, fi^irii) = A and = A indicate a node n, € Gi and a node rij G Gj are deleted. To guarantee that all the graphs under study are complete relational graphs, when a node is deleted, all arcs connecting it will be deleted too. Fig. 3.6 shows two examples of node mappings. From Definition 3.5 and Fig. 3.6 we can see that a node mapping f ^ leads to an arc mapping. Definition 3.6 Let A, = {A} and Aj = {A} be two sets of null arcs. Let Gi = [Ni Ai) and Gj = {Nj, Aj) be two graphs. An arc mapping led by a node mapping ( f ^ Ni U A, — Nj U \ j ) from G, to Gj is a function /a l A i U A i ^ Aj U Aj-satisfying the following conditions: 50 Chapter 3 Relational Graph Representations of Chinese Characters (a) (b) Figure 3.6: Two node mappings, (a) /yv(ni) = A, M n ! ) = ns, / n M = Uq, / " ( 4) = nr. (b) , v ( n i ) = A, = / “ =r i s , /,v(A) = rie, / " ( A ) = fa J For an arc (n^, m,) G # mi ni m, 6 Ni, i / / " ( n , ) /iv(mi) G Nj, then fAiinu Trii)) = (/jv(ni),/iv(m,)) G Aj (b) For an arc (n,’ m,) G A,, if fsi'^i) = ^ or •( i) = A or both fNirii) = A and fs{rni) = A then fA{{nu m,)) = A e Aj (c) For an arc (n,’ mj) € Aj, if fi^^irtj) = X or = X or both fN^rij) = A and f - f j U j ) = A, then f ((iy,mj)) = A G A^. As two examples, for Fig. 3.6(a), the arc mapping is: /A((ni,n2)) = A’yU((ni’n3)) = A,/.4((ni, ^4)) = A,/^((ria, "3)) = (n5 n6) fA((n2, ^4)) = ("5, 7)’ fA((n3, n4)) = (ne, Uy), and for Fig. 3.6(b), the arc mapping is: fA{{ni,n2)) = A,/A((ni n3)) = A,/>i((n2,713)) = {n4,ns)J^\{ne,n5)) = A, Here we call = n and /^((nj , mi)) = ( j i j ,mj) a node correspondence and an arc correspondence, respectively. The former associates node rij to node Hi, and the latter associates arc (uj, rrij) to arc (n,-, mi). 51 Chapter 3 Relational Graph Representations of Chinese Characters For expression simplicity, we will also use to denote node and arc correspondences if there is no confusion. For example, n, —> Uj is equiv-alent to fy(ni) = TLj. Similarly, the arc correspondences fA{{ni, rrii)) = A and f = A will be written as (n^, mi) — A and A (rij, rrij), respectively, and if rii A’ mi A, n* — r i j A and m, —rrij ^ A, then /^((nj , m,)) = {fij, TUj) will be denoted by (rii, m,) {nj, nij). We can see that each of these node and arc correspondences corresponds to an identical edit operation, so we will also use the cost function 7 to assign costs to these correspondences. Definition 3.7 Let /yv iVjUA,- -> iV}uAj be a node mapping from Gi = (iV, /Ij) to Gj iVj Aj), and let /a A, U A, Aj U Aj be an arc mapping led by /jv. The pair (Jn, M ^ termed a matching from G, to Gj. The matching cost is calculated by PUNJA) = Yi — r i j ) + Y . ^ A) 4- Y . 7 ( A ^ rij) n,-f-nj€C?i n.-fAeC?? A-tn_,GQ3 + S rrii) {nj, rrij)) (n,,m,)-^(nj,mj)€Q4 + 7(( mi) — A) + E 7(A (nj, mj)) (3.2) where (/IV, /A) determines the sets Qi-q, i.E., Qi is the set of rii —> Nj, rij € Nij Uj e Nj; Q2 the set of rii A, n, e Ni Q3 the set of X nj, Uj € Nj Qa the set of {RII, m,) {Uj, rrij), RII, RRII € iV,, n^, RUJ G Nj, RII # t; QS the set of [rii, rtii) A, n,, rrii ^ Qe the set of X — (n^, mj),nj, rrij € Nj. The matching distance from G, to Gj is defined as €(Gi Gj) = mm{0{fN, /^)|(/iv, Ja) is a matching from G, to Gj}. (3.3) 52 Chapter 3 Relational Graph Representations of Chinese Characters An optimal matching (/j^, f \ ) is a matching such that / )=^(Gj, Gj). Comparing the definitions of the node mapping and the arc mapping with the definition of edit transformation, we can easily find the similarity between them, which implies that there is a relation between and S(Gi,Gj). Lemma 3.1 For a matching (Jn, /a) from Gi to Gj, there exists a sequence E of edit operations, which transforms G, to Gj, such that j(E) fA). Proof. The node mapping /jv and the arc mapping consist of a set of node and arc correspondences, each of which is equivalent to an edit operation. These edit operations comprise a sequence E that transforms Gi to Gj. Thus It is worth noting that for any sequence E of edit operations that transforms Gi to Gj, there may not exist a matching (Jn, M such that /a) = 7(E). The reason is that clearly, /A) is finite/ so /?( /" /A) < P where P is a positive value large enough, but a sequence E that transforms Gi to Gj with 7( > P can easily be found because, say, a node correspondence n, —> Uj may be done by sufficiently many edit operations n,- —> m,m —> p,..., q — rij such that 7(71,- — m) + 7(m — p) H H 7(9 — rij) > P. Lemma 3.2 For any edit sequence E transforming G, to Gj, if the cost function fulfills the triangle inequality (7(a —c) < 7(0 — 6) + 7(6 — c)) besides the conditions (a), (b) and (c) in Theorem 3.1, then there exists a matching (/jv, /a) from Gi to Gj such that < y{E). I^n this thesis, we consider only finite positive costs of edit operations (or node and arc correspondences), and deal with finite graphs, i.e.’ the cardinalities of the node sets of these graphs are finite. 53 Chapter 3 Relational Graph Representations of Chinese Characters Proof. (1) Let Ni be a set of nodes in Gi, to each node of which no any edit operations are applied in the transformation Gi =>• Gj. This set of nodes in Gj is denoted by N[, where |iV\| = |iV;|. For a node ni G Ni, there is a node n\ € N[ such that the node correspondence cost 7(ni — n[) = 0. (3.4) Similar conclusion can be drawn for a set of arcs in Gi, to each arc of which no any edit operations are applied. (2) Let N2 be a set of nodes in G, which are deleted after Gi => Gj. Let 712 € N2 and let the sequence of edit operations applied to 712 be 773 —rrzi, rrii 7722, "T "^r —> A. Prom the triangle inequality for edit operations, we have the node correspondence cost 7(712 A) < 7(712 mi) + 7(mi m:) H f- 7(771^ — A). (3.5) Similar conclusion can be drawn for a set of arcs in G, which are deleted after Gi Gj. (3) Let iVJ be a set of nodes in Gj which are inserted after Gi Gj. Let n ; G N'^ , and let the sequence of edit operations applied to a null node A be A — Pi’Pi — p2, •••,Va — W3. We also have the node correspondence cost n ; ) < 7(A — P i ) + 7 ( P i — + • • . + l{p, — n[ (3.6) Similar conclusion can be drawn for a set of arcs in G j which are inserted after Gi Gj. (4) Let N^ be a set of nodes in Gi which are substituted after Gi Gj. Let 714 € N4, and let the sequence of edit operations applied to 714 be 714 qi,qi 54 Chapter 3 Relational Graph Representations of Chinese Characters q2, ...,qt where n\ 6 I7V4I = and N[ C Nj. We also have the node correspondence cost 7(n4 — n\) < 7(724 qi) + 7(^1 — + . • • + — < ) . (3.7) Similar conclusion can be drawn for a set of arcs in Gi which are substituted after Gi => The node correspondences, which correspond to the node correspondence costs on the left sides of (3.4)-(3.7), comprise a node mapping from Gi to Gj. The arc mapping Ja led by /V is not given explicitly for simplicity. The node edit operations, which, correspond to the edit operation costs on the right sides of (3.4)-(3.7)’ and the arc edit operations not given explicitly comprise the edit sequence E. Adding all the node and arc correspondence costs and adding all the edit operation costs, we obtain Theorem 3.2 The edit distance 5(Gi, Gj) is equal to the matching distance ^{Gi, Gj), if the cost function 7 satisfies the conditions in Theorem 3.1 and the triangle inequality. Proof. By Lemmas 3.1 and 3.2, Theorem 3.2 follows immediately. • Theorem 3.2 suggests that if 7 is a metric, ^(G,-, Gj) is also a metric. In what follows, we will use the matching distance to measure the similarity between two graphs. 55 Chapter 3 Relational Graph Representations of Chinese Characters 3.3 Representations of Chinese Characters 3.3.1 Stroke-Based Spatially Relational Representation Since strokes are the most basic elements constructing Chinese characters, the idea comes first that using the standard strokes (see Table 2.2) as primitives to represent the structural information of Chinese characters. In the last chapter, We have describe the approach to recognizing input strokes. Obviously, strokes with different types are part of the features that are employed by human beings to identify Chinese characters. As mentioned before, each Chinese character has its standard stroke writing order. If people almost always write a Chinese character according to its stroke order, the design of on-line Chinese character recognition systems will become much simpler. We can arrange the strokes of each model Chinese character in its standard stroke order to build a model stroke string base in advance. For an input character, the 2D recognition problem is now transformed into a ID string matching problem by finding in the base the best matching string with the input stroke string. In general, a ID recognition is easier to be solved and needs much less computational effort than a 2D one. String matching based approaches have been used in many on-line Chinese character recognition methods [21, 38 55, 56 64, 65, 86, 92]. However, there are lots of stroke order variations and stroke deformations in Chinese people's handwriting. These make it difficult to distinguish Chinese characters only by making use of the information of ID stroke strings. Consider the characters shown in Fig. 3.7. Strokes of the characters are labeled with numbers indicating the stroke orders. Character 1 is a standard one and char-56 Chapter 3 Relational Graph Representations of Chinese Characters 2 5 4 \ -/ 6 T f T Z I I / \ 8 9 10 11 12 2 : T / / -/ — / z \ \ Figure 3.7: A set of characters with their strings of decomposed strokes. A number near a stroke indicates the order of the stroke when the character is written. 57 Chapter 3 Relational Graph Representations of Chinese Characters acters 2 and 3 are its common handwritten styles. The orders of two strokes in character 2 are exchanged compared with character 1. Characters 4-12 are other different characters. In the 2D plane, we can easily find that characters 2 and 3 are more similar to character 1 than characters 4-12. But this conclusion is difficult to draw if we only compare the stroke strings of these characters. Let Si be the zth string and D(Si, Sj) be the distance between Si and Sj, € {1, 2 12}. By observing these strings, we have and D{SuS2) > D(S4,52), D(5I ,52) > D(Se,S2). These lead to the result that when S2 is inputted, it will be identified to be character 4 or 6 instead of character 1. Moreover, The strings of characters 10-12 are almost the same. In fact, from a matching point of view, all the strings in Fig. 3.7 are similar to each other, even though some of the characters have 1 more strokes than the others. Here we just give an example with the set of characters. Many similar examples exist in handwriting. Therefore, in order to develop a good on-line Chinese character recognition system, only the information of stroke strings of Chinese characters is not sufficient and the 2D structural features of Chinese characters must be utilized. The relational graphs, as a tool of representation, are very suitable to repre-sent the structural relations of Chinese characters, where nodes stand for prim-itives (strokes here), and arcs describe relations between these primitives. It is easy to come to mind that stroke types are used as the features of the primitives. 58 Chapter 3 Relational Graph Representations of Chinese Characters However, what relations between strokes are able to represent the deformation-tolerated features of a Chinese character as exactly as possible? Considering wide variations of handwriting, we choose three spatial relations to be the basic-relational features. They are relation 1 "below/above", relation 2 “right of/left o f and relation 3 "intersect/don't intersect". More detailed description of these relations is given in the following. Let Ci and Cj be the geometric centers of stroke i and stroke j of a model Chinese character, respectively. A vector r^j = {ajj, a? , af^), i j , is used to represent the basic spatial relations from stroke i to stroke j where , a? G {0,1, 2,3 4} and a? € {0,1 2}. a}j = 0’ 1 2’ 3 and 4 indicate that c, is below, is above, may be below or above, must be below, and must be above cj, respec-tively. Also, a j = 0’ 1 2’ 3 and 4 indicate that c, is on the right of’ is on the left of, may be on the right of or the left of, must be on the right of, and must be on the left of Cj, respectively, a? = 0,1 and 2 indicate that stroke i and stroke j do not intersect, intersect, and may intersect or not, respectively. Here, a^j = 0 or 1 {k = 1,2,3) is termed the "should" feature, a* = 2 (A; 1,2,3) the “don't care" feature, and af = 3 or 4 (fc = 1 2) the "must" feature. Note that the "must" feature is not used for a^j. This is because many strokes in Chinese characters are easily written intersecting each other while they are not supposed to do so in standard writing, and on the other hand, two strokes that should intersect may easily written as two non-intersected ones. The relational graph representation of a model Chinese character is obtained by assigning suitable values to each relational vector Investigating hand-written Chinese characters, we learn that a relation between two strokes can be 2 The term "basic" means that more other relations may be added when necessary. 59 Chapter 3 Relational Graph Representations of Chinese Characters instable, stable or very stable, so we use the "don't care" "should" or "must" features to denote it. As an example, consider the character shown in Fig 3.8(a). Let Ci {i — 1,2 5) be the geometric center of stroke i. It is not difficult for a person familiar with Chinese character handwriting to find the fact that d is sometimes on the left and sometimes on the right of C2, Ci is above C2 and they don't intersect in very high probability. Thus we set a f j = 1, a j j = 2, and a\2 = 0 . Because the component “ ’ must be located on the left of the the "must" feature is chosen so afg, af* afg, a 3 and 035 n in Fig 3.8(c). component “ are all set to be 4. Other relation value assignments can be Fig 3.8(b) shows the graph representing the character in Fig 3.8(a). A node of the graph represents a stroke by containing the stroke number (the upper number) and the stroke type (the lower number). All the standard stroke types are shown in Table 2.2. Note that now a new stroke type 0 which is not included in the table, is used to denote a short stroke that is easy to be written as a stroke belonging to one of the stroke types 1-4. In Fig. 3.8(a), stroke 5 is such a stroke so its type is set to 0. There is a relation between vectors r^j = (aj^ , a? ) and r) = (aj,, aj,, a ^jx = = 0 O -•= Q-ij = a^. u a 60 I 2 3 4 5 1 (1,2,0) (2,4,0) (2,4,0) (2,4,0) 2 (0,2,0) (2,4,0) (2,4,0) (2,4,0) 3 (2,3,0) (2,3,0) (2,2,1) (4.2,0) 4 (2.3,0) (2,3,0) (2,2,1) (2 0’2) 5 (2,3,0) (2,3,0) (3,2,0) (2,1,2) (C) Figure 3.8: Stroke-based spatially relational representation, (a) A Chinese char-acter. (b) Complete relational graph representing the character, (c) Correspond-ing spatial relation matrix. A point on or near a stroke indicates the geometric center of the stroke. Different strokes are labeled with different numbers. A node of the graph represents a stroke by containing its number and type (the lower number). 61 Chapter 3 Relational Graph Representations of Chinese Characters (b) 4 3 5 2 Chapter 3 Relational Graph Representations of Chinese Characters where k = 1,2, and if 4 = 2. This property is useful for saving the memory space of a model graph base. For an input character, the computation of its graph includes extracting every stroke, identifying the type of each stroke, and finding the relation vector r'mn = ( NJ) + YI — A) + 7(A ^ NJ) + [ 7 ( ( n „ TUi) ( J i j , rrij)) + 7((ni,mi) ^ A) 4- ^ -/(A ^ (nj^mj)) (3.9) is termed a matching cost. Here 7(n,- — rij),^{ni — and 7(A —> rij) are node correspondence costs, and j({ni, nii) — (n), rrij)) 7((n‘ m,) —> A) and 7(A — (rij, rrij)) are arc correspondence costs. For stroke-based graph matching, they correspond to stroke and stroke relation correspondence costs, respectively. The assignments of costs to different stroke (type) correspondences are done mainly according to human knowledge of the stroke type variations. Moreover, the stroke type recognition method presented in Section 2.3 is also helpful for determining whether two strokes are easily confused. We show the model strokes in Table 3.1 again for convenient description. In Chinese character handwriting, if stroke A may easily be written like stroke B but not like stroke C, then we should assign lower cost to the former case than to the latter. For example, Chinese people often use type 2 strokes to 69 Chapter 3 Relational Graph Representations of Chinese Characters Table 3.1: 18 model strokes. Type Strokes Type Strokes I 1 " 10 7 2 11 y / 3 , Z 12 4 \ 13 5 : / 14 6 : \ 15 7 J 16 C 8 < 17 9 : L 18 stand for type 7 strokes, and vice versa. Type 1 strokes and type 4 strokes are also easily confused. Moreover, from the experiments given in Section 2.3 we know that (type 8 type 9), (type 6, type 12) and (type 11 type 15) are similar pairs. Table 3.2 summarizes all the stroke type correspondence costs. We mention again that stroke type 0 is used to denote short strokes of model characters, and the three frequently-used (not standard) input stroke types 16, 17 and 18, if detected in preprocessing, are split into their corresponding stan-dard strokes. In Table 3.2, there is a new stroke type 20. It is used to denote the unknown-type input strokes that are considered unlike any model stroke in the stroke type recognition. Because a stroke of type 20 is generally a multi-segment stroke, we assign smaller costs to its correspondence with multi-segment standard strokes. Table 3,2 defines the cost function 7(ni —> n where n^ A) and n) • are two nodes in G and Gj respectively. For the node deletion cost 7(rzi —> A) 70 Chapter 3 Relational Graph Representations of Chinese Characters Table 3.2: Costs associated with stroke type correspondences. 0 I 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 0 0 2 2 2 2 6 6 6 6 6 6 9 9 9 9 9 9 1 2 0 7 7 3 3 7 7 7 7 5 8 8 9 9 9 9 2 2 7 0 3 3 7 5 3 6 7 8 8 8 9 9 9 9 3 2 7 3 0 7 7 7 5 7 8 8 8 8 9 9 9 9 4 2 3 3 7 0 7 4 7 7 7 8 8 8 9 9 9 9 5 6 3 7 7 7 0 7 7 7 7 8 8 8 9 9 9 9 6 6 7 5 7 4 7 0 7 7 5 8 8 4 9 9 9 8 7 6 7 3 5 7 7 7 0 7 7 7 7 8 9 9 9 8 8 6 7 6 7 7 7 7 7 0 3 7 7 5 9 9 9 8 9 6 7 7 8 7 7 5 7 3 0 7 7 4 9 9 9 8 10 6 5 8 8 8 8 8 7 7 7 0 4 9 9 9 5 8 11 9 8 8 8 8 8 8 7 7 7 4 0 9 9 7 4 7 12 9 8 8 8 8 8 4 8 5 4 9 9 0 7 9 9 7 13 9 9 9 9 9 9 9 9 9 9 9 9 7 0 9 9 7 14 9 9 9 9 9 9 9 9 9 9 9 7 9 9 0 9 7 15 9 9 9 9 9 9 9 9 9 9 5 4 9 9 9 0 7 20 9 9 9 9 9 9 8 8 8 8 8 7 7 7 7 ^ / 0 71 Chapter 3 Relational Graph Representations of Chinese Characters 7(A —> fij), after some experimental tests, we choose lini A) 7(A — rij) = 5. (3.10) Let Tii A) and rrii A) be two nodes in graph Gi, and n j A) and rrij A) be two nodes in Gj. Then the relations from Ui to m^ and the relation from rij to TUj are denoted by vectors r„• ‘ = a j ^ ^ ^ , aj,^^, and njruj — i^jijmj > ‘ ^ nj J K respectively. The arc correspondence cost 7((n,-, TUi) —> {rij, rrij)) is defined as 4 7((n“m,) — in j .m j ) ) = ^n^mj. jb=l where u/1-4 are weighting factors and t] is defined by Table 3.3. Table 3.3 Definition of the function 77. (3.11) 0 1 2 3 4 0 0 1 0 0 M 1 1 0 0 M 0 2 0 0 0 0 0 3 0 M 0 0 M 4 M 0 0 M 0 In Table 3.3, = 0 suggests that the kth relation from n, to m, is compatible with the kth. relation from rij to m and , a^^^p = 1 or M means the two relations are incompatible. For example, if = 0 (denoting that the geometric center of stroke n, is below that of stroke m^) and aj,^^^ = 0, 1, 2, 3 and 4 (denoting that the geometric center of stroke n) is below, is above, may be below or above, must be below, and must be above that of stroke m ), then it is clear that the relation implied by dn^ m, = 0 is compatible with the relation 72 Chapter 3 Relational Graph Representations of Chinese Characters implied by = 0, 2 or 3, and is incompatible with the relation implied by a i^^ mj = 1 or 4. For the latter case, a cost (> 0) is assigned to this incompatible arc correspondence. Obviously, incompatible "must" relation correspondence should be punished by paying much higher cost, so we set M = 10. From the definitions of the edit operations on complete relational graphs, we know that arc deletions are caused by node deletions. So we set 0 to all the arc deletion costs 7((71‘ rrii) A) and 7(A —> (rij, mj)). Let us consider the matching cost in (3.9). Large weights wi-4 mean that arc correspondence costs play a more important role in graph similarity comparisons; on the other hand, small weights make the importance of node correspondences increase. We found in our experiments that i^i = 6,W2 = 6, W3 = 4 and W4 = 6 can result in satisfactory recognition rate. As some strokes of Chinese characters are easily written intersecting each other while they are not supposed to do so in standard writing, we set W3 a smaller value. Remark. The node and arc correspondence cost function 7 defined above is not a metric, which leads to the fact that the matching distance ^(Gi, Gj), in general is not a metric either. (For example, 77(0,1) ij(0,2) + 77(2,1) makes 7 not satisfy the triangle inequality.) However, distances are not necessarily metrics in pattern recognition problems [30 34 35]. In fact, many distances (or similar measures) proposed in the pattern recognition literature are not metrics such as those in [30 34 35, 56’ 69, 79, 83], but they may still be useful in comparing the similarity between objects. By the way, to our knowledge, no authors claimed that the distance measures used in their on-line Chinese character recognition methods are metrics. 73 Chapter 3 Relational Graph Representations of Chinese Characters 3.4.2 Assigning Costs for Segment-Based Relational Graph Matching There are only six segment types: type 1 “ ” type 2 “ type 3 “ Z ” type 4 " \ \ type 5 “ ’ and type 0 (denoting short segments) in the segment-based relational graph representation. Hence the assignment of costs (7(n,- — nj),ni ^ X^rij A) to node (segment) correspondences is relatively direct, as shown in Table 3.4. Table 3.4: Costs associated with segment type correspondences. 0 1 2 3 4 5 0 0 1 1 1 1 7 1 1 0 7 7 2 2 2 1 7 0 2 2 7 3 1 7 2 0 7 7 4 1 2 2 I 0 7 5 7 2 7 7 7 0 Now we consider the assignment of costs to node deletions. The segment-based recognition methods proposed in this thesis are used to recognize more freely written Chinese characters that in general consist of more connected and missing strokes. As mentioned previously, the preprocessing algorithms pre-sented in Chapter 2 cannot detect all extra segments. Therefore, it is very often that the segment number of an input character (after preprocessing) is different from that of its corresponding model character. We regard it reasonable that the segment deletion costs vary with the aode numbers of the graphs under matching. For example, deleting any one of the segments of the 3-segment char-acter “ 7 \ “ yields a different character or symbol, but for characters with more 74 Chapter 3 Relational Graph Representations of Chinese Characters segments (e.g., 12), it may still remain recognizable after deleting one or two of its segments. Obviously, higher cost should be paid for the former case than the latter. Therefore, the node deletion costs are defined by 7(71, — A) = 7(A — rij) = k{p), (3.12) where Ui and n j are two nodes of graph G, and Gj under matching, respectively; p is the node number of the model graph (Gi or Gj). The cost k(p) is not difficult to choose with the help of some experiments. For example, when 9 < p < 15, we take k(p) to be 4 and when 1 < p < 5, we may take k(p) to be 5. The arc (segment relation) correspondence costs (7((ni’m‘ (rij, mj}}, (ui, rrii) ^ A, (rij, nij) • A) are defined the same as those in the last section, (see (3.11) and Table 3.3), and the arc deletion costs are also set to 0. 3.5 Summary In this chapter, we have formally defined the complete relational graphs and the distances for measuring the similarity between two graphs. With such graphs, we have proposed several relational representations for on-line Chinese character recognition. We have also dealt with the problem of assigning costs to node and arc correspondences in graph matching. The stroke-based representations may be used to recognize relatively neat Chinese character handwriting while the segment-based representations will ease the recognition of more freely written characters. These representations have the following advantages: 75 Chapter 3 Relational Graph Representations of Chinese Characters • The representations incorporate the human knowledge of Chinese charac-ters and can reflect their features well (except some very similar character pairs). In a complete relational graph, nodes describe primitive (stroke or segment) types and arcs represent the spatial and temporal relations between any two primitives. The proposed "don't care", "should" and "must" relational features allow us to represent unstable, stable and very stable primitive relations conveniently. Relations between any two prim-itives give much information and are very beneficial to the matching pro-cedures, which will be discussed in the next chapter. • The graph representations are directly based on strokes or segments. To obtain the representations, examining whether a stroke or segment be-longs to some component is not required. However, the representations in [18 68], as mentioned in Section 1.2 need to correctly extract components of Chinese characters first. The recognition method based on the represen-tation in [13] also need to find components before performing recognition of a character. In fact, wide handwriting variations make it very difficult to extract components of Chinese characters at a high rate of success. In 16], the authors adopted only the relations between segments within the same components in their graph representation. This results in two short-comings: (1) some relations represented in an input graph may not appear in its corresponding model graph, and vice versa; (2) most of the relations between segments are not utilized. • The spatial and temporal relations between primitives are, at the first time, unified into the graph representations, which fully captures the on-line information of handwriting. The use of the primitive order relations 76 Chapter 3 Relational Graph Representations of Chinese Characters enhances the discrimination ability of the representations and helps to speed up the graph matching. Because of the "don't care" feature, the representations can tolerate common stroke order deviations. • If the weight w^ in (3.11) is set to 0 in graph match, then the stroke order relation will be ignored and our recognition methods presented in the next two chapters will be stroke order free. Creation of a model character base for our recognition goal mainly depends on the human knowledge of Chinese characters, and thus is a relatively heavy task. It can be eased by constructing the graphs of components of Chinese characters first and then combining several component graphs to form the whole graph of a character. Parts of the results presented in this chapter have been published in [58’ 59’ 60 61 62, 63 . 77 Chapter 4 A State Space Search Method 4.1 Introduction In the last chapter, we have introduced several graph representations and de-fined the matching distance for on-line Chinese character recognition. Now the problem of recognition of an input Chinese character can be transformed into a problem of graph matching. The term graph matching, as conventional, is used to denote the process of finding the distance between two graphs. Unfortu-nately, so far there are no efficient algorithms for our graph matching problem. Given two graphs G, and Gj both with n nodes, a naive algorithmic approach to calculating the distance ^{Gi, G f j is to generate all n\ permutations of the nodes and test them for being a solution (suppose A, = A = 0, and Aj = Aj = 0, i.e., no node and arc deletions are carried out), then this algorithm has the computational complexity at least 0(nl) . Problems of such kind, which allow noise or structural deformations in practi-cal object recognition, are extensions of the NP-complete subgraph isomorphism 78 Chapter 4 A. State Space Search Method problem in graph theory [29, 46’ 47] and are also NP-complete [33 98). All the problems in this class are believed to be intractable, i.e., no polynomial algo-rithms exist for one of the problems. If there is an efficient algorithm for some NP-complete problem, that is to say, the worst-case complexity of the algorithm is bounded by a polynomial function of the problem's parameters, then there is an efficient algorithm for every problem in the class [25 75 . For our on-line recognition problem, because large categories of Chinese char-acters and the real-time recognition requirement, an algorithm is applicable only when it recognizes an input Chinese character with less than three sec-onds on some computer.^ The less computational power the computer has, the fast the algorithm has to be. In addition, since we are dealing with a NP-complete problem we cannot expect to obtain fast recognition speed by running an exponential-time algorithm without using any heuristic information. Let us consider an example. As mentioned above, the exhaustive search for the simplified calculation of distance ^(Gi, Gj) needs at least 0(n\) time-Assume that after a preclassification stage, the graph of an input character has to be matched with 300 model graphs, and the node numbers of all these graphs are 12- Thus the computational time to recognize the input character is at least A X 12! X 300 (ss X 1.4 X 10^^), where A is the time required by a computer to perform one basic calculation. If A denotes one addition operation, then the recognition of an input character will take at least 15000 seconds (> 4 hours) on a 166MHz PC/Pentium! It is clear that we have to seek efficient approaches 1 In general, a Chinese needs about three seconds to write a ten-stroke character. An on-line recognition system can be designed to work in the way while one is writing a character, the system is recognizing the characters preceding it. In this case, the system should not take more than three seconds to finish the recognition of an input character. 79 Chapter 4 A. State Space Search Method to our graph matching problem. In this chapter, we propose an efficient state space search method for the problem. The rest of this chapter is organized as follows. In Section 4.2, the graph matching is formulated as a search problem in a state space tree. The A" algorithm that is employed to perform the search is presented in Section 4.3. Several schemes for increasing the search efficiency of the A* are proposed in Section 4.4, including a lower bound estimate, a tree pruning strategy, and criteria for stopping the A* algorithm. The experimental results are provided in Section 4.5. Comparisons of our segment-based recognition method with several other studies are presented in Section 4.6. The summary of this chapter is given in Section 4.7. 4.2 State Space Formulation of the Graph Matching In an approach to problem solving by using state space search, a state space is a representation that consists of nodes and links, where each node denotes a state that is a description sufficient to determine the future, and each link connecting a tail node to a head node denotes a possible one-step transition from one state to another state. The goal state of a state space is where we want to be. The procedure to solve a problem is to find a sequence of transitions that leads from some initial state to the goal state. Now we transform our problem of calculating the distance between two graphs into a state space search problem. Let Gi = {Ni, Aj) and Gj — {Nj, Aj) be two graphs. The distance from G, 80 Chapter 4 A. State Space Search Method to Gj which is defined in Definition 3.7, is ^{GuGj) = min{/?(/Ar />i)|(/;v yU) is a matching from Gi to Gj}. If the arc deletion costs are all set to 0 (see Section 3.4), the matching cost /5(/IV, / A ) is then expressed by /A) = E — rij) + ^ -yiui — A) + ] X-^-njeQs + E (nj,mj)) (4.1) (n,,mi)->(nj,mj)6Q4 where /^v, and Q1-4, are defined in Definitions 3.5, 3.6 and 3.7, respectively. Definition 4.1 Let G, = (iVj, Ai) and Gj = (Nj, Aj) be two graphs, and G\ (iV/, i4J) and G'j = (ATJ, AJ) be two subgraph^ of Gi and Gj, respectively. A state is denoted by a set of node correspondences: S = {nj — TijlfJ^(Tii)= rij, rii € Nl U A,, nj G Nj U Aj}, where f'^ is a node mapping from G\ to G'” Nl C Ni and Nj C Nj. When a state covers all the nodes in G, and Gj, i.e., Nl Ni and Nj = Nj, it is called a goal state,* otherwise, a middle state. The initial state is a state where there are no any node correspondences. If S = [ui — n j \ f ' ^ ( n i ) = rij, 71, € Nl U Aj, rij G iVJ U Aj} is a middle state, a new state Si generated by expanding S is defined by Si = S U {n^ —> nji}, where nn e Ni U A,, Uji G Nj U Aj, nn N , riji 0 N- and n,i — riji \ — An example of a state space for matching between two graphs is shown in Fig. 4.1. The initial state, middle states and goal states are indicated by “0 ’ ‘V and “•” , respectively. The state space is actually a tree, so it is also called a state space tree or search tree, and a node^ of the tree denotes a state. 2The definition of an induced subgraph (or subgraph for short) of a graph is given in Definition 3.3. Moreover, we consider that an empty graph is also one of the subgraphs of a graph. 81 Chapter 4 A. State Space Search Method Gi Gj ,2 • L,>A.} Figure 4.1: A state space tree for matching between G and Gj. Symbols "o", “•’ and “•” denote the initial state, middle states and goal states, respectively. 82 Chapter 4 A. State Space Search Method From Definition 4.1 and Fig. 4.1, we can see that a goal state corresponds to a matching from Gi to Gj, where is an arc mapping led by f ^ (see Definition 3.6); a middle state corresponds to a matching (/J^, /a ) from subgraph of G: to subgraph G'j of Gj, where / is an arc mapping led by Definition 4.2 The cost of a state S = {rii —> nj\f'[^{ni) = ly rii €. N'/UAj, rij 6 Nj U Aj} that corresponds to the matching (/Jv> I'A) ^^ calculated by + Yi 7(( t mi) — (n mj)) (4.2) where (/Jv> / i ) determines the sets i.e., Q'l is the set of rii — n^, n,- G rij e N'j Q'2 the set of rii A,nj € N ; Q^ the set of X ^ Uj^rij 6 A/}; the set of (rii, i) — (rij, n,, rrii G N-, nj, mj e iVJ., n^ ^ m*; /a ) is actually a subgraph matching cost on condition that all arc deletion costs are set to 0. When (/Jv, Ia) = (/N, Ia), fHf'N, f'A) is a graph matching cost. Therefore, we define a best goal state in a state space tree as a goal state with the minimum cost among all the goal states. There may be several best goal states in a tree. Now the problem of computing the distance from Gi to Gj is transformed to the problem of finding a best goal state in a state space tree. We will use the heuristic algorithm, A• to perform the search. I^f we say such like "nodes of a graph", "node mapping" and "node correspondences", a node is referred to the node of a graph; otherwise, in this chapter, it stands for a node of a search tree. 83 Chapter 4 A. State Space Search Method 4.3 The A* Algorithm The A' algorithm is a popular heuristic search algorithm in problem solving, whose purpose is to find the cheapest path cost in a network [9’ 31 71]. For our application, it is used to find the best state in a search tree. Let u be a node in a search tree, g(u) be the cost of the path from the initial node to u, h*(u) be the minimal cost of a path from u to a goal node or a best goal node, and h(u) be an estimate of h'(u). Note that there is only a path from the initial node to another node in the tree. Let the state of node u corresponds to a matching (fj^^^, J . Thus we define 9(u) = (4.3) Let V — {u} be the set of the goal nodes to which there exist paths from u, and let = {“ }. (4.4) h*(u) is then given by h*(u) = g(v')-g(u). (4.5) The evaluation function of the A* is defined as /(2i) = g(u)-i-h(u). (4.6) It is a cost estimate of the minimal cost path constrained through node u. We call / (u ) the estimated value of node u, and h a heuristic function. The computation of k(u) is according to some problem-dependent information. h(u) > 0 is always assumed. The A* algorithm for our tree search problem is presented below. 84 Chapter 4 A. State Space Search Method The A' algorithm Step 1. Create a search tree, consisting of only the initial node. Put it on a list called OPEN. Step 2. If the first node on OPEN is a goal node, exit with the estimated value and the state of the goal node. Step 3. Remove the first node from OPEN. Expand it, generating the set of its successors. Calculate the estimated values of the successors using (4.6). Add these nodes to OPEN, and arrange these new nodes and the old nodes on OPEN in increasing order of their estimated values. (Thus the node having the smallest estimated value is at the first.) Step 4. Go to Step 2. The above algorithm is a special version of the A* for a search graph [31, 71]. The A* algorithm is always convergent for finite search graphs [71]. Two definitions and two theorems in [71] are given in the following. Definition 4.3 A search algorithm is called admissible i f , for any search graph, it always terminates in an optimal path from an initial node to a goal node when-ever a path from the initial node to a goal node exists. For a search tree in our application, the goal node in an optimal path is one of the best goal nodes we search for. 85 Chapter 4 A. State Space Search Method Theorem 4.1 The A* algorithm is admissible i f , for every node u of a search graph being examined, the following inequality is satisfied: h(u) < hr[u). (4.7) Definition 4.4 A heuristic function h is monotonlc i f , for every node u and any of its successors w, h{u) - h(w) < c{u,w) (4.8) with h{v) = 0, (4.9) where v is any goal node, and c{u, w) is the path cost from u to w. Theorem 4.2 If the monotone restriction is satisfied, the estimated f values of the sequence of nodes expanded by the A* is nondecreasing. In our tree search problem, w) = g{w) — g{u). Theorem 4.1 guarantees that the A* algorithm can find the best node if (4.7) is satisfied. Theorem 4.2 is useful for reducing computational time when the A* is used in our on-line Chinese character recognition, which will be described in Section 4.4.3. The search effectiveness of the k* algorithm relies heavily on how precise the estimate of/i* is. If h{u) = h*{u), the fewest nodes are expanded. Setting h{u) 0 assure admissibility but results in an inefficient breadth-first search. Although great efforts have been made to find good heuristic functions, in general, precise estimates of h'{u) are quite difficult for most applications [9, 44, 71, 76, 77, 78, 89 90, 101], and thus the A* algorithm has exponential complexity [9 104, 105 . From a probabilistic point of view, Pearl has made a thorough study about the relations between the precision of the heuristic estimates and the average complexity of the A* in [77 . 86 Chapter 4 A. State Space Search Method 4.4 Schemes for Speeding up the A* Algorithm As mentioned above, the search efficiency of the A* algorithm is not satisfactory (or is even pessimistic). However, the schemes proposed in this section, which do not guarantee that the A* always finds optimal solutions, can greatly speed up the A* when it is used in our on-line Chinese character recognition. 4.4.1 A Lower Bound Estimate The use of problem-dependent heuristic information, which is represented by the estimate function h, will make the A* expand fewer nodes than a search with h = 0. From Definition 4.2 and (4.3), we know that the cost of a node in a search tree is equal to a subgraph matching cost. Let a middle node be u, and two graphs under matching be Gi = (A ,^-, Ai) and Gj == (iV),/!)). Let the state of u correspond to a matching (/J^u / \u) from subgraph G -- (iV/, -4J) of G, to subgraph G'j = (iVj, A'j) of Gj. We use the costs of node correspondences from subgraph G" to subgraph. G'j as an estimate h(u) of h*{u), where G" = Ni \ Nl (G'J = = Nj \ N'” respectively) is the subgraph of Gi {Gj, respectively) by deleting the nodes of G\ from G, {G'j from Gj respectively) and deleting all the arcs connecting these nodes. h(u) is expressed by = min^ ^j) + H 7( —A) + E (4.10) 87 Chapter 4 A. State Space Search Method where /{J^ is a node mapping from G" to G" and it determines the sets Q"_3(it): Q'{{u) is the set of th Uj, n, € iVf, Uj e N"; Q'i^u) the set of n, A, rii e JV"; Qsiu) the set of A -v nj, n e N';. Theorem 4.3 Let u be a node in a search tree. h(u) defined in (4.10) is a lower bound on h*{u) defined in (4.5)’ i.e., h(u) < h*{u). Proof. Let V = {u} be the goal nodes to which there are paths from node u. By (4.4) and (4.5) we have "•(u) = - g(u) = m m s I Z — Tij) + ‘ - > A) “ ln,~mj€C?i(t ) n,-rAGQ2(u) + 7(A Tij) + [ 7((ni rrii) — (nj, m))) \ A-)>nj€Q3(u) {rt”mj)€C?4(t ) J - \ ^ 7(ni — n,) + — A) njeQ'i(u) n,— + Z rij) + m‘) — (nj, m,)) I ’ A—nj€Q'3(ti) (nj,mj)€Q'4(u) J where Qi-4(v) are defined in Definition 3.7 and Q[_4(u) are defined in Defini-tion 4.2. Let us look at Fig. 4.2. There is only one path from the initial node to node u and all the paths from the initial node to nodes in V go through u. Therefore Q'MQQ.iv), ’ 1 2 3’4. Therefore = mms H — n j ) + ^ 7(n,- ^ A) 88 Chapter 4 A. State Space Search Method initial node • • • • i i • … • V Figure 4.2: A search tree, u is a middle node and u is a goal node that can be reached from u. + E 7(A-> rij) + Y . (4.11) Since 7((71“ m,) — (rij, rrij)) > 0’ we have "•(u) > mm< — )+ 12 — A) + E 7 ( A - > n , ) i (4.12) A— eQ3(t;)\Q'3(u) J The right term in above inequality is an alternative expression of h(u) in (4.10). Thus the theorem follows. • Theorem 4.4 The heuristic function h defined in (4.10) is monotonic. Proof. Case 1. Let u be a middle node in a search tree and w be one of its successors generated by expanding u, as shown in Fig. 4.3. Let the states of u 89 Chapter 4 A. State Space Search Method Figure 4.3: A search tree, u is a middle node, w, x and y are successors of u. and w be S(u) and respectively. Then by definition 4.1 we have S(w)= S{u) U {Uia Tija}- Therefore c(u, w) - g(w) - g{u) = P U ' ^ . J A J - P U m u J A J = { E nj) + ^ 7(ni — A) + H 7(A — nj) + rUi) (uj, m,)) \ - { E - ^ r i j ) + T ( n i — A) + 7(A TIJ) + [ 7((n“ m,) (n)-’ MJ)) > = H 7 ( " » — r i j ) + Y . 7 ( " i — A) Uj) + Y. l{(ni,mi) {rij.mj)) ( 4 . 1 3 ) > I ] 7 ( n i U j ) + Y . — A) 90 Chapter 4 A. State Space Search Method Y1 7(A — rij] = 7 ( a — ( 4 . 1 4 ) ( 4 . 1 5 ) By definition, h{u) have = = m m + E -yi^i — ) + E 7 ( A - > n , ) l , = min I — rij) + — ) « lri,-^njeQ'i'(w) n.->A6Q'j'(u7) + E nj) I where / ^, is a node mapping from G = ( A ^ , ^ ^ to = and rir is a node mapping from G'^ = (iV , ^ ) to G ; = …A';J (see the definition of h{u) in (4.10) for more detailed description). Suppose is the optimal node mapping from G"^ to Gj^ such that h{w)= — n - + E 7(ni A) + E 70 Z 7 ( A — nj), where Qi^^iw*) are determined by / j j^ , . If rz, 6 N"^ and nja € then = N^w U {"ia} and = N'J^ U {rija}- Therefore, we can find a node mapping /JJ^. from G' to such that 7(ni — rij) + Yi — A) Z 7 ( A — rij) = — ^jc E — rij) + 91 Chapter 4 A. State Space Search Method where Qi'_3(u*) are determined by which suggests h{u) < E rij] + S 7(A n, =liriia — Tlja) + h{w). • i — A) Thus, it follows that h(u) - h(w) < 7("iQ rija] < c(u,w). (4.16) (4.17) The same result nja e be obtained if 6 iV". and Uja = A or if n,a = A and Case 2. Let w and i be successors of u, where w is generated by expanding u and X is generated by expanding w (see Fig. 4.3). Then we have h(u) - h(w) < c(u, w), and further h(w) - h(x) < c(w, x] h(u) — h{x) < c(u, ly) -f c(w, x] = g M -gW + gix) -9(w) = - 9M —c(u, r ) . The same conclusion can be obtained when y is any successor of u. Finally, for any goal node v, by (4.10), it is clear that h(v) 0. Therefore the theorem follows. • 92 Chapter 4 A. State Space Search Method Nilsson showed that, if hi{xi) < h*(u), h2(u) < h*{u) and hi(u) < /i2(u), then the A J using hi(u) expands at least as many nodes as does the A using h2(u) [71]. From (4.10) we know that only the costs of node correspondences from G" to G" are employed to estimate h'{u). The costs of arc correspondences from G" to G" are not considered. Such a scheme results in a relatively simple calculation of h{u), but the heuristic power of h{u) is not good enough. If the costs of arc correspondences are also used to estimate h'{u), the calculation of of the estimate might become another NP-compiete graph matching problem, which is also time-consuming. Even though we use only the costs of node correspondences to estimate h'{u), the calculation of h{u) is not trivial. The optimization problem in (4.10) can be transformed to a minimum weight matching problem in a weighted bipartite graph [75 . A graph B = (Vi U V2, E) with vertex set Vi U V2 and edge set E is called a bipartite graph if all its nodes can be partitioned into two subsets, Vi and V ,^ such that every edge in the graph connects some node in Vi to some node in Vo. If a bipartite graph has weights associated with its edges, it is called a weighted bipartite graph. A matching'* in B is a subset of edge K C E such that no two edges of K are adjacent. Here we use the terms "vertex" and "edge" for bipartite graphs instead of "node" and "arc" to avoid possible confusion. Now we consider an example. Let be a node mapping from G"= (N ', A' ) to G' = (A7, A' ) (see (4.10)). Suppose N;' = {n,i n‘2 ",3} and 4 Note the difference between a matching defined here and the term "matching" we use previously. The former denoting a matching in a bipartite graph is a conventional term in graph theory while the latter defined in Definition 3.7 represents a node mapping and an arc mapping from a complete relational graph to another complete relational graph. 93 Chapter 4 A. State Space Search Method VV(Vi,V2 ) V, V-(a) V, Figure 4.4: (a) A bipartite graph B = {Vi U V^2 ) where V^ = N^' U A,, I Ail = liV/l, V2 = N' U Aj, and |Aj| = (b) A matching in the bipartite graph. N" = {riji, nj2}. All possible node mappings can be represented by the match-ings of the weighted bipartite graph shown in Fig. 4.4(a) where Vi = N" U A,, I A,-1 = Vb = iVj'uA” and |Aj| = A vertex in Ki is connected to all the vertexes in V2. We denote by e(vi, V2) an edge connecting Vi € Vi and V2 € V2. A weight associated with e{vi, V2) is defined as the cost of node correspondence 7(1 1 —V2) when e(vi,v2) • e{A, A) and as 0 when e{vi, V2) = e(A, A). Now the calculation of h(u) in (4.10) is equivalent to the problem of finding a matching in B such that the sum of the weights associated with the edges of the matching is the smallest. The polynomial-time Hungarian method with complexity 0(|V'i|^) ( = 0 ( ( | i V f | + lA^i'l)^)) is a solution to this problem [75]. A matching in B of 94 Chapter 4 A. State Space Search Method Fig. 4.4(a) is shown in Fig. 4.4(b), which corresponds to such a node mapping: {nil — rii2 A, 71,3 — ^ji}-The effort to computer /i(u) is one of the important factors that influence the search efficiency of the A' [71]. In our experiment, we found that if for every generated node in a search tree, the Hungarian method with complexity + lATj 1)^ ) is used to calculate h{u), the speed of the A* is too slow to accept. Therefore, we propose the following greedy algorithm to approximately calculate h{u). Greedy algorithm for calculating h{u) Input: Node sets N" and N". (comment: suppose \N"\ < |iVj'|) Output An approximate value of h. begin h := 0; while N ' # 0 do begin choose a node rii in iV"; remove n, from N"; find a node n) in N" such that — Tij) = min {7{nj — n^)}; if 7(ni Tij) < 7(ni — A) then begin h := h + 7(ni — rij); 95 Chapter 4 A. State Space Search Method remove rij from iV" end else h :— h + A); end while # 0 do begin choose a node n/ in N"\ h := k + 7(A Til)] remove rti from N": end end It is not difficult to analyze the running time of the greedy algorithm. Assum-ing \N"\ < The main computation is, for every remaining node in iV", to find a remaining node rij in N" such that 7(72^ — Uj) = —> HA )}. This requires 0{\N"\. |iV"|) time, which is also the complexity of the algorithm. Remark. Let us denote the estimate of h*{u) by h'{u) obtained with the greedy algorithm. Since h'(u) is an approximate value of h(u) defined in (4.10), we have h'{u) > h{u). Theoretically, there exists a possibility that h'(u) is not a lower bound on h*{u) when h'{u) > h{u). However, by (4.11) and (4.12) in the proof of Theorem. 4.3 we have h*{u) = h{u) + ^ 7((… m) -> {rij, nij)), 96 Chapter 4 A State Space Search Method where the last term is the sum of some arc correspondence costs. Because of this terra, we observed in our experiments that h'{u) < h'{u) holds for almost all nodes w in a search tree. Even if h'(u) > h'(u) for some nodes u, it is not meant that the A* must not find the best goal node. Nilsson has pointed out that heuristic power can often be gained at the expense of admissibility by using some function for h that is not a lower bound on h* [71 . 4.4.2 A Tree Pruning Strategy In Section 4.5 we will give some experimental results to show that the A' algorithm using only the evaluation function f(u) = g{u) + h{u) to guide its search is too slow in on-line Chinese character recognition. To increase the search efficiency of the A• we propose a tree pruning strategy that imposes geometric position constraints on strokes of Chinese characters and thus avoids expanding lots of nodes that are very impossible to be located in the optimal path from the initial node to the best goal node in a search tree. Let us look at the two characters shown in Fig. 4.5. When one is asked to identify between the two characters the stroke pairs that he/she considers compatible, it is easy for he/she to get the correct answer: {1 —2, 2 3, 3 (b) Figure 4.5: A model character (a) and its handwritten version (b). Chapter 4 A. State Space Search Method initial node uio« depth I Mil "12 "13 «I4 11-2.2-6) "15 Figure 4.6: A partial tree for matching from the character character in Fig. 4.5(b). depth Fig. 4.5(a) to the 4 4 — 1 5 — 5,6 — 6 7 — 8 8 — 7 9 9}. The reason why we can find the answer easily and quickly is that we have a bird's eye view of the two characters. A tree search is hard just because a search algorithm does not have this function. Consider the partial search tree (Fig. 4.6) for matching from the graph of the character in Fig. 4.5(a) to the graph of the character in Fig. 4.5(b). The states of the nodes are shown near the nodes, which represent corresponding stroke correspondences. At depth 1 nodes wi_io are generated by expanding the initial node. Determining which node will be expanded next is according to the estimated f values of the nodes. For this example, the A' algorithm will find f{u2) = /(U3) = / = / (Ue) = / ). Thus every node in the set {u2, u^, u^, uq, ug} may be a candidate to be expanded. Similarly, at depth 2 the calculated estimated values /{uu) , /(W12), /(wia), /(UH), and /{uis) are all the same. Node un that is in the optimal path has 98 Chapter 4 A. State Space Search Method 0 (c) Figure 4.7: (a) A Chinese character and the smallest rectangle ABCD surround-ing the character, (b) Geometric illustration of Do_3(2). (c) 8 directions. no priority over nodes Ui2_i5 in being expanded first. We have observed that in the matching between a model character and its handwritten character having stroke type and relation deformations, the A* tends to spend much time to discriminate among the paths whose costs do not vary significantly, and thus it is very possible that all nodes at depth 1 are expanded in order to find the optimal path. However, glancing at Fig. 4.5 we are easy to know only U2 and uii in Fig. 4.6 are located in the optimal path. The objective of the pruning strategy presented below is to add more or less the function of this bird's eye view into the A* algorithm. In on-line recognition, an input Chinese character as a whole can be regarded as no rotation variety. Hence a lot of information about the geometric positions of strokes of the character may be used to assist the A* algorithm when searching. Fig. 4.7(a) shows a Chinese character and the smallest rectangle ABCD that surrounds it. What are the stable geometric position features of the strokes of the character in daily handwriting? Intuitively, strokes 1 and 2 are written near the upper side of the ABCD; strokes 3-7 in the middle; stroke 8 near the 99 Chapter 4 A State Space Search Method lower side or the lower-left corner; stroke 9 near the lower side or the lower-right corner. Now we formulate these character- and stroke-dependent features in the following. Let abed be the smallest rectangle that surrounds stroke j of an input char-acter with t strokes (Fig. 4.7(b)). Eight directions shown in Fig. 4.7(c) are used to denote the directions of eight distances Dq-tO ), where D _3(j) are distances from aio A,b to S , c to C, and d to D, respectively, as shown in Fig. 4.7(b), and Z>4_7(j) are the distances from the geometric center of abed to the respective four sides of the ABCD. A notation od(Dq(j)), q € { 0 , 1 , 7 } , is used to denote that Dq(j) is the od{Dq{j))-th. smallest distance among … Dq(t)}. For example, od(Dq{j)) = 1 means that the distance Dq(j) of stroke j is the smallest, and od{Dq{j)) = m,m < t, means that there are m — 1 distances among Dq{t)} which are smaller than Definition 4.5 The geometric position features (GPF) of the strokes of a model Chinese character with s strokes is defined as a set of s 3-tuples GPF = {(d“ yi)\i = 1 2,… s}, (4.18) where di G { 0 , 1 , 7 } denotes one of the 8 directions, and x , and jji are two end points of the integer interval [xj,?/,], Xi, xji € { 1 , 2 , 5 } . {di, x,, i/i) gives a geometric position constraint on input strokes in the sense that only the input strokes j 's, satisfying x, < od(Dd,{j)) < y,, can have the chance of being as the node (stroke) correspondences i j or j —i in matching. A model stroke i and an input stroke j are called compatible if Xi < od{Dd^(j)) < yi. An example is given for better understanding of the GPF. For the character Chapter 4 A. State Space Search Method shown in Fig. 4.7(a), a possible assignment of its GPF is GPF = {(4 1 2) 4,2’ 4) (4 3’ 6) (0 1 4), (4 4 8) (6, 4,7), (6, 2, 5), (6,1,3), (6,1,2)}. (4.19) Here (di,xi,yi) = (4,1,2) denotes that the two input strokes j i and j2 written at the top of an input character and satisfying 1 < od{D^{jk)) < 2, k = 1,2 are compatible with stroke 1 of the model character. Now, for each model Chinese character, in addition to the features of its stroke number, stroke types and relation matrix, a new feature GPF is added. For an input character with t strokes, we not only recognize its t stroke types find the relations between any two strokes, and create its relational graph, but also calculate its od(Dg{j)), j == 1 2 ’ t ; q = 0,1,7. The GPF can be used to prune the search tree efficiently. When searching the tree for the optimal matching from graph G, = (iV“ Ai) of a model character to graph Gj = (Nj, Aj) of an input character, the A* algorithm runs with a pruning operation inserted. The generation of a successor node in the tree means that besides the node mapping from subgraph G: = (N!’ A'j) of Gi to subgraph G'j = (iVJ, o i G j in its father node in. the tree, a new node (stroke) correspondence i —> j is yielded, where i € {Ni \ iV/) U Aj, j € (Nj \ iVj) U Aj, and i j ^ X —r X. Now suppose i ^ X and # A. Let the fth element of the GPF of the model character be (d,-, x,, yi). If x, < od(Dd,(j)) < Vu i.e., the stroke i is compatible with stroke j , then the newly-generated node will be put on the OPEN of the A*; otherwise, the node is pruned away. For the partial search tree in Fig. 4.6, with the GPF in (4.19), since {di,xi, yi)= (4,1,2) and , 2,1/2) = (4,2,4), the tree after pruning will become a much 101 Chapter 4 A State Space Search Method initial node depth 1 depth 2 Figure 4.8: A partial tree obtained by pruning the tree in Fig. 4.6. All possible nodes at depths 1 and 2 are shown. smaller tree as shown in Fig. 4.8, where all the nodes at depths 1 and 2 are given. As can be seen, only three nodes at depth 1 have the chance to be ex-panded but ten nodes do before pruning. Also expanding a node at depth 1 in Fig. 4.8 generates three or four successors, while expanding u“ i e {1 2,… 9} in Fig. 4.6 yields nine successors and expanding Uio yields ten. Comparing the two trees, we see that the nodes uq and uis in Fig. 4.6 which are very possible to be expanded but are impossible to be located in the optimal path from a human view, now no longer appear on the OPEN of the A*. In this sense, the A* has more or less the function of a bird's eye view. The above content of this section discusses the tree pruning strategy for the stroke-based graph matching problem, where the GPF is the geometric position features of strokes of model characters and is used to impose constraints on input strokes. The same idea is also suitable for dealing with the segment-based graph matching problem. But the GPF is the geometric position features of Chapter 4 A State Space Search Method segments of model characters and is used to impose constraints on segments of input characters. For the character shown in Fig. 4.7(a), its GPF in (4.19) can be used in both the stroke-based Chinese character recognition and the segment-based Chinese character recognition since the strokes of the character are also the segments of the characters. By the way, it is unnecessary for the Chinese characters with less than five strokes (or segments) to have the GPF. The search trees used for matching between these characters are small. R e m a r k . The construction of GPF for each model is based on the human knowledge of stroke (segment) structure of Chinese characters. Obviously, it is not unique and is character- and stroke-dependent. We hope that for the GPF = = 1 , 2 , 5 } of a model character with s strokes, inter-vals [ii’yil’i = 1 , 2 , s , are designed as small as possible. In the case of Xi — yi, i = 1,2, ...,5, the fastest search is obtained. However, considering wide stroke variations in daily handwriting, the GPF of a character should be designed carefully. Too strict constraints on stroke positions may result in in-correct recognition. Therefore, in the construction of GPF, we prefer to give looser constraints to facilitate higher recognition rate. 4.4.3 Criteria for Stopping the A* Algorithm The A* algorithm will stop if it finds a best goal node in a search tree. The best goal node corresponds to the optimal matching between two graphs. A fact in Chinese character recognition is that the number of model characters similar to an input character are much less than that of the other characters. It is unnecessary to obtain the final optimal matching between two characters (in Chapter 4 A. State Space Search Method other words, we may terminate the A* before it reaches the best goal node) if we know they are dissimilar while searching. Here, “two similar characters" means that the distance between the graphs of the two characters is relatively small. On the contrary, the distance between the graphs of two dissimilar characters are expected to be large. Theorems 4.2 and 4.4 can help reduce much search time. From Theorem 4.2 we know that if a heuristic function h satisfies the mono-tone restriction, the estimated / values of the sequence of nodes expanded by the A* is nondecreasing. The estimated value of the best goal node equals the distance for which the A* searches. Theorem 4.4 states that the heuristic func-tion h in (4.10) is monotonic. However, before giving the criteria for stopping the A•’ we would like to clarify whether the approximate values h'{u) of h{u) obtained with the greedy algorithm presented in Section 4.4.1 is also monotonic. Let u be a node in a search tree and w be a successor of u generated by expanding u. Let h'{u) be the estimate of h*{u) and h'{w) be the estimate of h*(w), obtained by the greedy algorithm. By (4.13)-(4.17), we have c{u, w) = j{nia Uja) + ^ j{{ni, rrii) — (nj, rrij)), (4 ,20) and h(u) - h(w) < 7… ,-a — rija) < c(u, w). (4 .21) Since h'(u) and h'(w) are the approximate values of h(u) and h(w), respectively, we cannot derive that h' is also monotonic from the statement of h being mono-tonic. However, because of the right most item in (4.20), we have observed that h'{u) — h'(w) < c(u, w) holds in all the experiments we did for checking this 104 Chapter 4 A State Space Search Method monotone restriction. We have also observed that the estimated / values of the sequence of nodes expanded by the A* are indeed nondecreasing. Three examples are given in Figs. 4.9-4.11. Figs. 4.9(a)-4.11(a) show three pairs of characters to be matched. The curves in Figs. 4.9(b)-4.11(b) illustrates the aondecreasing characteristic of the estimated f values with respect to the sequences of nodes expanded by the A•’ when the A* searches for finding the optimal matching between two characters of each pair. The A* algorithm runs with the pruning operation introduced in the last section. In the experi-ments, the primitives used for representing characters are strokes and thus the matching procedure is a stroke-based recognition method. The two characters in Fig. 4.9(a) are the most similar. The two characters in Fig. 4.10(a) belong to the same class but the input character has a connected stroke. The two char-acters in Fig. 4.11(a) belong to different classes and are most dissimilar. From Figs. 4.9(b)-4.11(b), we can clearly see that in these experiments, the more dis-similar the two characters under matching, the greater their matching distance and the more the nodes expanded by the A* to reach a best goal node. By the way, the number of nodes generated by the A* is greater than the number of nodes expanded by the A* in a search. In general, expanding a node generates several successors of it in a tree. For the three examples, the node numbers generated by the A* are 27, 43 and 58, respectively. Now we continue discussing the criteria to stop the A*. Fig. 4.11(b) shows that the estimated f value reaches 40 quickly and then increases slowly in the matching between the two dissimilar characters. If we terminate the A* when the current estimated f value is greater than a threshold (say, 40) much search time will be saved. Therefore, we use the following criteria for stopping the A*. 0 2 4 6 8 10 12 14 16 18 expanded node u Figure 4.9: Example 1 for showing the nondecreasing estimated f values, (a) A model character and one of its handwritten characters, (b) Estimated f values. The node numbers denote the sequence of the nodes expanded by the A•’ when the A* searches a tree for the optimal matching between the two characters. The A* terminates after expanding 14 nodes. The matching distance between the two characters is 7. Chapter 4 A State Space Search Method 8 10 12 14 16 18 expanded node u Figure 4.10: Example 2 for showing the nondecreasing estimated f values, (a) A model character and one of its handwritten characters, (b) Estimated f values. The node numbers denote the sequence of the nodes expanded by the A•’ when the A* searches a tree for the optimal matching between the two characters. The A* terminates after expanding 16 nodes. The matching distance between the two characters is 15. 107 Chapter 4 A State Space Search Method i y ft f (a) 10 15 20 25 30 35 40 expanded node u (b) Figure 4.11: Example 3 for showing the nondecreasing estimated f values, (a) A model character and the input character in Fig. 4.10(a). (b) Estimated / values. The node numbers denote the sequence of the nodes expanded by the A*, when the A* searches a tree for the optimal matching between the two characters. The A* terminates after expanding 36 nodes. The matching distance between the two characters is 57. 108 Chapter 4 A State Space Search Method y (a) Chapter 4 A State Space Search Method Criterion 1. The A* algorithm will be terminated if it reaches a best goal node. Criterion 2. The A* algorithm will be terminated if the current estimated / value is greater than a global threshold Tg. Criterion 3. The A* algorithm will be terminated if the current estimated / value is greater than a varied threshold Ty. The global threshold Tg is a constant and is usually set not to be too small so that the model character corresponding to an input character may not be missed. Criterion 3 is used to further speed up the A* algorithm. In Chinese character recognition, even after a preclassification procedure, in general, there are still many model characters, ranging from tens to hundreds, which need to be matched with an input character. Suppose there are m such model characters Gi, G 2 , G m - Let the matching distance between a model Gi and an input G be ^(Gj, G). If a model character G, is more similar to the input than all the remaining characters G^+i Gi+2 …’ (that have not been matched with G yet), then we have ’ G) < G), J• “ 1, i + 2 , m . Now suppose the first i model characters have been matched with the input character. If we let T” = ’ G) = G)}’ (4.22) Chapter 4 A State Space Search Method then we can reduce the search time of the A* in successive matchings when Tu < Tg. For example, the A' is now searching a tree for the optimal matching between and G. If the current estimated / value is greater than Tv{i), then that f is nondecreasing implies +i’G) > 7 = G) and thus no more search for the current matching is needed. It is conventional that after finishing the recognition of an input character, an on-line recognition system gives several model characters that are considered most similar to the input, in order that the real model may not be missed. To reach this goal, we may set T, = T,{i) = e + mmi^Gk, G)}’ (4.23) l 0 is a predefined value. Remark. Criteria 2 and 3 utilize the nondecreasing characteristic of the esti-mated / values of the sequence of expanded nodes to speed up the A*. In a search for the optimal matching between two dissimilar characters, if / increases very slowly, the two criteria may produce little effect. Recall that we use complete relational graphs to represent Chinese characters. The spatial and temporal rela-tions between any two primitives (strokes or segments) give much information for the matching aim. In the matching between two dissimilar characters, with the help of the GPF constraints on input primitives, many relations between model primitives are not compatible with those between input primitives and thus large estimated f values yield quickly. Fig. 4.11 clearly shows such an example, in which if we set Tg = 40 then the A' will stop after expanding 5 nodes instead of 110 Chapter 4 A State Space Search Method expanding 36 nodes. Therefore, the Criteria 2 and 3 are very efficient for speed-ing up the A.. It is also worth noting that in the other two common methods for graph matching, maximal clique-based approaches [5’ 7 10, 12, 13 16] and relaxation labeling approaches [18, 23, 33 41, 51, 54, 99 106], the iteration pro-cedures of the algorithms of these approaches are not directly relative to their corresponding distance measures. Whether or not a matching distance is large can be known only when an algorithm has terminated. This is one of the reasons why these approaches require large computational time when they are applied to on-line Chinese character recognition (13, 16, 18 . 4.5 Experimental Results In this section, we give some recognition results to demonstrate the performance of the graph matching based on-line Chinese character recognition method. We also provide some data to show the search efficiency of the A* with the pruning operation. All algorithms, including the preprocessing algorithms, are imple-mented in C. The computer used is a PC/Pentium at 166MHz. 4.5.1 Stroke-Based Recognition In the stroke-based recognition method, the primitives are strokes and the stroke-based representation of Chinese characters is employed. 300 frequently-used Chinese characters each with stroke number between 9 and 11 are selected for testing the performance of the proposed method. (A Chinese character has an average of 10 strokes [88].) Some of the model character are shown in Fig.4.12. The global threshold Tg and the parameter e (see (4.23)) are set to be 40 and 111 Chapter 4 A State Space Search Method 15 respectively. More than 7000 Chinese characters written by 10 people were used as test data. The subjects were asked not to write the characters in their cursive styles, but there were no stroke order constraints on their writing. A set of handwritten characters having correct stroke numbers are shown in Fig.4-13, in which their corresponding model characters are also given. These characters are all recognized correctly. For such characters with no connected strokes, the recognition rate is about 98.7%. Fig.4.14 shows another set of correctly-classified characters each having one or two connected/split strokes. For such characters, the recognition rate is about 91.2%. If we consider the first 5 model candidates that are regarded as the most similar to an input character, we obtain a recognition rate of 93.6%. Stroke-based recognition methods cannot tolerate too many connected strokes. The reason is that connected strokes change the stroke types, stroke spatial relations and stroke numbers of characters, all of which make the matching distance between an input character having several connected strokes and its model increase greatly. The average time for classifying an input character is about 0.3 second. Such a satisfactory recognition speed is due to the heuristic estimate, the pruning strategy and the criteria for stopping the A*. We have explained how Criteria 2 and 3 can save the computational time of the A* in Section 4.4.3. Now we will present three matching examples each in three cases to demonstrate the usefulness of the pruning operation and the heuristic function h. The pairs of characters in Figs. 4.9-4.11 are employed in the experiments. Three cases are considered: (1) neither heuristic information nor the pruning operation is used {h = 0 GPF = 0); (2) the heuristic estimate defined in (4.10) but not the 112 Chapter 4 A State Space Search Method ^ t r 5 ^fL y c / ^^ I m Figure 4.12: Some model Chinese characters for testing the stroke-based recog-nition method. 113 Chapter 4 A State Space Search Method 1 / f 1 J , r f I Figure 4.13: Some test characters having correct stroke numbers, together with their corresponding models. 114 Chapter 4 A State Space Search Method 1 % / i # 0U I c f e . f Figure 4.14: Some test characters having one or two connected strokes, together with their corresponding models. 115 Chapter 4 A State Space Search Method Table 4.1 Numbers of nodes generated by the A* for three matching examples each, in three cases. -= 0 GPF = 0 780 1202 5540 114 380 2458 27 43 58 pruning operation is used (h • 0, GPF = 0); (3) both the heuristic estimate and the pruning operation are used {h 0, GPF 0). Table 4.1 gives the numbers of nodes generated by the A* algorithm in search. As can be seen, the A* needs to generate a lot of nodes to obtain the optimal matching between two characters in case 1. In case 2 the A* can be speeded up by using the heuristic information, but the results are still not satisfactory, especially when two characters are not similar. In case 3, the search efficiency of the A* is improved significantly by adding the tree pruning operation. 4.5.2 Segment-Based Recognition In the segment-based recognition method, the primitives are segments and the segment-based representation of Chinese characters is employed. 54 Chinese characters (a subset of the models in the stroke-based recognition experiments) each with stroke number between 9 and 11 are used for testing. The values of the global threshold Tg and the parameter e are also chosen to be 40 and 15 respectively. More than 6000 Chinese characters written by 9 people were used as test data. No stroke number and order constraints were imposed on the writing. 116 Chapter 4 A State Space Search Method The recognition rate mainly varies with the numbers of connected strokes ap-pearing in the handwritten characters. For the characters each having less than 3 connected strokes such as those in Figs.4.13 and 4.14, the recognition rate achieves 98.2%. For the characters written each with 4 to 7 strokes, as shown in Fig.4.15 the recognition rate is 94.2%. For the characters written each with only 1 to 3 strokes, as shown in Fig.4.16, the recognition rate is 88.6%. The average recognition rate is about 95%. These results are very promising. Compared with the stroke-based recognition method, the segment-based recognition method can allow more connected strokes in freely-written char-acters. This is because (1) most spatial relations among the segments (not including extra segments in connected strokes) of an input character remain unchanged, and (2) the rules in the segment preprocessing are very useful for breaking connected strokes and deleting some of the extra segments (see Sec-tion 2.4). In the experiments, we found that the assignment of temporal relations be-tween strokes/segments of the model characters almost tolerated all the hand-writing order deviations in the test data, thanks to the "don't care" temporal relations. Of course, incorrect recognition will occur if too many stroke order deviations exist in input characters. For example, Fig. 4.17 shows a Chinese character, the strokes of which are labeled with the numbers indicating their standard order of writing, and one of its handwritten characters having many stroke order deviations.^ In this case, a user may choose the re-classification phase, which do not utilize the stroke/segment order information of Chinese ^One who is not familiar with Chinese characters such as a foreigner may write the character that stroke order. Chapter 4 A State Space Search Method nJ Figure 4.15: Some test characters written each having 4 to 7 strokes, together with their corresponding models. 118 Figure 4.16: Some test characters written each having 1 to 3 strokes, together with their corresponding models 119 Chapter 4 A State Space Search Method A ( A A -\ I I P Chapter 4 A State Space Search Method (a) (b) Figure 4.17: (a) A Chinese character whose strokes are labeled with the numbers that indicate their standard order of writing, (b) A handwritten version of (a) which has many stroke order deviations. characters for recognition,® to obtain a correct recognition result, without the need to write the character again. Therefore, our segment-based recognition method is stroke number and stroke order free. The average time for classifying an input character is about 0.09 second when there are 54 model characters. The stroke-based recognition method re-quires 0.06 second to classify an input if the 54 models are also used. The reason why the segment-based method takes more time is that (1) the number of segments of a character is greater than or equal to the number of strokes of the same character, and allowing more freely written characters often leads to extra segments (the more the node numbers of two graphs under matching, the larger the state space tree for the graph matching) (2) the heuristic function h in the stroke-based method is more precise than that in the segment-based method because the stroke-based method uses more stroke types (15 standard stroke types), which provide more information of Chinese characters than the 5 segment types. Comparing the segment-based and the stroke-based methods, 6The recognition method of the re-classification is the same as the original one except that the weight in (3.11) is set to be 0. 120 5 9 1 \ 6 0 4 6 I \ /y5 \ \ n 0 Chapter 4 A State Space Search Method we prefer the former if the computer running it is not too slow to accept. 4.6 Comparisons of the Segment-Based Recog-nition Method with Several Other Studies In this section, we will make some comparisons between our segment-based recognition method and several other methods published recently in interna-tional journals. Generally speaking, it is difficult, if not impossible, to compare the recognition results of various methods for on-line Chinese character recog-nition. This is because of different subjects in different experiments, different constraints imposed on handwriting, no standard on-line captured Chinese char-acter databases for testing a method, and so on. A Chinese may write the Chi-nese characters that is very difficult to be recognized by others if there are no constraints on his/her writing. For example, our method fails to recognize the characters shown in Fig. 4.18, which are written either too cursively or having great distortions in shape. Thus we cannot say a method is absolutely better than another just according to the recognition rates reported. Besides the recog-nition rate we also have to consider more factors such as recognition time, stroke _ * f f L Figure 4.18: Some handwritten characters that the segment-based method can-not recognize. Their corresponding model characters are also shown. 121 Chapter 4 A State Space Search Method number and stroke order constraints, and degrees of deformation of handwritten characters. Lin et ad. [56] proposed a deviation-expansion model to represent Chinese characters. The dynamic programming is used to perform the character match-ing. Their approach is stroke-based and in essence a string matching one. The approach requires that an input character should not have more than one stroke number variation and more than two connected strokes. There were 5400 mod-els in their experiments. A preclassification step was employed. A recognition rate of 87.4% and an average recognition time of 2.5 seconds per character on a PC/386 at 25MHz were reported. Chou et ai. [21] extended the above model to a segment-based deviation tree. The approach is also a string matching one, so cannot tolerate more than two stroke order deviations. There were 5104 models in the experiments and a preclassification step was employed. They reported a recognition rate of 94.88% for untrained characters and a recognition time of 0.7 second per character on a PC/486 at 25MHz. In [17], Chen et ai. developed a stroke-sequence decision tree and position matching method, which can only recognize the handwritten characters with less than two stroke number variations and is not stroke order free. No recognition rate and recognition time were reported. Tsay and Tsai [92] used attributed string matching by split-and-merge for on-line Chinese character recognition. The proposed method can recognize cur-sive characters but imposes the constraint of correct stroke orders on them. There were 3100 model characters with stroke numbers ranging from 1 to 24. A recognition rate of 96.2% and a recognition time of 2.5 seconds per character on 122 Chapter 4 A State Space Search Method a PC/AT were reported. Chou and Tsai [22] proposed a discrete iteration scheme to solve the problem. Their method is not stroke order free. The provided test characters are in block style and almost have no connected strokes. There were 5401 models in their experiments. A preclassification stage was used. A recognition rate of 91.8% and a recognition time of less than 2 seconds were reported. But the authors did not mention what kind of computer was used. In [40], Hsieh et al. employed a greedy algorithm for bipartite matching to complete the recognition. The method is stroke order free. The provided test characters each with less than 10 strokes are neatly written, some of which have one or two connected strokes. There were 452 models in the experiments. No preclassification stage was used. A recognition rate of 89.7% was reported. If the first three candidates were considered, they obtained a recognition rate of 96.29%. The average recognition time was 39 seconds per character on a Sun workstation. Chen and Lee [16] proposed a fuzzy attribute representation, for Chinese char-acters and used a NP-complete maximum clique finding algorithm to perform the graph matching. There were 650 models each with a stroke number between 1 and 12 in their experiments. A preclassification stage was utilized to reduce the number of models required to be matched with an input. A recognition rate of 95.64% and a recognition time of 2 seconds per character on a Sun SPARC-II workstation were reported. The method is stroke order free, but no test data was provided. In summary, the above methods except the last two are not stroke order free and impose basically correct stroke order constraint on handwriting. The 123 Chapter 4 A State Space Search Method method in [40] is stroke order free but it can only recognize the neatly written characters, some of which have one or two connected strokes. The tolerance of stroke number variations in the method of [16] is difficult to judge since no test data was given. In general, when there are several thousand model characters in a recognition system, a preclassification stage is required to save the overall recognition time. Our segment-based method is stroke order and stroke number free. From the test data provided in our and the other experiments, it is seen that our method can recognize more cursively written characters, and at the same time imposes no stroke order constraint on the handwriting. There is not much difference between the recognition rate obtained in our experiments and the others. As mentioned above, the recognition rate is just one of the factors to judge how good a recognition method is. When there are several thousands of model characters added in our recogni-tion system, a preclassification stage is also necessary. We will discuss the pre-classification problem, in Section 7.2. If 500 models are required to be matched with an input character after preclassification, our segment-based method can complete a recognition within one second on average. Now we compare the recognition time required by our method and those in [40] and [16], all of which are stroke order free. Since the computational power of the PC/Pentium is sim-ilar to that of the Sun workstations used in [40] and [16], our method requires much less recognition time than the method in [40]. It is also faster than that in [16]. The recognition time of 2 seconds per character reported in [16] was obtained under the conditions: there were 650 models each with stroke number 124 Chapter 4 A State Space Search Method from. 1 to 12 and a preclassification stage was employed/ while our method can have a recognition time of about 1 second per character if there are 500 models each with stroke number from 9 to 11 and no further preclassification is per-formed. From the above comparisons, we see that our segment-based recognition method is very promising. 4.7 Summary In this chapter, we have formulated the graph matching as a state space search problem. The optimal matching between two graphs is equivalent to finding the best goal node in a search tree. State space search itself do not change the NP-complete property inherent in the graph matching problem. To obtain good search efficiency, we have used the A* algorithm to perform the heuristic search, and proposed the following schemes to speed up the A*. • A heuristic function h which has been proved to be a lower bound on h* and monotonic, is defined to make the A* expand fewer nodes in a search tree. • A tree pruning strategy, which employs the geometric position features of strokes (or segments) of Chinese characters to prune a search tree, is proposed to let the A* have more or less the function of a bird's eye view, in other words, to let the A* avoid searching the nodes that have very little chance to be located in the optimal path from the initial node to the best goal node in a tree. Obviously, the recognition speed of this method is too slow when there are several thou-sands of models. 125 Chapter 4 A State Space Search Method • Criteria 2 and 3 are presented to stop the A•’ together with Criterion 1 by utilizing the monotone of the estimated / value. The two criteria are based on the fact that in Chinese character recognition, finding the final optimal matching between two dissimilar characters is not necessary if we have known their distance is great enough. The experimental results show that the recognition speeds of our stroke-based and segment-based recognition methods are sufficiently fast for practical applications, even if the frequently-used 5000 or more Chinese characters are added. In common recognition of input Chinese characters (the first phase), the methods can tolerate most of the stroke order variations due to the "don't care" temporal relations between strokes (segments). To deal with a character with great stroke order deviations, the re-classification stage (the second phase) can be effected (without the need to write the character again), which ignores the stroke (segment) order information to perform recognition. Therefore, the methods are stroke order free. The results also show that the segment-based method can recognize the handwritten characters having many connected strokes, so it is stroke number free too. We have made some comparisons between, our segment-based method and several other studies published recently in international journals. Considering their recognition rates, recognition time and tolerances of stroke order and stroke number variations, we see that out method is very promising. Parts of the results presented in this chapter have been published in [58, 59, 60 61, 62 63 . 126 Chapter 5 A Two-Layer Assignment Method 5.1 Introduction The assignment problem is a well known one in operations research and can be solved by the Hungarian method [39, 70, 75]. In this chapter, we propose a two-layer assignment method for the on-line Chinese character recognition problem. The objective of the first layer assignment is to estimate the costs of primitive (stroke or segment) correspondences between two Chinese characters according to their primitive types and spatial-temporal relations, and the objective of the second layer assignment is to find the primitive correspondences between the two characters such that the total correspondence cost is minimized. We also present two schemes to save computational time, which reduce the complexity of the method from 0(n^) to O(n^), where n is the greater number between the two primitive numbers of two characters. 127 Chapter 5 A Two-Layer Assignment Method In Section 5.2, we briefly review the assignment problem and the methods to solve it. The two-layer assignment formulation of on-line Chinese character recognition is given in Section 5.3. The two complexity reduction schemes are discussed in Section 5.4. Some experimental results are presented in Section 5.5. In the last of this chapter is the summary. 5.2 The Assignment Problem The assignment problem is a special type of the linear programming problem. An assignment is useful for modeling a situation, in which there are two distinct sets of objects of equal numbers (say, n), and we need to form them into pairs on a one-to-one base. There is a cost c^j associated with mapping object i to object j, i j = 1, 2 , n . We call [cijJ„xn a cost matrix. Given [c,: assignment problem of order n is to find a permutation matrix^ P = [x to Minimize cost(P) = ^ JZ Gj ij :1 7 = 1 (5.1) Subject to ^ Xij = 1 for i = 1 to n (5.2) ^ = 1 for j = 1 to n i=l € { 0 , 1 } . (5.3) (5.4) ^ A permutation matrix is a square matrix U E i = i ij = 1 and Xij = 0 or 1. whose elements satisfy 128 (a) (b) Figure 5.1: (a) A complete bipartite graph, (b) A complete matching of a bipartite graph corresponding to the assignment in (5.5). connected to each right vertex by an edge. The cost of an edge joining left vertex i with right vertex j is defined as d j . A complete matching^ of the 2In graph theory, a matching in a graph is a set of edges, no two of which share a vertex. When the cardinality of a matching is the largest possible the matching is termed complete. 129 Chapter 5 A Two-Layer Assignment Method We call min{cosf(P)} the minimum total assignment cost. Here is a feasible solution for an assignment of order 4: 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 (5.5) The cost matrix [ l„xn can be represented by a complete bipartite graph, (see Fig. 5.1(a)). The vertexes in the left column denote a set of objects and those in the right column denote the other set of objects. Each left vertex is o • 0 • 0 © & © -0 -& © Chapter 5 A Two-Layer Assignment Method bipartite graph corresponds to an assignment, and vice versa. For example, the matching in Fig. 5.1(b) corresponds to the assignment in (5.5). Thus the assignment problem in (5.1)-(5.4) is equivalent to that of finding a minimum cost complete matching in the bipartite graph of Fig. 5.1(a). That is why it is also known as the weighted bipartite graph minimum cost complete matching problem. We also call [c,j]nxn an edge cost matrix. The Hungajicm method is a popular one with the complexity O(n^) for solv-ing the assignment problem [75]. In addition, there are several other methods for it, such as the cost scaling algorithm [32], the auction algorithm [8j and the auction algorithm incorporating scaling [74 . 5.3 A Two-Layer Assignment Formulation of on-Line Chinese Character Recognition The similarity comparison between two characters can be made by the two steps: (1) find the segment^ correspondences between the two characters; (2) use a measure to calculate their similarity based on the segment correspondences. We will discuss these two steps respectively in the following. 5.3.1 Finding Segment Correspondences between Two Characters Fig.5.2 shows a model character and its handwritten character after the prepro-cessing. We want to obtain such segment correspondences: 3 In the rest of this chapter, the segments of characters are used as primitives. Chapter 5 A Two-Layer Assignment Method (a) (b) Figure 5.2: A model character (a) and its handwritten character (b). 1 — 1’ 2 — 2’ 3 — 3 4 — 4’ 5 — 5,6 — 6 7 — 8’ 8 4 9,9 4 10 A — 7’ 5.6) where A — 7 denotes a correspondence between a dummy segment in character (a) and the extra segment in character (b). Now we represent various segment correspondences using a bipartite graph, as shown in Fig.5.3. Segments 1-9 in Figure 5.3: A bipartite graph formulation of segment correspondences between the two characters in Fig. 5.2. character (a) are denoted by vertexes 1-9 in the left column, respectively, and segments 1-10 in character (b) are denoted by vertexes 1-10 in the right column, respectively. Vertex A (a dummy segment) is added in the left column to make the segment numbers of the two characters equal. 131 Chapter 5 A Two-Layer Assignment Method To sum up, for obtaining the segment correspondences between character 1 with segment number ni and character 2 with segment number n2 (assume ni < 712 without loss of generality), a complete bipartite graph is constructed by two vertex columns each with n vertexes. There is an edge joining a vertex in the left column with a vertex in the right column, rii vertexes in the left column denote ni real segments of character 1 and the other 712 — Ui vertexes denote 712 — n! dummy segments. In the right column, 712 vertexes denote 722 real segments of character 2, Now a critical issue is how to derive the edge costs of a bipartite graph such that after solving the corresponding assignment problem, we can have the desirable segment correspondences between two characters. The idea in graph matching discussed in the last two chapters may be borrowed, the goal of which, roughly speaking, is to find an optimal matching between two characters such that the sum of all the costs of segment type correspondences and the costs of relation correspondences between segments is minimized, in other words, to make the segment type correspondences and the relation correspondences be-tween two characters as compatible as possible. Keeping this in mind, we will propose an approach for deriving the costs of edges of a bipartite graph, i.e., the cost matrix [ctj)„xni where Cij is the cost of edge joining vertex i in the left vertex column with vertex j in the right column in the bipartite graph. First, we define = d r if i denotes a dummy segment and j a real segment, or vice versa, where dr is a positive value and is determined by experiment. Let Bn be the bipartite graph created with character 1 and character 2, and its edge cost matrix be [ctjjnxn- Let i and j be two real segments in characters 1 and 2 respectively. The information used to derive [Cijj„xn is the segment types and 132 Chapter 5 A Two-Layer Assignment Method the spatial-temporal relation matrixes of the two characters. We define Cij = l(i j ) + Pij, (5.7) where 7(2' —> j) is the correspondence cost between the type of segment i and the type of segment j as defined in Table 3.4 4 and is the minimum cost of a complete matching in a new bipartite graph B (or the minimum total assignment cost of the new assignment problem corresponding to The left column of consists of all the vertexes except vertex i in the left column of Bjij and the right column of consists of all the vertexes except vertex j in the right column of The edge cost matrix [c^„_i)x(rt_i} of B^J i^ is directly derived by 4 = (5.8) where k and I are two real segments in characters 1 and 2 respectively, 7(fc is the segment type correspondence cost, (z, k) and (j, /) are the spatial-temporal relation from i to k and from j to I respectively, and 7((i fc) —> (j, i)), called relation correspondence cost, is defined the same as the arc correspondence cost in (3.11). If there is one dummy segment between segment k and segment I, we define cj^ / = . Now we explain the meaning of the definition of Cij. Cij is equal to a sum of two terms. The first is a segment type correspondence cost. When the type of segment i is the same as that of segment j, this term costs the least (7(2 —> j) = 0). To better understand the second term ptj, more description is needed. In the following, by using an example, we will state that if the two •*Note that in this chapter, Cij is called a segment correspondence cost, and 7(1 — j) is called a segment type correspondence cost. 133 Chapter 5 A Two-Layer Assignment Method a Figure 5.4: A Chinese character (a) and its very similar handwritten style (b). characters under comparison is very similar to each other and if segment i is just the segment that should correspond to segment j in character 2 then Pij = 0. Consider two very similar characters, character 1 and character 2 as shown in Figs.5.4(a) and (b). The segment correspondences: 1 — 1,2 2’ 3 — 3 4 4 5 — 5’ 6 4 6’ 7 — 7’ 8 — 8’ 9 — 9 (5.9) are what we want to obtain. We say they are very similar in the sense that (1) the costs of segment type correspondences in (5.9) are all equal to 0; (2) the relation correspondence cost 7((2i f2) —> (ji, J2)) = 0, where z'l, j i , 22,72 satisfy the condition that ii —> j i and 22 —>• j i are any two different segment correspon-dences in (5.9). Now suppose i = 9 and j = 9. Then C99 = 7(9 -)> 9) + pgg. P99 corresponds to a new assignment problem of order 8. Let [c x8 be the cost matrix of the assignment problem. Without loss of generality, suppose the subscripts k and I on cj® denote just segment k and segment I in character 1 and character 2 respectively. By (5.8), we then have cZn = l{m — rn) + 7({9, m) — (9, m)) = 0 ’ m = 1,2, ... 8. (5.10) Thus all diagonal elements in [cjjjgxs are 0. A permutation matrix P ' = [arjt/Jsxs with Xmm = 1, m = 1, 2, ... 8 and Xki = 0’ fc is a solution to the assignment 134 Chapter 5 A Two-Layer Assignment Method problem and the minimum total assignment cost pgg = 0. Obviously, if the two characters are not so similar, the costs of the segment type correspondences or segment relation correspondences discussed above will not be 0 and we have pgg 0. Besides, if i = 9 and j = 1 by (5.8) we can see that all the elements of the cost matrix [cjfjgxs (of another assignment problem) will be greater than 0 because of the incompatible segment type correspondences and relation correspondences. This results in a minimum total assignment cost P9l > 0. Table 5.1 gives the whole matrix [Ajl9x9 by solving the 81 assignment prob-lems of order 8 and Table 5.2 shows the corresponding cost matrix [Cjj]9x9’ for finding the segment correspondences between the two characters in Fig. 5.4. By applying the Hungarian method to the assignment problem with this cost ma-trix [cjjJgxQ, we can obtain the segment correspondences in (5.9). Therefore Cij defined in (5.7) better reflects the degree of incompatibility between segment i and segment j. Fig.5.5 shows the structure for obtaining segment correspondences between two characters. There are n x n assignment problems of order n — 1 in layer 1 and there is one assignment problem of order n in layer 2. Obtained by solving the assignment problem with the cost matrix [c!j{](n-i)x(n-i) in layer 1, pij is used together with the segment type correspondence cost 7(2 —> j) to estimate the cost Cij in the assignment problem in layer 2. The cost cjj is calculated by utilizing the information of segment types and relations of the two characters. Note that if one of the segments i and j is a dummy segment, c,j is simply set to be dr. We do not specify these cases in the figure for simplicity. By Fig. 5.5, we call the proposed method a two-layer assignment method. 135 Chapter 5 A Two-Layer Assignment Method Table 5.1: Matrix [/9ij]9x9 for estimating the cost matrix [cij 1 2 3 4 5 6 7 8 9 1 0 12 30 39 48 58 72 85 96 2 12 0 12 21 30 40 56 67 78 3 24 12 0 15 18 28 44 55 72 4 21 15 15 0 37 37 59 50 91 5 54 36 42 57 0 26 30 67 48 6 64 52 34 55 20 0 16 35 50 7 72 60 48 63 36 16 0 7 30 8 79 67 55 56 41 23 7 0 29 9 96 84 78 103 42 38 24 35 0 Table 5.2: Cost matrix [QjJgxg for finding the segment correspondences between two characters in Fig. 5.4. 2 3 5 6 7 8 9 136 Chapter 5 A Two-Layer Assignment Method segment types of character 1 and character 2 relation matrixes of character I and character 2 )/ layer layer 2 Figure 5.5: The structure for obtaining segment correspondences between char-acter 1 and character 2. 137 Chapter 5 A Two-Layer Assignment Method It is not difficult to estimate the computational complexity of the two-layer assignment method. Let the numbers of segments of two characters under com-parison be rii and 712 respectively, and n = max{7ii, 722}. Then the n x n assignment problems of order n — 1 in layer 1 can be solved by applying the Hungarian method n x n times, each requiring 0( (n - 1)3) time. The effort to create the cost matrix of each assignment problem in layer 1 is 0((n — 1)-). Thus the total effort made in layer 1 is O(n^). There is only one assignment problem of order n in layer 2, which can be solved in O(n^) time. Therefore, the complexity of the entire two-layer assignment method is O(n^). 5.3.2 Calculating the Similarity of Two Characters After solving an assignment problem in layer 2 we obtain a permutation matrix P* that denotes a set of segment correspondences between two characters. With the P* = [x'jjnxn and the cost matrix [cijjnxn of the assignment problem, we have the minimum total assignment cost {MTAC): M T A C i P n ^ E Z ^ i j ^ h - (5.11) i=lj=l It can be used as a distance to compare two characters. However, in our exper-iments we found that it is not good enough. Suppose A and B are two model characters and C is an input handwritten character belonging to the class of A. Let and P*BC be the permuta-tion matrixes for segment correspondences between. A and C and for segment correspondences between B and C respectively. In general, MTACi^*ac) < MTACi^*Bc)i but sometimes, when C is not very similar to A, we may have MTAC(V*AC) > MTACCP^BC)- However, if the matching cost 0 defined in 138 Chapter 5 A Two-Layer Assignment Method the following (5.12) is used, we will still have 0AC < where (3AC and 0BC are the matching cost between A and C and the matching cost between B and C respectively. By Definitions 3.5 and 3.6 it is not difficult to see that P* corresponds to a node mapping /jy (and an arc mapping / led by /y ) . Therefore, we use the following matching cost, which is similar to (4.1) and is a simplified version of (3.2) in Definition 3.7, to calculate the distance between two characters: ^U^^.Ta) = E — E A) + Y. 7(A ^ i ) i^j^Qx i^ AeQa + E — ( j ’ 0 ) (5-12) where Qi is the set of correspondences between real segments; Q2 and Q3 are the sets of correspondences between real segments and dummy segments; Q4 is the set of relation, correspondences from (i, k) to (J l) i + \ i \ k A, I ^ A. Compared with MTAC, (3 reflects more directly and fully the relation compatibility of segment correspondences between two characters. 5.4 Two Complexity Reduction Schemes Although the two-layer assignment method can be implemented in polynomial time 0(n®), we find that an algorithm with such running time is not suitable for on-line recognition of Chinese characters. Hence, we propose two complexity-reduction schemes for the recognition problem. Chapter 5 A Two-Layer Assignment Method 5.4.1 A Lower Bound Estimate In the two-layer assignment method, the main computational effort is to solve the n X n assignment problems of order n — 1 with the Hungarian method. This results in an 0(n®) algorithm in layer 1 the aim of which is to obtain pij (f, j = 1, 2 …’ n) and then derive the cost matrix [ci_,]„xn- The following theorem is useful for obtaining an estimate of p”’ Theorem 5.1 Given an assignment problem: n n Minimize cos (Q) = ^ ^ dijXij, 1=1 j=i where Q = [:Eijjnxn aTid [dijjnxn a permutation matrix and a cost matrix respectively, a lower hound estimate e of the minimum value of cost{Q), i. e., imn{co5i(Q)} > e (5.13) can be found by e = ^ min-Ti + ^ min.Cj, (5 .14) 1=1 j=i where minjTi = mm{rfii, d i 2 , i s the smallest element in row i in the cost matrix [rf,j]nxn» an^f min.Cj = d'2j ..” d'^^} is the smallest element in col-umn j in the cost reduction matrix [dy„xn, where d\j dij—min-ri, for all z, j = X12 ••” Th* Proof. Investigating the Hungarian method in [48], after steps (a) and (b) of the method, we have a reduction of costs that is exactly equal to e. Let the total reduction of costs in the steps following step (b) be e'. Then we have inin{co5^{Q)} = e -h e'. (5.15) 140 Chapter 5 A Two-Layer Assignment Method Since e' > 0, (5.13) holds. • Theorem 5.1 provides us an approach to approximately obtain pij (and thus Cij), by using (5.14) instead of the Hungarian method. Calculating e from an n X n matrix needs O(n^) time. With this estimate, the effort made in layer 1 now becomes which is also the complexity of the two-layer assignment method. 5.4.2 Geometric Position Constraints The derivation of the cost matrix [Cjjjnxn consumes the most computational time in the two-layer assignment method. Cij is the correspondence cost between segment i of a character and segment j of another character. In fact, with the help of the geometric position features (GPF) of a model character, which are defined in Definition 4.5,5 we can reduce much computational time spent in layer 1, Look at Fig.5.4, the correspondences between segment 1 in character (a) and one of the segments 5-9 in character (b) is obviously unreasonable. Now we can use the GPF of a model character to decide whether a cost Cij in the cost matrix in layer 2 needs to be estimated in layer 1. Let i be a segment of a model character and j be a segment of an input character. Let the zth element of the GPF of the model be If Xi < od(Dd,(j)) < pi, i.e., the geometric position of segment i and that of segment j are compatible, then c" will be estimated in layer 1; otherwise, just set Cij a sufficiently large positive value. With the geometric position constraints, the computational complexity of the two-layer assignment method can be reduced further. In general, an integer ®In Definition 4.5, the primitives are strokes of Chinese characters. When it is applied to the segment-based recognition method, the primitives should be segments. 141 Chapter 5 A Two-Layer Assignment Method interval [xi, y,] (see Definition 4.5) satisfies y, — Xi < K (say, 5). In this case, at most Kn elements in a cost matrix [c,j]„xn need further computation. Therefore, by using the above two complexity reduction schemes, the time required by the two-layer assignment method is O(Kn^) — O(n^). 5.5 Experimental Results The two-layer assignment method has been implemented in C on a PC/Pentium at 166MHz. The primitives of the method are segments. Before an input char-acter is recognized, its segment types and the relations between its segments are extracted first in the preprocessing procedure. 54 Chinese characters, which have been used in the experiments for testing the segment-based state space search for graph matching method (see Section 4.5.2), are also used here. The parameter dr is chosen to be 5 which is equivalent to the cost of a segment deletion in graph matching. The test data consists of more than 3000 Chinese characters written by 6 peo-ple. No stroke number and stroke order constraints were imposed on their writ-ing. Fig. 5.6 shows a set of the test characters that are all recognized correctly. The recognition rate varies with the numbers of connected strokes appearing in the handwritten characters. The stroke numbers of the model characters are between 9 and 11. For the characters each having less than 3 connected strokes, the recognition rate is 96.3%. For the characters written each having 4 to 7 strokes, the recognition rate is 92.0%. The average recognition rate is 93.8%. The average time for recognizing an input character is 0.085 second. If the two complexity reduction schemes are not used, the recognition time is about 142 Chapter 5 A Two-Layer Assignment Method . V Z 1 / A lA t 4 L U x < Figure 5.6: Some test data in the experiments. 143 Chapter 5 A Two-Layer Assignment Method Table 5.3: Performance comparison of three methods method average recognition rate average recognition time stroke-based state space search 90.9% 0.06 second segment-based state space search 95.6% 0.09 second segment-based two-layer assignment 93.8% 0.085 second 1 second, but the recognition rate almost remains the same. This is because (1) even if the lower bound estimate is not precise in some cases, the segment correspondences obtained by solving an assignment problem with an estimated cost matrix in layer 2 are still correct, and (2) the geometric position constraints of model characters are loose enough to tolerate most of the segment position variations in handwriting. The set of test data was also used to test the stroke-based and segment-based state space search methods. Table 5.3 shows the results. Comparing these three methods, we can see that the stroke-based state space search method runs fastest but the recognition rate is lowest, and the segment-based state space search method runs slightly slower than the segment-based two-layer assignment method but has the highest recognition rate. 144 Chapter 5 A Two-Layer Assignment Method 5.6 Summary In this chapter, we have proposed a two-layer assignment method for on-line Chinese character recognition. Finding segment correspondences between two characters is formulated as a weighted bipartite graph minimum cost complete matching problem, which corresponds to an assignment problem of order n and can be solved by the Hungarian method in O(n^) time, where n is the greater number between the two segment numbers of the two characters. In order to derive the cost matrix of the assignment problem in layer 2, n x n assignment problems of order n — 1 are created in layer 1. The costs of segment type corre-spondences and relation correspondences between the two characters are used to generate the cost matrixes in layer 1. To save the computational time, a lower bound estimate is employed to obtain the approximate value of the minimum total assignment cost of each assignment problem in layer 1. In addition, the geometric position features of model characters are used to avoid wasting com-putation on unreasonable segment correspondence costs. These two schemes reduce the complexity of the method from 0(n®) to O(n^). The experimental results are satisfactory. Compared with the segment-based state space search for graph matching method, the method in this chapter runs slightly faster and has a little lower recognition rate when they were used to recognize the characters with an approximate degree of deformation. Chapter 6 A Fast String Matching Method 6.1 Introduction We have proposed two methods for on-line Chinese character recognition in the last two chapters. They all use both the types of primitives (strokes or segments) of Chinese characters and relations between primitives to carry out recognition. The experimental results have shown that they have the ability to tolerate wide stroke order and stroke number deviations in handwriting. How-ever, they require relatively large amounts of computation and are suitable to be implemented on relatively high-end CPUs such as a PC/486 or above. In various practical applications, many products such as portable electronic di-aries, electronic Chinese-English dictionaries, multi-functional telephones and simple Chinese typewriters, may need fast small-memory-requirement recogni-tion methods, due to their low-end CPUs and limited memory space equipped. Each Chinese character has a standard stroke writing order and Chinese peo-ple write a Chinese character basically according to its standard stroke order. 146 Chapter 6 A Fast String Matching Method On-line devices can capture the temporal information of the writing, including the order, number and direction changes of strokes. If model characters and input handwritten characters are all represented by their corresponding primi-tive (stroke or segment) strings, then the twodimensionct! character recognition problem can be transformed to a relatively simple one-dimensional string match-ing problem. This fact makes some researchers study string matching based recognition methods [21 22 55 56, 57, 92] ‘ In this chapter, based on Wagner and Fischer's string matching (WFSM) algorithm [96], we propose a recognition method that incorporates the geometric position constraints of primitives into the WFSM algorithm. The method is very fast. Its running time is 0(mn) for matching two characters with primitive numbers m and n. In Section 6.2, we briefly review the WFSM algorithm. Its application, to on-line Chinese character recognition is presented in Section 6.3. Some experimental results are given in Section 6.4. In Section 6.5, we propose an extension of the string matching method when there may be several primitive strings to represent a model character. The summary in Section 6.6 ends this chapter. 6.2 The WFSM Algorithm In [96], Wagner and Fischer discussed the string-to-string correction problem and suggested the application of the WFSM algorithm to spelling correction. Let S = SiS2--.Sjn be a finite string of m symbols. A null symbol is denoted by A. An edit operation is a pair (a, b) ^ (A, A) and is written as a — 6, where a and b are two strings of length 0 or 1. Three edit operations on strings are as 147 Chapter 6 A Fast String Matching Method follows: • code insertion: A -> a • code substitution: a —> 6 • code deletion: a —> A The application of an edit operation a — 6 to string S results in string R’ which is written ds S R via a — b. Let be a sequence ei,e2,Cp of p edit operations. An edit transformation of string S to string is a sequence of strings Sq, Si, ...’ Sp such that S = Sq, R = Sp and 5,-_i 5, via e, for 1 < z < p. In order to measure the similarity (or distance) between two strings. Costs associated with the edit operations are necessary. Let 7 be a cost function that assigns to each edit operation a —> 6 a nonnegative real number 7(a —> 6). 7 can also be extended to a sequence of edit operations E = Ci, 6 2 , C p by-setting 7( = 7 ( e , ) . If p = 0 i.e., no edit operation is applied, we define 7(£") = 0. The edit distance (or distance for short) between strings S and R is defined as 5(5, R) = min{7(£')|£^ is a sequence of edit operations that transforms 5 to i?}. (6.1) To simplify the calculation of the edit distance ^2) between two strings 5i = SiS2-- Sjn and S2 = Wagner and Fischer defined a structure called a trace as follows. A trace from Si to S2 is a triple ( T , ^ 2 ) (or simply T when the strings 5i and S2 are understood), where T is any set of ordered pairs of integers (2, j) satisfying: (a) 1 < t < m and I < j A) then i := i - 1; else if D[i,j] = D[iJ - 1] 4- 7(A — s;) then j j - 1; else begin print((2,j)); i i — 1; j := j — 1; end end 151 Chapter 6 A Fast String Matching Method 6.3 Application of the WFSM Algorithm to on-Line Chinese Character Recognition Wagner and Fischer suggested that the WFSM algorithm can be applied to spelling correction. In fact, it may also be applied to some pattern recognition problems such as the chain code string matching given in Section 2.3.2 and the on-line Chinese character recognition presented in this section. For better understanding traces, the WFSM algorithm, and the extension of the WFSM algorithm to be presented in Section 6.5 we construct a network as shown in Fig. 6.2 in which each path from the source node (0,0) to the target node (m, n) corresponds to a trace from Si siSa-.-STn to S2 = s'ls^—s'^. For example, the bold path P from node (0,0) to node (7,5) @ Fig. 6.3 corresponds Figure 6.2: A network for calculation of the distance between a string of length m and a string of length n. to the trace T in Fig. 6.1. We may assign a cost 7(5, — A) to the arc from node (i — 1, j ) to node (i, j), a cost 7(A — s'j) to the arc from node (i, j — 1) to node (z, j ) , and a cost 7(5^ — Sj) to the arc from node ( f - 1 , J - 1 ) to node (z,j), 152 Chapter 6 A Fast String Matching Method q Y O ^ ^ @ yis^^x) @ ~ ‘ @ @ —— —— @ ~ ^ ( 5 ) ~ p={(0,0),( 1 1 ) ( 1 ’ 2)’(2’3) (3 3) (4 3)’(5 4)’(6 4)’(7 5)} Figure 6.3: A network for calculation of the distance between two strings in Fig. 6.1. The bold path P corresponds to the trace T = {(1,1), (2, 3), (5,4), (7, 5)}. 153 Chapter 6 A Fast String Matching Method for all i = 1,2 ..., m and j = 1 , 2 ’ n , as shown in Fig. 6.3. If the cost cost(P) of a path from the source node to the target node is defined as the sum of the costs of all the arcs in the path, it is easy to see that cost{P) = cost{T), (6.4) where P corresponds to T. Therefore, the calculation of the distance between two strings is equivalent to finding the least cost of a path from the source node to the target node in the corresponding network. We call such a path the shortest path. Considering Fig. 6.3, we see that the problem of finding the shortest path in the network can be divided into 8 stages in the vertical direction (or 6 stages in the horizontal direction). Such an optimization problem can be solved by a dynamic-programming algorithm with the following recursive relationship be-tween two successive stages: D[i, j] = mm{D[i - 1, j - 1] + y(si — s'j),D[i - 1 j l -f- -yisi A), + (6.5) where D[i,j], D[i - 1, j — 1], D[i - 1, j j and D[i, j — 1] are the least costs of the paths from node (0,0) to node (t, j ) , to node (i— — 1), to node (z - l . j ) , and to node — 1), respectively. Comparing (6.5) with the algorithmic equations in the two-layer for-loops in the WFSM algorithm, we will find that the WFSM is actually a dynamic-programming algorithm. The application of the WFSM algorithm to the on-line Chinese character recognition problem is direct. We represent a model character with a string S — 5i52...5Tn> where 5i, 52, Sm are the primitive (stroke or segment) types 154 Chapter 6 A Fast String Matching Method of the m primitives of the character, arranged in its standard order of writing. We also represent an input character having n primitives (after preprocessing) with a string R = rir2..,r„, where ri, r 2 , r „ are the primitive types of the n primitives of the character, arranged in its input order of writing. Then the comparison of similarity between S and R can be formulated as the problem of calculating the least cost of a path from the source node to the target node in the network formed with S and R. The WFSM algorithm can be used to carry out the calculation. If the primitives are strokes (segments, respectively), then the stroke (seg-ment, respectively) type correspondence costs in Table 3.2 (Table 3.4 respec-tively) may be used as the stroke (segment, respectively) substitution cost 7(Si —> Sj), and the stroke (segment, respectively) insertion cost and the stroke (segment, respectively) deletion cost are defined as 7(5,- ^ A) = 7(A —5^) = idl {id2, respectively). However, using only the information of primitive (type) strings of Chinese characters is not sufficient to distinguish a character from the others when there are stroke type variations and connected strokes in handwriting, as mentioned in Section 3.3.1. We have found that it is true after we implemented the WFSM al-gorithm. Now we propose a scheme in the following to make the string matching method have better ability to do the recognition work. Recall that we have defined the geometric position features (GPF) of a model character in Definition 4.5. In the string matching method, the GPF of a model character can help to assign a cost to the primitive substitution between a model primitive and an input primitive. Let i denote the fth primitive of a model character and j a primitive of an Chapter 6 A Fast String Matching Method input character. Let the zth element of the GPF of the model be ( 1, 4 -h- 2; character (e) 7 8, 8 ^ 7; Chapter 6 A Fast String Matching Method / ZI a . Figure 6.4: Some test characters each having less than three connected strokes. 159 Figure 6.5: A model character (a) and a set of its input handwritten characters (b)-(j) having different stroke orders. character (f) 8 9, 9 8; character (g) 2 3, 3 ^ 2, 7 8, 8 7; character (h) 5 -e)- 6, 6 5, character (i) 2 -h- 3, 3 ^ 2, 8 9, 9 8; character (j) 1 3, 2 4, 3 ^ 1, 4 ^ 2, 7 ^ 8, 8 7; Let (model p, input q) be the distance between model character p and input character q. Then we have J(model a, input k) = min{(J(model 1, input 5(model 2, input k), (5(model 300, input fc)} where model a is the model character in Fig. 6.5(a), and input k, k E {6, c, d, e, / ’ g, h} is one of the input characters in Figs. 6.5(b)-(h). This means that these characters are recognized correctly. However, — © (b) Chapter 6 A Fast String Matching Method _ _ _ ~ ~ ~ @ ~ ~ ~ idi @ ~ ^ ® ~ ~ ~ idx idx M X I X X © • © ~ ^ ^ - Q ~ - O idx idx \ l \ l \ © ~ ~ ~ ~ idx ~ ® - © ~ - © ~ - © - © @ ^ ^ ~ ~ ® ~ © ~ ~ ~ © ~ ~ ~ (C) Figure 6.9 (a) A network for calculating <5(5; R). (b) A network for calculating 5(52, (c) A network obtained by combining (a) and (b). Chapter 6 A Fast String Matching Method © - 1 r - © — © (a) (b) Figure 6.10: (a) Combining strings Si, So, S3 and S4 together, (b) An input string R. cost{Pi) = cost(P[), where cost(P[) is the sum of all the costs of the arcs in P(. Let P' be any path from node (0,0) to node E in network (c). Then the problem of finding min{S{S[, R), 6(82, R)} respectively in networks (a) and (b) is now transformed to the problem of finding min{co5i(P')} in network (c). Network (c) is also a multi-stage one. Obviously, this optimization problem can be solved by a dynamic-programming algorithm. As the arc number in network (c) is 84 (< 112), the new problem requires less time to be solved. Let us consider a more complicated example. Suppose a model character is represented by the following four strings: Si = S1S2S3S4SS, S2 — 51S2535455, S3 = S4 = S'iS'2S3S:S'5’ Combining 5i_4 as shown in Fig. 6.10(a). Now we want to find mm{5iSuR). 6(82, H), R), 5{S4, /?)} where R = 7*1 2 3 4 is an input string (Fig. 6.10(b)). The WFSM algorithm can be used to calculate 5(Si, R), <5(52, R)J{S3, R) and 5(54, H), respectively. The corresponding four networks (a)-(d) are shown in Fig. 6.11. There are 276 arcs in the four networks. Combining these networks together, we obtain the 167 I \ I \ \ o — o — o — o — c t N . \ l \ l \ l \ r^ ^ idi idi idi idx V W V dx \ Q ~ - O — O ~ - Q ~ < idi r^ ‘r\ - ^ -w w w w — w w - O ~ O - O (C) Figure 6.11: Four networks (a)-(d) for calculating 6{Si,R), 5{S2,R), 5{S3,R), and (5(54, -R). respectively. ~ - O - o ~ - o ~ (a) - r\ “r\ \ 1 \ I \ K - O ~ - O - o (b) © Chapter 6 A Fast String Matching Method idx. r^ ‘r\ - ^ - r^ ‘r~\ \ )—— \ idi KKi KKi KKi J • ‘ ) ‘ Chapter 6 A Fast String Matching Method ® i - o ~ - o - o O — 0 - — o — o — o a — id, 0 — 0 — 0 — 0 — 0 YuX^ \ \ - 0 ~ - 0 0 - Q ~ ~ • " ~ 0 — Figure 6.12: A network obtained by combining networks (a)-(d) in Fig. 6.11. 169 Chapter 6 A Fast String Matching Method network in Fig. 6.12 in which there are 132 arcs. Comparing Fig. 6,11 with Fig. 6.12 it is not difficult to see that for a path P2 from node Bi to node Ei in network (a), from node Bo to node E2 in net-work (b), from node B^ to node E3 in network (c), or from node S4 to node E4 in network (d) there exists a path P, from node B to node E in the net-work in Fig. 6.12 such that cos (P2) = cost{p2). Therefore, using a dynamic-programming algorithm and the network, we can also obtain the solution to the problem of finding niin{5(5i, R), 6(82, R), SiS^, R), 5(84, R)}. 6.6 Summary In this chapter, we have proposed a fast string matching method for on-line Chi-nese character recognition, which incorporates the geometric position constraints of primitives of Chinese characters into Wagner and Fischer's string matching algorithm. The experiments show that when input characters are written not having great stroke order deviations, the method can obtain good recognition results. Moreover, the segment-based method may tolerate more cursive hand-writing than the stroke-based one. To allow more stroke order deviations for some characters, using two or more strings to represent one of these model char-acters is necessary. In this case, we suggest a scheme to save computational time, which combines two or more separate networks into one and employs a dynamic-programming procedure to solve the shortest path problem. The string matching method is very fast and requires small memory space. When the primitives used are segments, it runs 25 times as fast as the segment-based graph matching method does. Therefore, this method can be implemented 170 Chapter 6 A Fast String Matching Method low-end CPUs. 171 Chapter Conclusions and Suggestions 7.1 Contributions of this Thesis The aim of this thesis is to derive efficient methods for on-line Chinese character recognition. We have addressed the three aspects: preprocessing of input hand-writing, representations of Chinese characters, and recognition methods. Now we summarize our major contributions and results as follows. 1. Preprocessing of input handwriting • In order to facilitate the recognition of the types of strokes and segments, an input stroke is represented with a polyline by using the efficient poly-line fitting algorithm and the line merging algorithm. This approach can handle some handwriting noise (such as wild points and hooks) well. • A method for recognizing the types of strokes with more than two segments is proposed, which consists of three procedures: normalization of strokes, extraction of stroke chain code strings, and matching between input code 172 Chapter 7 Conclusions and Suggestions strings and model code strings. The experimental results show that the method works well. It can be used not only in stroke-based but also in segment-based on-line recognition of Chinese characters. • Some rules are presented to detect most of frequently-occurred connected strokes and then delete the extra segments in such strokes. These rules make our recognition methods have the ability to recognize more freely-written Chinese characters 2. Representations of Chinese characters we have formally defined the complete relational graphs and the distances for measuring the similarity between two graphs. With such graphs, we have pro-posed the stroke-based and segment-based spatially-temporally relational rep-resentations for on-line inputted Chinese characters. We have also dealt with assigning costs to node and arc correspondences for calculating the graph match-ing distances. The stroke-based representations may be used to recognize relatively neat Chinese character handwriting while the segment-based representations will ease the recognition of more freely written characters but make a recognition method need more computational time and memory space. These representations have the following advantages: • The representations incorporate the human knowledge of Chinese charac-ters and can reflect their features well (except some very similar character pairs). The novel "don't care” "should" and "must" relational features allow us to represent unstable, stable and very stable primitive relations 173 Chapter 7 Conclusions and Suggestions conveniently. Relations between any two primitives give much information and are very beneficial to the matching procedures. • The proposed complete graph representations are directly based on strokes or segments. To obtain the representations, examining whether a stroke or segment belongs to some component is not required. However, the graph representations in [18 68] need to correctly extract components of Chinese characters first. The recognition method based on the graph representation in [13] also needs to find components before performing recognition of a character. In fact, wide stroke type variations and connected strokes make it very difficult to extract components of Chinese characters at a high success rate. In [16], the authors adopted only the relations between segments within the same components in their graph representation. This results in two shortcomings: (1) some relations represented in an input graph may not appear in its corresponding model graph, and vice versa; (2) most of the relation information between segments are not utilized. • The spatial and temporal relations between primitives axe, at the first time, unified into the graph representations, which fully captures the on-line information of handwriting. The use of the primitive order relations enhances the discrimination ability of the representations and helps to speed up the graph matching. Because of the "don't care" feature, the representations can tolerate common stroke order deviations. • If the weight w^ in (3.11) is set to 0 in graph match, then the stroke order relations will be ignored and our recognition methods will allow writing a Chinese character in any stroke orders. 174 Chapter 7 Conclusions and Suggestions The disadvantage of our representations is that the creation of a model char-acter base is a relatively heavy task. It can be eased by constructing the graphs of components of Chinese characters first and then combining several component graphs to form the whole graph of a character. We will discuss this problem in the next section. 3. A state space search method We have formulated the graph matching as a state space search problem. The optimal matching between two graphs is then equivalent to finding the best goal node in a search tree. To obtain good search efficiency we have used the A* algorithm to perform heuristic search, and proposed the following schemes to speed up the A*. • A heuristic function h which has been proved to be a lower bound on h* and monotonic, is defined to make the A' expand fewer nodes in a search tree. • A tree pruning strategy, which employs the geometric position features of strokes (or segments) of Chinese characters to prune a search tree, is proposed to let the A* have more or less the function of a bird's eye view, in other words, to let the A* avoid searching the nodes that have very little chance to be located in the optimal path from the initial node to the best goal node in a tree. • Two new criteria, together with the original one, are presented to stop the A* by utilizing the monotone of the evaluation function of the A*. They are based on the fact that in Chinese character recognition, finding the 175 Chapter 7 Conclusions and Suggestions final optimal matching between two dissimilar characters is not necessary if we have known their distance is great enough. The experimental results show that the recognition speeds of our stroke-based and segment-based recognition methods are sufficiently fast in practical applications, even if the frequently-used 5000 or more Chinese characters are added. In common recognition of input Chinese characters (the first phase), the methods can tolerate most of the stroke order variations due to the "don't care" temporal relations between strokes (segments). To deal with a character with great stroke order deviations, the re-classification stage (the second phase) can be effected (without the need to write the character again), which ignores the stroke (segment) order information and takes a little more time to perform a recognition. Therefore, the methods are stroke order free. The results also show that the segment-based method can recognize the handwritten characters having many connected strokes, so it is stroke number free too. We have made some comparisons between our segment-based method and several other studies published recently in international journals. Considering their recognition rates, recognition time and tolerances of stroke order and stroke number variations, we see that out method is very promising. 4. A two-layer assignment method Finding segment correspondences between two characters is formulated as a weighted bipartite graph minimum cost complete matching problem, which cor-responds to an assignment problem (in layer 2) and can be solved by the Hun-garian method. The cost matrix of this assignment problem is derived by the assignment problems in layer 1. The costs of segment type correspondences and 176 Chapter 7 Conclusions and Suggestions relation correspondences between two characters are used to generate the cost matrixes in layer 1. To save the computational time, a lower bound estimate is proposed to approximately solve each assignment problem in layer 1. In ad-dition, the geometric position features of model characters are used to avoid wasting computation on unreasonable segment correspondence costs. These two schemes reduce the complexity of the method from 0{n^) to O(n^). The experimental results are satisfactory. Compared with the segment-based state space search method, the two-layer assignment method runs slightly faster and has a little lower recognition rate when they were used to recognize the char-acters with an approximate degree of deformation. The two-layer assignment method is also stroke order and stroke number free. 5. A fast string matching method Incorporating the geometric position constraints of strokes (or segments) of Chi-nese characters into Wagner and Fischer's string matching algorithm, we have proposed a fast string matching method for on-line Chinese character recogni-tion. The experiments show that when input characters are written not hav-ing great stroke order deviations, the method can obtain good recognition re-sults. Moreover, the segment-based string matching method can recognize cur-sive handwriting, so it is stroke number free. To allow more stroke order deviations for some characters, using two or more strings to represent one of these model characters is necessary. In this case, we have proposed a scheme to save computational time, which combines two or more separate networks into one and employs a dynamic-programming procedure to solve the shortest path problem. 177 Chapter 7 Conclusions and Suggestions The string matching method is very fast and requires small memory space. When the primitives are segments, it runs 25 times as fast as the segment-based graph matching method does. Therefore, this method can be used in the products that are required to be able to recognize on-line inputted Chinese characters but equipped with low-end CPUs and small memory, such as portable electronic diaries, electronic Chinese-English dictionaries, and multi-functional telephones. 7.2 Suggestions for Further Research Several methods for on-line Chinese character recognition have been proposed in this thesis. They are ready for practical applications. However, to develop a whole perfect system, more work needs to be done. Below we would suggest some directions for further research. 1. Creation of a model graph base The creation of a model graph base for out state space search method or two-layer assignment method is a relatively heavy task. There may be three ap-proaches to this goal. • Completely based on the human knowledge of Chinese characters, this work is done by the people who are familiar with Chinese character hand-writing. In this case, machine learning is not necessary, but the creation and modification of a base are heavy and boring. • The second scheme consists of two steps. In step 1 first, for each model character, collect a set of learning samples which are written in correct 178 Chapter 7 Conclusions and Suggestions stroke orders and without connected strokes, and then with each set of learning samples, build corresponding model graph by examining the prim-itive types and spatially-temporally relations between primitives using a simple program. The relation features generated in step 1 only contain the "don't care" and "should" features. In step 2 some of the "should" features are changed to "must" features by people who are familiar with Chinese character handwriting. This scheme needs to collect a large set of Chinese characters. Changing and modifying the features by people are also a heavy and boring task. • The third approach is based on the fact that Chinese characters are con-structed by a set of components (radicals). Because the number (< 250) of the components are much less than that of frequently-used Chinese charac-ters and the stroke number of each component is less than seven/ building the complete relational graphs of these components is much easier. Using the components, we can form a Chinese character on the screen of a computer. Then by moving the components, we obtain the "don't care" and "should" relations between the primitives of two components. To further obtain the "must" features, we may select two sets of primitives and then choose one of the “must” features ("left o f "right o f "above" and "below") to be the relations between the primitives in the two sets. The movement of components can be completed by one using a mouse or by a program according to some rules. We have estimated that a model graph can be generated within a minutes. This scheme makes the creation 1 Traditionally, there are components with more than six strokes. These components can be constructed by the components with fewer strokes. Chapter 7 Conclusions and Suggestions of a model base easier. In addition, if we change the relations of some component, the modification of relations for all the characters containing this component may be made automatically. We are now developing a tool for this goal. 2. Preclassification and detailed recognition problems There are about 4000 Chinese characters which cover more than 99.9% of the daily-used ones [102]. To extend our state space search method or two-layer as-signment method to recognizing 4000 or more Chinese characters, a preclassifi-cation stage is required for saving computational time. The numbers of segments of Chinese characters is a useful features for choosing primary model character candidates. Estimating some possible components (radicals) in input charac-ters is also a common approaches to preclassification. Many methods have been proposed for the preclassification purpose in on-line or off-line Chinese charac-ter recognition [14, 20, 21, 22, 50, 52, 57 67, 93 102, 103]. By comparing the performances of these methods and combining some features of them, it is not difficult to obtain a good preclassification method. There are some pairs of very similar characters, such as ( )’ , and ( £ , £ ) . The methods proposed in this thesis, like other general recognition methods, cannot distinguish them. Adding an ad hoc detailed recognition stage, we may solve this problem. 3. Improvement of performance of the proposed methods • In the graph representations of Chinese characters, using more spatial relations between, some primitives will enhance its ability to distinguish Chapter 7 Conclusions and Suggestions between very similar characters. The new relations may be the relative spatial relations between the beginning points of two primitives, between the beginning point of a primitive and the end point of another primitive, and so on. Of course, that will increase the work load to build a graph base. • Finding more precise heuristic function /i is a way to farther speed up the A* algorithm. Besides the primitive types, we might use spatial and tem-poral information among primitives to estimate h*. In this case, we have to test whether the total search time is reduced because more computation for calculating h is needed. • In the two-layer assignment method, the spatial and temporal relations between segments are employed to estimate the cost matrixes of the as-signment problems in layer 1. An alternative way is to use the information of segment coordinates. It is worth making a comparison between them to see which is better in the future. 4. Extension to off-line Chinese character recognition Off-line Chinese character recognition is a more difficult task than on-line Chi-nese character recognition. The existing methods can only recognize very neatly-written Chinese characters. By ignoring the temporal relations between primi-tives, our state space search, method and two-layer assignment method can be extended to off-line Chinese character recognition if an approach for extract-ing the segments of a Chinese character is available. Many researchers have investigated the segment or stroke extraction problem [1 2 15’ 24, 49, 53 73 181 Chapter 7 Conclusions and Suggestions Therefore, we expect that our methods will also yield good results when they are applied to off-line Chinese character recognition. 182 Bibliography 1] I. S. I. Abuhaiba, M. J. J. Holt, and S. Datta. Processing of off-line handwritten text: polygonal approximation and enforcement of temporal information. CVGIP: Graphical models and Image Processing, 56:324-335, 1994. 2] L S. L Abuhaiba, M. J. J. Holt, and S. Datta. Processing of binary images of handwritten text documents. Pattern Recognition, 29:1161-1177, 1996. H. Arakawa. On-line recognition of handwritten characters Alphanu-merics, Hiragana, Katakana, Kanji. Pattern Recognition, 16:9-16, 1983. H. Arakawa, K. Odaka, and I. Masuda. On-line recognition of handwritten characters — Alphanumerics, Hiragana, Katakana, Kanji. In Proc. 4th Int. Joint Conf. Pattern Recognition, pages 810—812 Nov. 1978. 5] E. Balas and C. S. Yu. Finding the maximum clique in an arbitrary graph. SI AM J. Comput, 15:1054-1068 1986. 6] D. H. Ballard and C. M. Brown. Computer Vision. Prentice-Hall, Engle-wood Cliffs, New Jersey, 1982. 183 7] P. Bennan and G. Schnitger. On the complexity of approximating the independent set problem. In Lecture N'otes in Computer Science 349, pages 256-267. Springer-Verlag 1989. 8] D. P. Bertsakas. The auction algorithm: a distributed relaxation method for the assignment problem. Annals of Operations Research 14:105-123, 1988. 9] L. Bole and J. Cytowski. Search Methods for Artificial Intelligence. Aca-demic Press, San Diego, CA, 1992. 10] C. Bron and J. Kerbosch. Finding all cliques of an undirected graph. Comm. ACM, 16:575-577, 1973. 11] H. Bunke and G. Allermann. Inexact graph matching for structural pattern recognition. Pattern Recognition Letters, 1:245-253, 1983. 12] R. Carraghan and P. M. Pardalos. An exact algorithm for the maximum clique problem. Operations Research Letters, 9:375-382, 1990. 13] K. P. Chan and Y. S. Cheung. Fuzzy-attribute graph with application to Chinese character recognition. IEEE Trans. Syst. Man Cybem., 22:153-160, 1992. 14] H. D. Chang and J. F. Wang. Preclassification for handwritten Chinese character recognition by a peripheral shape coding method. Pattern Recog-nition, 26:711-719, 1993. [15] H.-D. Chang and J.-F. Wang. A robust stroke extraction method for handwritten Chinese characters. In H. Bunke, P. S. P. Wang, and H. S. 184 Baird editors, Document Image Analysis, pages 1223-1239. World Scien-tific Publishing, Singapore, 1994. 16] J. W. Chen and S. Y. Lee. On-line handwritten Chinese character recog-nition via a fuzzy attribute representation. Image and Vision Computing, 12:669-681, 1994. 17] K.-J. Chen, K.-C. Li, and Y.-L. Chang. A system for on-line recogni-tion of Chinese characters. Computer Processing of Chinese & Oriental Languages, 3:309-318, 1988. [18] L. H. Chen and J. R. Lieh. Handwritten character recognition using a 2-layer random graph model by relaxation matching. Pattern Recognition, 23:1189-1205, 1990. 19] W.-T. Chen and T.-R Chou. A hierarchical deformation model for on-line cursive script recognition. Pattern Recognition, 27:205-219, 1994. [20] K.-S. Chou, K.-C. Fan, and T.-I. Fan. Horizontal segments and peripheral features for use in coarse classification of Chinese characters. In Proc. Int. Symp. Multi- Technology Information Processing, pages 343-348, Taiwan, 1996. [21] K. S. Chou, K. C. Fan, T. I. Fan, C. K. Lin, and B. S. Jeng. Knowledge model based approach in recognition of on-line Chinese characters. IEEE J. Selected Areas Communi., 12:1566-1574, 1994. 185 22] S.-L. Chou and W.-H. Tsai. On-line Chinese character recognition through stroke-segment matching using a new discrete iteration scheme. Computer Processing of Chinese and Oriental Languages, 7:1-20, 1993. 23] W. J. Christmas, J. Kittler, and M. Petrou. Structural matching in com-puter vision using probabilistic relaxation. IEEE Trans. Pattern Anal. Mach. Intell, 17:749-764, 1995. [24] C.-T. Chuang and L. Y. Tseng. A heuristic algorithm for the recognition of printed Chinese characters. IEEE Trans. Syst Man Cybem., 25:710-717, 1995. 25] T. H. Gormen, C. E. Leiserson, and R. L. Rivest. Introduction to Algo-rithms. The MIT Press, MA, 1994’ 26] F. Depiero, M. Trivedi, and S. Serbin. Graph matching using a direct classification of node attendance. Pattern Recognition, 29:1031-1048, 1996. [27) R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley-Interscience, New York, 1973. 28) M. A. Eshera. Image Understanding by Hierarchical Symbolic Represen-tation and Inexact Matching of Attributed Graphs. PhD thesis, Electrical Engineering, Purdue University, 1985. 29] L. R. Foulds. Graph Theory Applications. Springer-Verlag, New York, 1992. [30] K. S. Fu. Syntactic Pattern Recognition and Applications. Prentice-Hall, Englewood Cliffs, New Jersey, 1982. 186 [31] K. S. Fq, Z. X. Cai, and G. Y. Xu. Artificial Intelligence and its Applica-tions. The Publishing House of the Tsinghua University, Beijing, China, 1987. 32] H. N. Gabow and R. E. Tarjan. Faster scaling algorithms for network problems. SIAM J. Corn-put, 18:1013-1036, 1989. 33] S. Gold and A. Rangaxajan. A graduated assignment algorithm for graph matching. IEEE Trans. Pattern Anal Mack. IntelL, 18:377—388 1996. [34] L. Goldfarb. A unified approach to pattern recognition. Pattern Recogni-tion, 17:575-582 1984. 35] L. Goldfarb. A new approach to pattern recognition. In L. N. Kanal and A. Rosenfeld, editors, Progress in Pattern Recognition 2, pages 241-402. Elsevier Science Publishers B. V. North-Holland, 1985. 36] V. K. Govindan and A. P. Shivaprsad. Character recognition — a review. Pattern Recognition, 23:671-683, 1990. 37] S. Hanaki, T. Temma, and H. Yoshida. An on-line character recognition aimed at a substitution for a billing machine keyboard. Pattern Recogni-tion, 8:63-71, 1976. 38] Y. Hidai, K. Ooi, and Y. Nakamura. Stroke re-ordering algorithm for on-line handwritten character recognition. In Proc. 8th Int. Conf. Pattern Recognition, pages 934-936, 1986. 39] F. S. Hillier and G. J. Lieberman. Introduction to Operations Research. McGraw-Hill, New York, 1990. 187 40] A. J. Hsieh, K. C. Fan, and T. I. Fan. Bipartite weighted matching for on-line handwritten Chinese character recognition. Pattern Recognition, 28:143-151 1995. 41] R. A. Hummel and S. W. Zucker. On the foundations of relaxation labeling processes. IEEE Trans. Pattern Anal. Mach. Intell, 5:267-287, 1983. [42] K. Dceda, T. Yamamura, Y. Mitamura, S. Fujiwara, Y. Tominaga, and T. Kiyono. On-line recognition of hand-written characters utilizing posi-tional and stroke vector sequences. In Proc. 4th Int. Joint Conf. Pattern Recognition, pages 813-815, Nov. 1978. 43] S. Impedovo, B. Marangelli, and V. L. Plantamura. Real-time recognition of handwritten numerals. IEEE Trans. Syst. Man Cybem., 6:145-148, 1976. 44] K. B. Irani and S. I, Yoo. A methodology for solving problems: problem modeling and heuristic generation. IEEE Trans. Pattern Anal. Mach. Intell., 10:676-686, 1988. 45] M. S. Kamel, H. C. Shen, A. K. C. Wong, and R. I. Campeanu. System for the recognition of human faces. /BM Systems Journal 32:307-320 1993. '46] R. M. Karp. Reducibility among combinatorial problems. In R. E. Miller and J. W. Thatcher, editors, Complexity of Computer Computations, pages 85-103. Plenum Press, New York, 1972. [47] J. Kobler, U. Schoning, and J. Toran. The Grapg Isomorphism Problem: its Structural Complexity. Birkhauser, Boston, 1993. 188 48] B. Kreko. Linear Programming. Pitman, London, 1968. [49] K.-M. Ku and P. P. K. Chiu. Fast stroke extraction method for hand-written Chinese characters by cross region analysis. Electronics Letters, 30:1210-1212, 1994. 50] T. Kumamoto, K. Toraichi, T. Horiuchi, K. Yamamoto, and H. Yamada. On speeding candidate selection in handprinted Chinese character recog-nition. Pattern Recognition, 24:793-799, 1991. 51] P. Kuner and B. Ueberreiter. Pattern recognition by graph matching combinatorial versus continuous optimization. Int. J. Pattern Recognition and Artificial Intelligence, 3:527-542, 1988. 52] S.-R. Lay, C.-H. Lee, N.-J. Cheng, C.-C. Tseng, B.-S. Jeng, R-Y. Ting, Q , Z. Wu, and M.-L. Day. On-line Chinese character recognition with effective candidate radical and candidate character selections. Pattern Recognition, 29:1647-1659, 1996. 53] S. Lee and J. C. Pan. Offline tracing and representation of signatures. IEEE Trans. Syst. Man Cybem., 22:755-771, 1992. 54] S. Z. Li. Matching invariant to translations, rotations and scale changes. Pattern Recognition, 25:583—594 1992. 55] C.-K. Lin and B.-S. Jeng. On-line recognition of handwritten Chinese characters and alphabets. In Proc. Int. Conf. Acoustics, Speech, Signal Processing, pages 2029-2032 1990. 189 56] C. R, Lin, K. C. Fan, and F. T. P. Lee. On-line recognition by deviation-expansion model and dynamic programming matching. Pattern Recogni-tion, 26:259-268, 1993. '57] T. Z. Lin and K. C. Fan. Coarse classification of on-line Chinese characters via structure feature-based method. Pattern Recognition, 27:1365-1377, 1994. 58] J.-Z. Liu, W. K. Cham, and M. M. Y. Chang. Online Chinese character recognition using attributed relational graph matching. lEE Proc. Vision Image Signal Process., 143:125-131, 1996. 59] J.-Z. Liu, W. K. Cham, and M. M. Y. Chang. Stroke order and stroke number free on-line Chinese character recognition using attributed rela-tional graph matching. In Proc. Int. Conf. Pattern Recognition, pages 259-263, 1996. [60] J.-Z. Liu, W. K. Cham, and M. M. Y. Chang. On-line Chinese character recognition by incorporating human knowledge. Int. J. of Uncertainty, Fuzziness, and Knowledge-Based Systems, 5:13-29 1997. 61] J.-Z. Liu, W. K. Cham, and M. M. Y. Chang. A spatial-temporal approach to on-line Chinese character recognition. In Proc. Int. Conf. Computer Processing Oriental Languages, pages 258-261, 1997. 62] J.-Z. Liu, W. K. Cham, and M. M. Y. Chang. A spatial-temporal method for on-line Chinese character recognition. Communications of COLIPS, 7:31—39 1997. 63] J.-Z. Liu, W. K. Cham, and M. Y. Chang. On-line Chinese character recognition with attributed relational graph matching. In R. T. Chin, H. H. S. Ip, A. C. Naiman, and T. C. Pong, editors, Image Analysis Applications and Computer Graphics pages 189-196. Springer, 1995. 64] Y. J. Liu and J. W. Tai. An on-line Chinese character recognition system for handwritten in Chinese calligraphy. In From Pixel to Features III —Frontiers in Handwriting Recognition, pages 87-99. Elsevier Science Publishers B. V. 1992. [65] S.-C. Loh, C.-W. Chan, and S.-C Chan. On-line recognition of hand-written Chinese characters. In Int. Conf. Computer Processing Chinese & Oriental Languages, pages 58—61 1988. 66] W. W. Loy and I. D. Landau. An on-line procedure for recognition of handprinted alphanumeric characters. IEEE Trans. Pattern Anal. Mach, IntelL, 4:422-427, 1982. 67] H. Lu and P. Yang. A preclassification method for Chinese character recog-nition based on peripheral stroke structure. Communications of COLIPS, 2:73-79, 1992. [68] S. W. Lu, Y. Ren, and C. Y. Suen. Hierarchical attributed graph rep-resentation and recognition of handwritten Chinese characters. Pattern Recognition, 24:617-632, 1991. 69) A. Marzal and E. Vidal. Computation of normalized edit distance and applications. IEEE Trans. Pattern Anal. Mach. Intell, 15:926-932, 1993. 191 70] K. G. Murty. Network Programming. Prentice-Hall, Englewood Cliffs, New Jersey, 1992. [71] N. J. Nilsson. Principles of Artificial Intelligence. Tioga Publishing Co., Palo Alto, CA, 1980. 72] K. Odaka, H. Arakawa, and I. Masuda. On-line recognition of handwritten characters by approximating each stroke with several points. IEEE Trans. Syst Man Cybem., 12:898-903, 1982. 73] H. Ogawa and K. Taniguchi. Thinning and stroke segmentation for hand-written Chinese character recognition. Pattern Recognition, 15:299—308 1982. 74] J. B. Orlin and R. K. Ahuja. New scaling algorithms for the assignment and minimum cycle mean problems. Mathmatical Programming, 54:41-56, 1992. 75] C. H. Papadimitriou and K. Steiglitz. Combinatorial Optimization — Algorithms and Complexity. Prentice-Hall, Englewood Cliffs, New Jersey, 1982. 76] J. Pearl. Knowledge versus search: a quantitative analysis using a*. Arti-ficial Intelligence, 20:1-13 1983. 77] J. Pearl. Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison-Wesley, Reading, MA, 1984. 78] J. Pearl. Some recent results in heuristic search theory. IEEE Trans. Pattern Anal. Mach. IntelL, 6:1-13 1984. 192 79] A. Sanfeliu and K.-S. Fu. A distance measure between attributed relational graphs for pattern recognition. IEEE Trans. Syst. Man Cybem. 13:353-362 1983. 80] S. Sarkar and K. L. Boyer. Computional structure for preattentive per-ceptual organization: graphical enumeration and voting methods. IEEE Trans. Syst. Man Cybem., 24:246-267, 1994. 81] R. Schalkoff. Pattern Recognition: Statistical, Structural and Neural Ap-proaches. John Wiley k Sons, Inc., New York, 1992. 82] D. S. Seong, H. S. Kim, and K. H. Park. Incremental clustering of at-tributed graphs. IEEE Trans. Syst Man Cybem., 23:1399-1411, 1993. 83] L. G. Shapiro, J. D. Moriarty, R. M. Haralick, and P. G. Mulgaonkar. Matching three-dimensional objects using a relational paradigm. Pattern Recognition, 17:385-405, 1984. 84] P. N. Suganthan, E. K. Teoh, and D. P. Mital. Pattern recognition by graph matching using the Potts MFT neural networks. Pattern Recogni-tion, 28:997-1009, 1995. 85] P. N. Suganthan, E. K. Teoh, and D. P. Mital. Pattern recognition by homomorphic graph matching using hopfield neural networks. Image and Vision Computing, 13:45-60, 1995. 86] J. W. Tai and Y. J. Liu. Chinese character recognition. In Syntactic and Structural Pattern Recognition, Theory and Applications^ pages 415-452. World Scientific, 1989. 193 87] C. C. Tappert. Speed, accuracy, flexibility trade-offs in on-line character recognition. Technical report, IBM Res. Rep. RC13228 1987. 88] C. C. Tappet, C. Y. Suen, and T. Wakahara. The state of the art in on-line handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell” 12:787-808, 1990. [89] C. Thornton and B. D. Boulay. Artificial Intelligence through Search. Kluwer Academic Publishers, The Netherlands, 1992. 90] W.-H. Tsai and K.-S. Fu. Error-correcting isomorphisms of attributed relational graphs for pattern analysis. IEEE Trans. Syst. Man Cybem., 9:757-768, 1979. [91] W.-H. Tsai and K.-S. Fu. Subgraph error-correcting isomorphisms for syntactic pattern recognition. IEEE Trans. Syst. Man Cybem., 13:48-62 1983. 92] Y. T. Tsay and W. H. Tsai. Attributed string matching by split-and-merge for on-line Chinese character recognition. IEEE Trans. Pattern Anal. Mach. Intell. 15:180-185, 1993. 93] C.-C. Tseng, S.-L. Lay, B.-S. Jeng, and K.-S. Chou. Candidate selection in on-line Chinese character recognition system using voting scheme. In Proc. Int. Symp. Multi-Technology Information Processing, pages 355-360, Taiwan, 1996. 194 94] J. Tsukumo. Handprinted Kanji character recognition based on flexible template matching. In Proc. 11th Int. Conf. Pattern Recognition, pages 483-486, 1992. 95] J. Tsukumo and H. Tanaka. Classification of handprinted Chinese char-acters using non-linear normalization and correlation methods. In Proc. 9th Int. Conf. Pattern Recognition, pages 168-171, 1988. 96] R. A. Wagner and M. J. Fischer. The string-to-string correction problem. J. ACM, 21:168-173, 1974. 97] T. Wakahara, H. Murase, and K. Odaka. On-line handwritten recognition. Proc. IEEE, 80:1181-1194 1992. [98] J. T. L. Wang, K. Zhang, and G.-W. Chirn. Algorithms for approximate graph matching. Information Sciences, 82:45-74, 1995. 99] R. C. Wilson and E. R. Hancock. A bayesian compatibility model for graph matching. Pattern Recognition Letters, 17:263—276 1996. [100] A. K. C. Wong, S. W. Lu, and M. Rioux. Recognition and shape synthesis of 3-d objects based on attributed hypergraphs. IEEE Trans. Pattern Anal. Mach. Inteli, 11:279-290, 1989. 101] A. K. C. Wong, M. You, and S. C. Chan. An algorithm for graph optimal monomorphism. IEEE Trans. Syst. Man Cybem” 20:628-636, 1990. [102] Y. Wu and X. Ding. Chinese Character Recognition—Principles, Methods and Implementations. The Higher Education Publishing House, China, 1992. 195 103] Y. Wu, N. Xu, and X. Ding. A new clustering method for Chinese character recognition system using neural networks. ACTA Electronica Sinica, 22:1-8, 1994. [104] B. Zhang and L. Zhang. A new heuristic search technique algorithm SA. IEEE Trans. Pattern Anal. Mack. Intell., 7:103—107 1985. 105] B. Zhang and L. Zhang. Statistical heuristic search. J. Computer Science Technology, 2:1-11 1987. 106] S. W. Zucker, E. V. Krishnamurthy, and R. L. Haar. Relaxation processes for scene labeling: convergence, speed, and stability. IEEE Trans. Syst. Man Cybem., 8:41-48, 1978. IMAGE EVALUATION TEST TARGET (QA—3) /APPLIED A IIVL^GE • Inc 1653 East Mam Street Rochester. NY 14609 USA 716/432-0300 Rights Reseiveo A: •5 Mmk • Li 0 u i. wu I.I 1.4 0 m m 25

On-line Chinese character recognition.

Abstract

Similar works

Full text

Available Versions

CUHK Digital Repository