This paper discusses the use of pen pressure as a feature in writer-independent on-line handwriting recognition. We propose two kinds of features related to pen pressure: one is the pressure representing pen ups and downs in a continuous manner; the other is the time-derivative of the pressure representing the temporal pattern of the pen pressure. Combining either of them with the existing feature (velocity vector), a 3-dimensional feature is composed for character recognition. Some techniques of interpolating the pen pressure during the pen-up interval is also proposed for a
pre-processing purpose. Through experimental evaluation using 1,016 elementary Kanji characters compared with the baseline performance using velocity vector only, the additional use of pen pressure improved the performance from 97.5% to 98.1% for careful writings and from 91.1% to 93.1% for cursive writings

Nakai, Mitsuru

Sagayama, Shigeki

Shimodaira, Hiroshi

Sudo, Takashi

Edinburgh Research Archive

16th International Conference on Pattern Recognition (ICPR 2002), Vol.III, pp.220-223 (2002-08)Pen Pressure Features for Writer-Independent On-line Handwriting RecognitionBased on Substroke HMMMitsuru NAKAI, Takashi SUDO, Hiroshi SHIMODAIRA and Shigeki SAGAYAMA∗Graduate School of Information Science, *Graduate School of Information Science and Technology,Japan Advanced Institute of Science and Technology The University of Tokyo{mit,sudotaka,sim}@jaist.ac.jp sagayama@hil.t.u-tokyo.ac.jpAbstractThis paper discusses the use of pen pressure as a featurein writer-independent on-line handwriting recognition. Wepropose two kinds of features related to pen pressure: oneis the pressure representing pen ups and downs in a contin-uous manner; the other is the time-derivative of the pres-sure representing the temporal pattern of the pen pressure.Combining either of them with the existing feature (velocityvector), a 3-dimensional feature is composed for charac-ter recognition. Some techniques of interpolating the penpressure during the pen-up interval is also proposed for apre-processing purpose. Through experimental evaluationusing 1,016 elementary Kanji characters compared with thebaseline performance using velocity vector only, the addi-tional use of pen pressure improved the performance from97.5% to 98.1% for careful writings and from 91.1% to93.1% for cursive writings.1. IntroductionHandwriting recognition is a useful character inputmethod to small PDAs (personal digital assistant), in whichit is desirable to be able to input quickly like memorandumwriting. In such an environment, we may write characterswhich are hard to read by appearance, since a stroke tendsto overlap and a character sometimes lose its shape. Ourproposed on-line handwriting recognition system based onsubstroke HMM[4] is robust to such overlaps of strokes byemploying a velocity feature vector that is a difference be-tween two consecutive pen positions. For the further im-provement of the recognition accuracy, it is necessary to in-vestigate the high-dimensional features that combined vari-ous information obtained from an input device.Today, we can detect handwriting pressures on severalpressure-sensitive tablets. The pen pressure informationcontains much individuality of writers[7] and it is often em-ployed for writer identification[2]. However, for writer-independent handwriting recognition, pen pressure infor-mation is used only as a binary feature indicating pen-up/pen-down[3, 6], since it is thought that the pen pres-sure pattern is not a discriminant feature for handwrittencharacters. In Japanese, the pen pressure pattern is onlyused in a few researches[5, 8]. In those researches, somenormalizing technique is needed for removing writer’s indi-viduality from the pen pressure. Accordingly, their recog-nition processing is performed after a character is written.Here, we propose on-line pen pressure features for writer-independent handwriting recognition. That is, the pen pres-sure features are extracted by using short-time analysis andit does not need off-line normalization processing. There-fore, these features can be directly used for our HMM-basedrecognition system.2. Handwriting Recognition Based on Sub-stroke HMM with Pen Pressure2.1. Extraction of Writing Velocity, Direction andPressure FeaturesThe proposed system basically consists of input features,HMMs, dictionaries and a decoder as shown in Fig. 1. Inthis paper, we use pen positions (x, y) and pen pressure(z) sampled at a certain interval from the pen tablet. Let(dx, dy) be the difference between two consecutive pen posi-tion samples and (r, θ) be the primary feature vector, wherer means the velocity (√dx2 + dy2) and θ means the direc-tion of the velocity vector[1].In this paper, we propose two new features, the pen pres-sure itself and its time-derivative, and combine them withthe existing velocity feature vector.2.1.1 Pen PressureThe pen pressure represents pen ups and downs in a con-tinuous manner. In our conventional research, we did notuse pen-up-down information whether the pen leaves (pen-up) or touches (pen-down) the tablet surface. Instead, apen-up is probabilistically recognized by observing velocityfeature vectors, where the feature (dx, dy) during pen-up in-terval means comparatively large displacement vector fromthe pen-up position to the next pen-down position. On the— 220 —ReferencesDecodetemporal sequenceDictionaryFeatures (velocity vector)HMMsSubstrokeHandwriting木才 :木 :材 :一 =十 =一 +才 =十 +木 =才 +材 =木 +才林 =木 +木森 =木 +林Figure 1. System configuration.other hand, in this paper, the pen pressure (z) is sampled ata certain interval and let the value itself be one of featureset. Also, a trajectory of the pen-up interval is sampled si-multaneously; here we use a pen tablet that can detect penpositions even if the pen leaves the tablet surface about 8mm.2.1.2 Time-Derivative of Pen PressureAlthough strength of the pen pressure is different in eachwriter, it is thought that the pattern of pen pressure fluctu-ation is common to many writers. Therefore, the pen pres-sure is regarded as the 3rd axis and we observe the pen tra-jectory (xt, yt, zt) in 3-dimensional space. Combining withthe existing features, let (rt, θt,∆zt) be the feature vector,where ∆zt is a regression coefficient of pen pressure and itis analyzed by∑Li=−L izt+i/∑Li=−L i2. In the following sec-tions, we call this feature a delta pressure.2.2. Substroke HMMsWe model 25 substrokes of eight directions as shownin Fig. 2; eight long strokes (‘A’–‘H’), eight short strokes(‘a’–‘h’), eight pen-up movement (‘1’–‘8’) and onepen-up-down movement (‘0’). The HMMs of thesesubstrokes have a topology of left-to-right model asshown in Fig. 3. The pen-down models have threestates representing different stroke velocities, and pen-upmodels have only one state. It differs from the con-ventional models[4] that self-transition probability isadded to pen-up model. Here, let λ(k) = (A(k), B(k),π(k))CABDEFGHabcdefgh34 28765 10Figure 2. Substroke categories: A–H (a–h) arelong (short) substrokes with pen-down and0–8 are the direction of pen-up.S 1 S 2 S 3 S 1π1a11a12a22a23a33a34 π1a11a12b1(o) b2(o) b3(o) b1(o)Figure 3. Substroke HMMs : (Left) pen-downmodel, (Right) pen-up model.be the set of HMM parameters of substroke k, in whichA(k) = {a(k)i j } : the state-transition probability distri-butions from state S i to S j,B(k) = {b(k)i (o)} : the probability distributions of obser-vation symbols o at state S i,π(k) = {π(k)i } : the initial state probability distribu-tions.The observation probability distribution is represented by aM mixtures of Gaussian distribution given bybi(o) =M∑m=1cimexp(− 12 (o− µim)tΣ−1im (o− µim))√(2π)n|Σim|,with mean vector µ, covariance matrix Σ and weighting co-efficient c. Here, the direction feature (θ) has a continuousprobability distribution with 2π cycle[1]. These model pa-rameters can be trained by Viterbi training or Baum-Welchmethod.2.3. RecognitionA decoder recognizes an input pattern by referring to thecharacter’s substroke sequence expanded from the hierar-chical structure dictionary[4]. For example, the definitionof the character “ 二 ” is ‘a 6 A’ that represents two pen-down strokes ‘a’ and ‘A’ are connected with pen-up model‘6’ in standard stroke order. Similarly, “子 ” is ‘A f 0 G d 4A’ and “字 ” is ‘g 5 g 3 A f 6 A f 0 G d 4 A’, where “子 ”is the partial structure of “ 字 ” and those have a common— 221 —• •• ••• • •• • • • • • • • • • • •• • • • • • •• • •• • • • • • • • • • ••• • • • •• • •• • • ••0 10 20 30 40 50 600.00.51.0• •• ••• • •• • • • • • • • • • • •• • • • • • •• • •• • • • • • • • • • •••• • • • • • • • • •• • •• •• • • • • ••0 10 20 30 40 50 600.00.51.0(c) Interpolated pattern by cubic splines(b) Original pen pressure pattern(a) HandwritingPressurePressureTimeTimeFigure 4. Cursive handwriting “ 下 ” and itsinterpolated pattern of the pen pressure.substroke sequence. The decoder generates the connectionmodel of each character from these character definitions andsubstroke HMMs, and then calculates the probability thatan input pattern is produced from those HMMs by Viterbisearch algorithm in a substroke network[4].3. Pre-processing of Feature Extraction forCursive WritingThe pen-up of handwriting can be probabilistically rec-ognized in both case of using conventional velocity featurevector and using the pressure together. For example, evenif we write “ 了 ” (two strokes) with one stroke or write“廴 ” (three strokes) with two strokes, probably they can berecognized correctly. However, in extreme cursive writings,using the pressure information may conversely reduce therecognition accuracy, since the pressure characterizes pen-up-down and a pen-up may sometimes disappear by writ-ing two or more strokes with one drawing. To overcomethis problem, we propose a technique of interpolating thepen pressure information on the interval that carried out thepen-up correctly. Fig. 4 is an example of the cursive writing“ 下 ” (three strokes) and the 1st stroke and the 2nd strokeare drawn continuously (connected-stroke). Fig. 4(c) showsthe result of approximating the pressure value of the pen-upinterval to the fall-rise pattern of the connected-stroke, sothat we can train only one substroke HMM per pen-up la-bel. The interpolation algorithm is as follows. Let t1 be atime when the pen leaves tablet and t2 be a beginning timeof next pen-down. At first, we set a pen pressure differencebetween zt1 and zt2 to 0.0 by adding a value zt2 − zt1 to thepressure zt (t > t2). Next, using the cubic splines method, Spoints in the pen-up interval is interpolated. Here, the penpositions (x, y) during the pen-up are also interpolated sothat the velocity feature vector may become constant.4. Experimental EvaluationHandwriting database used in this evaluation is theJAIST IIPL (Japan Advanced Institute of Science and Tech-Table 1. Comparison of features in carefulhandwriting recognitionN-best accumulative recognition rate (%)Features 1 ∼ 2 ∼ 3 ∼ 5 ∼ 10(a) with pen-up trajectoriesr, θ 94.81 97.69 98.53 99.01 99.36r, θ, z 98.08 99.53 99.71 99.77 99.83r, θ,∆z 96.14 98.41 99.02 99.28 99.57(b) without pen-up trajectoriesr, θ 97.47 99.23 99.60 99.77 99.87r, θ, z 97.98 99.56 99.78 99.84 99.89r, θ,∆z 97.54 99.27 99.63 99.80 99.87nology, Intelligence Information Processing Laboratory)database that consists of several kinds of data sets. Amongthem, we used two kinds of dataset written with standardstroke order; one was the careful writing dataset (γ set) withright stroke-count and the other was cursive writing dataset(ε set). Both datasets cover 1,016 Japanese characters of oldand new educational Kanji.4.1. Experiment 1: Careful Writing Recognitionwith Pen PressureAn experimental comparison of features was carried outon the 1,016 Japanese educational Kanji recognition task.The odd-numbered 30 writers from γ dataset were usedfor estimating the HMM parameters by Viterbi trainingmethod, and the remaining even-numbered 30 writers wereused for test. Additionally, we also compared two casesabout whether pen-up trajectories were used or not. TheHMMs were trained as 2 mixture Gaussian distributions.The recognition results according to the feature sets areshown in Table 1. Generally, the recognition rates becomehigh when the pen pressure information is used. About thetrajectory of the pen-up interval, the recognition rate is im-proved only when it is used together with the pressure fea-ture (z).4.2. Experiment 2: Cursive Writing Recognitionwith Pen-up InterpolationFrom the ε dataset, the odd-numbered 34 writers wereused for training HMMs, and the remaining even-numbered34 writers were used for test. The other experimental con-ditions were the same as Experiment 1. However, the eval-uation was carried out with or without the delta pressurefeature (∆z). It is thought that the pressure feature (z) isunsuitable to use in the cursive writing recognition, wherestrong pen pressure is observed during pen-up interval ofconnected-stroke.Fig. 5 shows the recognition rates when varying the num-ber of interpolated points S from 1 to 8. In S ≥ 3, the delta— 222 —2 4 6 8909192931 3 5 7Recognition Rate (%)Number of Interpolated Points: S(r, θ)(r, θ,∆z)Figure 5. Recognition rate with the number ofinterpolated points.0102030401 5 10 15 20 25 30 34010203040WriterError Rate (%)(r, θ)(r, θ,∆z)Figure 6. Error rate reduction for each writer.pressure feature was effective in improving the recognitionaccuracy. As a result of training pen-up HMMs by usingsamples written with only one drawing, the self-transitionprobabilities of pen-up models were from 0.7 to 0.85. Then,the expected number of observed point in a pen-up intervalwas from 3 to 6. In correspondence to that, the recognitionrate became the maximum in the case of S = 5.The recognition rate for each writer is shown in Fig. 6.We can see that a writer-independent model with the deltapressure was effective to many writers; the recognition ratewas improved for 19 writers. This result means that the pat-tern of pen pressure fluctuation is common to many writers.The improved sample of handwritings are shown inFig. 7. Conversely, the samples that have changed to in-correct recognition are shown in Fig. 8. Here, “団→因 ”means that the recognition result changed from “ 団 ” to“因 ” by using the delta pressure features. Totally, the num-ber of improvement was 1,029 writings and the number ofincorrectly changed characters was 89 writings.Incidentally, the recognition rate of careful writingrecognition is also improved to 97.91%. It turns out that theeffectiveness of the pen-up interpolation does not depend onthe quality of handwritings.団→因引→市砂→声仏→以賀→損大→士Figure 7. Improved samples by using ∆ pres-sure feature.冊→皿系→条切→功耳→可公→父戸→平Figure 8. Error samples by using ∆ pressurefeature.5. ConclusionWe have proposed on-line pen pressure features forwriter-independent handwriting recognition and shown itsvalidity through experiments. Furthermore, the pre-processing technique for handwritings has been proposed,and improvement in the recognition rate was achieved inboth of cursive writings and the careful writings. Weexpect that the recognition accuracy of cursive writingswould be improved further by combining substroke context-dependent model.References[1] N. Akira, M. Nakai, H. Shimodaira, and S. Sagayama. “AnInvestigation into the Representation of Feature Vectors andtheir Stochastic Modeling by Stroke-based HMM for On-lineHandwriting Character Recognition” (in Japanese). Technicalreport of IEICE, PRMU2000-134:31–38, Dec. 2000.[2] M. Kikuchi and N. Akamatsu. “Development of Speedy andHigh Sensitive Pen System for Writing Pressure and WriterIdentification”. Proc. ICDAR’01, pages 1040–1044, Sept.2001.[3] A. Kosmala, J. Rottland, and G. Rigoll. “An Investigationof the Use of Trigraphs for Large Vocabulary Cursive Hand-writing Recognition”. Proc. ICASSP’97, 4:3373–3376, Apr.1997.[4] M. Nakai, N. Akira, H. Shimodaira, and S. Sagayama. “Sub-stroke Approach to HMM-based On-line Kanji HandwritingRecognition”. Proc. ICDAR’01, pages 491–495, Sept. 2001.[5] Y. Sato and H. Adachi. “Online Recognition of Cursive Writ-ings” (in Japanese). IEICE Trans. (D), J68-D(12):2116–2122,Dec. 1985.[6] K. Takahashi, H. Yasuda, and T. Matsumoto. “A Fast HMMAlgorithm for On-line Handwritten Character Recognition”.Proc. ICDAR’97, 1:369–375, Aug. 1997.[7] Y. Yamazaki, Y. Mizutani, and N. Komatsu. “Extraction ofPersonal Features from Stroke Shape, Writing Pressure andPen Inclination in Ordinary Characters”. Proc. ICDAR’99,pages 426–429, Sept. 1999.[8] P. Zhao, Y. Sato, and M. Yoshimura. “Generating GenericReference for Cursive Wrirings” (in Japanese). IPSJ Trans.,34(3):418–425, Mar. 1993.— 223 —

Pen pressure features for writer-independent on-line handwriting recognition based on substroke HMM

https://era.ed.ac.uk/bitstream/1842/958/1/Nakai2002ICPR.pdf

Pen pressure features for writer-independent on-line handwriting recognition based on substroke HMM

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Archive