Pece, Arthur

Petkov, Nikolay

English

A new algorithm is introduced which is related to matching pursuit but allows updating more than one coding coefficient per iteration: the updated coefficients correspond to mutually orthogonal elements of the dictionary. Coding experiments on natural images show that the new method achieves the same trade-off as matching pursuit between number of coding coefficients and reconstruction error, but significantly faster convergence

ARTS repository - University of Groningen

   University of GroningenFast atomic decomposition by the inhibition methodPece, Arthur; Petkov, NikolayPublished in:EPRINTS-BOOK-TITLEIMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2000Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Pece, A., & Petkov, N. (2000). Fast atomic decomposition by the inhibition method. In EPRINTS-BOOK-TITLE University of Groningen, Johann Bernoulli Institute for Mathematics and Computer Science.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 08-06-2022Fast atomic decomposition by the inhibition method ArthurPece Department of Comput�r Science University of Reading P.O.Box 225, RG6 6AY Reading. UK. A.E.C.P ece@reading.ac.uk Abstract A new algorithm is introduced which is related to match­ing pursuit but allows updating more than one coding co­efficient per iteration: the updated coefficients correspond to mutually orthogonal elements oj the dictionary. Coding experiments on natural images show that the new method achieves the same trade-off as matching pursuit between number oj coding coefficients and reconstruction error. but significantly Jaster convergence. 1. Introduction In many image-coding schemes, an input image is rep­resented as a linear combination of other images, which are elements of a set called the dictionary or code book. It is often necessary to decompose the input image into as small a number of elements as possible: such a strategy (called atomic decomposition, adaptive representation, or sparse coding) has obvious advantages for storage, transmission. de-noising, and pattern recognition. When the dictionary is over-complete, finding the expansion with the smallest re­construction error for a given number of (non-zero) coding coefficients is an NP-hard problem [6]. Several iterative al­gorithms have been proposed for finding a sub-optimal so­lution (e.g. [4, 10, 11, 13]). The main drawback of these algorithms is their computational cost. Since its introduc­tion, matching pursuit [10] has been the standard of com­parison for this class of algorithms. This article introduces a new algorithm for atomic decomposition, the inhibition method, which can be considered a generalization of match­ing pursuit and offers faster convergence to a given recon­struction error, while achieving the same sparseness of en­coding. In the next Section, matching pursuit and the in­hibition method are discussed within the same framework; experimental results are presented in Section 3 followed by the conclusions in Section 4. 0-7695-0750-6/00 $10.00 © 2000 IEEE 211 Nikolay Petkov Department of Computing Science University of Groningen P.O.Box 800, 9700 AV Groningen The Netherlands petkov@cs.rug.nl 2. Description of the algorithms The image to be encoded is the vector f of n pixel values; the linear image size is L = v'n. The dictionary D includes m (normalized) elements gi E D, Ilgill = 1, i = 1. . . m. These elements are vectors of n pixel values. Given that the dictionary is overcomplete, n < m. e is a vector of m cod­ing coefficients. The linear expansion 2:�1 cigi is the re­constructed image. The difference between input image and reconstructed image r = f - 2::1 cigi is the residual im­age. The squared L2 norm ofr, normalized by the squared norm of f, is the normalized squared reconstruction error NSE = IIrll2 / IIf1l2, which is used as the criterion of how well the reconstructed image approximates the input image (thePSNR measure is linearly related to the logarithm of the NSE). p is the vector of m inner products of the residual with the m dictionary images: Pi = g[ . r, i = 1 . . .  m. When required for clarity, we write e(t), ret) , pet), where t is the iteration number. The symbol .:le denotes the update of the coefficients at each iteration: .:le( t) = e(t) - e(t - 1). Similarly, .:lr(t) = ret) - ret - 1) and .:lp(t) = pet) - pet - 1). 2.1. Basic iteration Rather than describing matching pursuit and the inhibi­tion method separately, it is easier to introduce a common framework for both methods and highlight the differences within this framework. After the vector of inner products p(O) is initialized, the basic iteration consists of two steps: (I) updating one or more coding coefficient and (II) re-computing the inner products between all the dictionary elements and the new residual image. The coefficients to be updated should be chosen so as to combine a large decrease of NSE with a small increa�e of the number of non-zero coefficients. One possible heuristic is to update only the one coefficient which can bring the largest decrease of NSE, i.e. the coefficient corresponding to the largest element of p. This heuristic is the basis of matching pursuit. To achieve faster convergence, it is desirable to update more than one coefficient per iteration. However, most of the largest inner products are likely to arise from strongly correlated dictionary elements, i.e. elements representing the same feature of the input image: updating all of the cor­responding coefficients would produce a redundant, rather than sparse, representation. A solution to this problem is to update coefficients corresponding to dictionary elements which have larger (in absolute value) inner products with the residual than any other non-orthogonal dictionary ele­ment. This heuristic is the basis of the inhibition method. The name 'inhibition method' is inspired by the analogy with reciprocal inhibition of neurons with non-orthogonal Gabor-like receptive fields in the visual cortex (see e.g. [2]). In detail, the two steps of the iteration are as follows: I. Select one or more coefficients and update them by adding the values of the corresponding inner products: if gi E S(t} { Pi(t} o otherwise (i = 1 . . . m) (1) The definition of the subset SeD is the essential differ­ence between matching pursuit and the inhibition method. In matching pursuit, S is defined by: SMP = {gi ED I 'r/j i= i IPil > Ipjl} (2) Obviously, by Eq.2, SMP includes only one element. Ideally, in the inhibition method, S should be defined by: S1 = {gi E D I 'r/j =t i Igf . gjl > f '* IPil > Ipjl} (3) where t: is a small constant (t: = 0.05 in our experiments). In practice, implementing Eq.3 would require too many com­parisons between elements ofp. However, comparisons can be restricted to the k largest elements of p (for k < < m): by this approximation, the coefficients to be updated corre­spond to the elements of the set S 1M = Sk U S I, where Sk is the set of the k dictionary elements having largest in­ner products with the residual image (in our implementation k = 2047). By setting k = 1, matching pursuit is obtained as a special case of the inhibition method. A look-up table can be used to determine whether two dictionary elements are orthogonal. By exploiting symmetries in the dictionary, e.g. if the elements are shifted, scaled and rotated versions of one or a few kernels, this table requires storage of O(m}, rather than O(m2}. It is easy to see that the inhibition method will converge whenever matching pursuit converges: the largest (in abso­lute value) element of the vector p is always selected, there­fore at least one coefficient is updated at each iteration. 212 II. Update the vector of inner products, taking into ac­count the change in the residual. Step II can be broken down into two sub-steps: lIa. Update the residual image: m Ar(t + 1) = -L: ACi(t) gi ;=1 (4) If few coefficients have been updated, the computational cost of sub-step IIa is negligible compared to the cost of sub-step lIb. For large numbers of updated coefficients, if the dictionary can be decomposed into shifted versions of a few kernels, then the update of the residual can be efficiently carried out by the application of the FFT. This is the case with our dictionary. lIb. Re-compute the inner products of the residual image r with all dictionary elements: 6.Pi(t + 1) = g; . Ar(t + 1} (i = 1. .. m) (5) Again, this convolution can be carried out efficiently by the application of the FFT. Alternatively, step II can be carried out by updating the inner products directly, using the relation: m 6.Pi(t + 1} = -L 6.Cj . (gf . gj) (i=1 .. . m) j=l (6) If Eq.6 is implemented, then the computational cost of step II becomes proportional to the number of updated coeffi­cients. In this case, updating a number II of coefficients by the inhibition method in one iteration would require as much computation as updating II coefficients by matching pursuit in II iterations. However, this technique is compet­itive only if most dictionary elements are mutually orthog­onal (which is not the case in a highly oversampled dictio­nary), the inner products between elements can be quickly computed analytically, and Fourier techniques cannot be used to speed up convolutions and deconvolutions (i.e. Eq.4 and Eq.5). None of these conditions applies in our case. 2.2. Dictionary The dictionary used in the coding experiments is com­posed of two-dimensional real-valued Gabor functions [5] of unit L2 norm and zero mean (so that the DC compo­nent of the input image was encoded separately). The Gabor functions are of the form: x=(�) I: _ ( ax 0 ) - 0 a" x = Xo + u cos e - v sin e Y = Yo + u sin e + v cos e where Z is a normalizing constant; u and v are image coordinates; Xo, yo, CJ,; and CJy are parameters of a two­dimensional Gaussian window; ). and t.p are parameters of a sinusoidal grating and e is the angle between the u axis of the image and the x axis of the Gabor function. In our dictionary, two of the parameters are fixed by the relations: (7y/(7x = -./2 and )../CJ,; = -./27r. The remaining scale pa­rameter ranges over 11 different scales [,\ == 2 . a8 pixels, a = -./2, 0 :::; s :::; loga(L/4)]. The parameters Xo and Yo assume all integer values between 1 and L, i.e. the Gabor functions are centered on all image pixels (no downsam­pling). The other parameters range over 8 different orien­tations [0 == (q/8)7r radians , 0 :::; q < 8] and 2 different phases ['I' E {a, 7r /2} J, except at the smallest spatial scale (,\ == 2 pixels), for which only Gabor functions in cosine phase ('I' == 0) and with horizontal or vertical orientations [6 E {O, 7r /2}] are used, to avoid aliasing. The dictionary is thus overcomplete by a factor of 10 . 8 . 2 + 2 = 162. For images of linear size L = 128 pixels. this sampling scheme results in approximately 2.65 million dictionary elements. Gabor functions with odd symmetry (t.p = 7r /2) have zero mean. The DC component of Gabor functions with even symmetry ('I' = 0) was set equal to zero in the fre­quency domain. 3. Coding results The inhibition method was tested on a set of 19 natu­ral images. Fig.l shows the results obtained with the Lena image: the NSE is plotted as a function of both number of iterations and number of coefficients; the two curves are undistinguishable for matching pursuit. As can be seen, the two algorithms offered the same trade-off between NSE and number of coding coefficients: the data points (crosses) for the inhibition method lie on the data line for matching pur­suit. On the other hand, the number of iterations required to reach a given NSE was smaller for the inhibition method than for matching pursuit, except for the first two iterations, in which the inhibition method only updated one coefficient per iteration. As the main low-frequency, large-scale com­ponents of the image were removed, it became possible for the inhibition method to detect more and more mutually orthogonal image components simultaneously, so that the number of updated coefficients increased with the iteration number. Over the entire set of 19 images, the number of iterations needed for the inhibition method to reach NSE = 0.01 was 42.1±4.4 (average ± standard deviation) and the number of coefficients needed to reach NSE = 0.01 was 3370 ± 1179 213 § III c:: 0 '£ " � c: 0 <.> '" 0:: 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1 o o o o 10 M.P. ­LM. (itemtions) 0 I.M. (coefficients) + 100 1000 ]0000 Numbers of iterations and of coding coefficients Figure 1. Performance of the two algorithms on the Lena image. (average ± standard deviation); in other words, the num­ber of iterations was approximately constant, irrespective of the number of coefficients required to encode the image. The average number of coefficients updated per iteration was 77.0. Assuming that one iteration of the inhibition method has almost twice the computational cost of one iter­ation of matching pursuit (because the deconvolution step, Eq.4, has negligible cost in matching pursuit), we conclude that the inhibition method converges about 38 times faster than matching pursuit. This average hides the fact that the inhibition method is not much faster than matching pursuit in the first few iterations, but can be over 100 times faster in the last few iterations. These estimates are based on the assumption that match­ing pursuit would encode all the images with about the same number of coefficients as used by the inhibition method. In addition to the Lena image (Fig. I), this assumption was tested on the two images most likely to prove it wrong: the image which the inhibition method encoded with the largest number of coding coefficients; and the image (shown in Fig.2) which was encoded at the fastest rate by the inhi­bition method: the more coefficients are updated per iter­ation, the higher the likelihood that redundant coefficients are updated. In both cases, the plots of NSE VS. number of coefficients were overlapping, as was the case for Lena. 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0 0 M.P. ­LM. (iterations) 0 I.M. (coefficients) + 0 '--��"'-�3o<o.""""��""""'�-"",,"""'" I 10 100 1000 10000 Numbers of iterations and of coding coefficients Figure 2. Performance of the two algorithms applied to a landscape. 4. Conclusions Several auxiliary techniques have been developed to speed up the matching pursuit iteration [1, 12, 3], but most of these techniques work by speeding up the computation of inner products between the dictionary elements and the residual image (step II), i.e. reducing the computational cost of one iteration; therefore, they could be combined with the inhibition method to achieve the same result. Only hierar­chical parallel matching pursuit [7] is similar to the inhibi­tion method in allowing the updating of several coefficients per iteration, although the method used to select these coef­ficients is different. The elements of the vector p can be interpreted as mea­sures of similarity between the residual image and the ele­ments of the dictionary. Recently, it has been shown that a more sparse encoding can be achieved with a different simi­larity measure [8]. This different measure can be combined with the inhibition method as well as with matching pursuit, since it consists of a modification of Eq.5. This paper has presented results on images, because this is an application which puts a heavy computational load on matching pursuit; however, the inhibition method can be ap­plied to one-dimensional signals as well. Atomic decompo­sition has proved useful in a number of pattern-recognition applications (e.g. [9, 14, 15, 16]). The inhibition method, by its faster convergence, allows the practical application of 214 atomic decomposition to new areas. Acknowledgements: The authors thank Dr. J. Roerdink for helpful comments on an earlier version of this article. Most of the images were obtained from the PIeS image archive (http://pics.psych.stir.ac.uk). References [1] F. Bergeaud and S. Mallat. Matching pursuit: adaptive repre­sentations of images and sounds. Compo Appl. Math., 15:97-109,1996. [2] M. Carandini and D. Heeger. Summation and division by neurons in primate visual cortex. Science, 264:1333-1336, 1994. [3] T.-H. Chao, B. Lau, and W. Miceli. Optical implementa­tion of a matching pursuit for image representation. Optical Enf:i., 33(7):2303-2309,1994. [4] S. Chen and D. Donoho. Examples of basis pursuit. Pl'oc. SPIE, 2569(2):564-574, 1995. [5] J. Daugman. Uncertainty relations for resolution in space, spatial frequency, and orientation optimized by two­dimensional visual cortical filters. J. Opt. Soc. Am. A, 2:1160-1169,1985. [6] G. Davis, S. Mallat, and M. Avellaneda. Greedy adap­tive approximation. Journal of Constructive Approximation, 13:57-98,1997. [7] H. Feichtinger, A. TUrk, and T. Strohmer. Hierarchical par­allel matching pursuit. Proc. SPIE, 2302:222-232, 1994. [8] S. Jaggi, W. Karl, S. Mallat, and A. Willsky. High resolution pursuit for feature extraction. Applied and Computational Harmonic Analysis, 1998. [9] S. Jaggi, W. Karl, s. Mallat, and A. Willsky. Silhouette recognition using high-resolution pursuit. Pattern Recog­nition. 32(5):753-771,1999. [10] S. MalIat and Z. Zhang. Matching pursuit with time-frequency dictionaries. IEEE Trans. Signal Proc., 41(12):3397-3415,1993. [11] M. Nafie, M. Ali, and A. Tewfik. Optimal subset selection for adaptive signal representation. ProC. IEEE Int. Con/. Acoust. Speech Signal Proc., pages 2511-2514,1996. Ll2] R. Neff and A. Zakhor. Very low bit-rate video coding based on matching pursuits. IEEE Trans. Circuits and Systems for Video Technology, 7(1):158-171, 1997. [13] Y. Pati, R. Rezaiifar, and P. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with ap­plications to wavelet decomposition. Proc. 27th Asilomar Conference on Signals, Systems and Computers, pages 40-44,1993. [14] Y. M. Ro, R. Neff, and A. Zakhor. Matching pursuited data acquisition in MRI. Proceedings on SPIE, Medical Imaging, 3202(1):530-540,1997. [15] A. Shmilovici and O. Maimon. Application of adaptive matching pursuit to adaptive control of nonlinear dynamic systems. lEE Proceedings - Control T heory and Applica­tion, 145(6):575-582, November 1998. [16] H. Yoshida. Matching pursuit with optimally weighted wavelet packets for extraction of microcalcifications in manunograms. Applied Signal Processing, 5(3):127-141, 1999. 

Fast atomic decomposition by the inhibition method

https://pure.rug.nl/ws/files/14402266/2000ProcICPRPece.pdf

Fast atomic decomposition by the inhibition method

Abstract

Similar works

Full text

Available Versions

ARTS repository - University of Groningen