Chan, YH

Hong, SW

Siu, WC

IEEE Transactions on Circuits and Systems for Video Technology

PolyU Institutional Repository

4 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 8, NO. 1, FEBRUARY 1998Transactions LettersA Practical Postprocessing Technique forReal-Time Block-Based Coding SystemYuk-Hee Chan, Sung-Wai Hong, and Wan-Chi SiuAbstract—In this paper, a noniterative postprocessing methodis proposed to restore the images encoded with block-based com-pression standards such as Joint Photographic Experts Group(JPEG). This method classifies small local boundary regions ac-cording to their intensity distribution and then selects appropriatelinear predictors to estimate the corresponding boundary pixelsin the regions. This approach is easy to implement and we foundin our simulations that its restoration performance was veryrespectable compared with the reported postprocessing methods.Index Terms— Block-based coding, codec, deblocking, imagerestoration, JPEG, postprocessing technique, vector quantization.I. INTRODUCTIONTHE Joint Photographic Experts Group (JPEG) imagecompression standard [1] has been widely used as anindustrial standard for still image compression. In this stan-dard, source images are first divided into nonoverlapped blocksof size 8 8 each. Each block is then transformed withdiscrete cosine transform (DCT). Transform coefficients arequantized and encoded with an arithmetic or Huffman codingtechnique. Since it is basically a block-based process, little ofthe interblock correlation is exploited during the compression,and therefore it causes visible discontinuities among adjacentblocks. This blocking effect is generally considered to be themost noticeable artifact in the reconstructed image.Two common strategies have been used to reduce theblocking effect. One tries to retain some information of theinterblock correlation in the transmitted message. This canbe done by embedding interblock correlation in the transformcoefficients [2]–[4] or by directly sending extra information tothe receiver via additional channel bandwidth [5], [6]. Someof these methods can achieve a remarkable performance ineliminating the blocking effect. However, since these methodsgenerally have to modify the basic transform coding structureof the existing JPEG standard, it is difficult or even impossibleto integrate them with the current industrial standard.The other strategy is to restore or enhance the image qualityby using postprocessing techniques. Typical examples includeblock-boundary filtering [6] and convex projection (POCS)[7]–[11]. This strategy has appeal since it only requires theManuscript received November 3, 1995; revised May 10, 1997. This paperwas recommended by Associate Editor K.-H. Tzou. This work was supportedby The Hong Kong Polytechnic University Grant A/C 350.255.A3.420.The authors are with the Department of Electronic Engineering, The HongKong Polytechnic University, Hung Hom, Kowloon, Hong Kong.Publisher Item Identifier S 1051-8215(98)01517-1.decompressed image and is fully compatible with the existingJPEG standard.However, though most of them were proven to be veryeffective in eliminating the blocking effect, there are still somelimitations. For instance, projection methods are not suitablefor real-time application since they are basically iterativealgorithms which typically require exhaustive computationand take time for the estimate to converge to a desirablesolution [7]–[11]. On the other hand, some methods aim at theenhancement of the image quality instead of the restoration ofthe image, and therefore the distortion may even be enlargedafter the process [6], [8].In this paper, a noniterative postprocessing method is pro-posed to restore the JPEG-encoded images. This methodclassifies small local boundary regions according to their inten-sity distribution and then selects appropriate linear predictorsto estimate the corresponding boundary pixels in the regions.This approach is easy to implement and we found in oursimulations that its restoration performance is very respectablecompared with the reported postprocessing methods [6]–[11].II. ALGORITHMTo a certain extent, the elimination of the blocking effect canbe viewed as a process of image restoration. This restorationprocess involves the modification of the JPEG-encoded image.When there is no extra information about the original image,any modification is solely based on assumptions. In otherwords, postprocessing could cause degradation instead ofimprovement in the image quality. To minimize the possibilityof the occurrence of degradation, it is reasonable to modifyas few image pixels as possible during the postprocessingprocess. Since the blocking effect appears as the discontinuityamong adjacent blocks, it would be a good idea to smooththe discontinuity by modifying the intensity of the boundarypixels. However, as for the intrablock pixels, it is betterto leave them without modification in order to minimizethe disturbance to the image pixels, especially when thecompression rate is not too high.It is well known that the blocking effect is a consequenceof ignoring the interblock correlation during the compression.Therefore, one of the best ways to restore the original imageis to make use of the residue interblock correlation in theJPEG-encoded image to recover the block boundary pixels. Inthe proposed method, we investigate the connection betweenthe residue interblock correlation in the encoded image and1051–8215/98$10.00  1998 IEEEAuthorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 23:20 from IEEE Xplore.  Restrictions apply. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 8, NO. 1, FEBRUARY 1998 5the boundary pixels in the region concerned and then makeuse of the statistics obtained to predict the original pixelintensity. Since there is typically a considerable amount ofresidue interblock correlation remaining in the JPEG-encodedimages, a good prediction result can be obtained based onthe statistics even though only a simple linear prediction isperformed.Consider an eight-pixel line segment which is perpendicularto the block boundary as shown in Fig. 1. This segment iscomposed of two block boundary pixels and six neighboringpixels in line with them, which forms a pixel vector. Withoutlosing the generality, we assume that the pixel vector obtainedafter JPEG encoding is .Our objective is to predict the original boundary pixelsand as accurately as possible. This can be done with linearprediction by making use of the intensity correlation amongthe original boundary pixels and their neighboring pixels inthe JPEG-encoded image. Assume that the predicted valueis given by(1)where ’s are scalar prediction filter coefficients, ,and . The selection of the filter coefficientsshould be based on the least-squares minimization criterion[12]. In other words, one has to minimize the prediction errorin an average least-squares sense,where is the expectation operator. The minimization ofcan be achieved by satisfying the criteria for all, which results in the following formulation:..................(2)Prediction coefficients ’s can then be determined providedthat exists.The prediction filter for can also be developed in asimilar way. However, to avoid doubling the number of filtersrequired, pixel can be predicted by processing the reversedvector instead of withthe same prediction filter.To assure the robustness of the prediction, pixel vectors areclassified into different classes such that a dedicated predictionfilter can be designed based on the statistics of a particularclass of pixel vectors by making use of the above technique.As most of the interblock correlation can be reflected bythe smoothness of the intensity distribution across the blockboundary, pixel vectors should be classified according to theirintensity variation. One of the simplest but most effective waysto achieve this objective is to make use of the concept of meanremoval vector quantization.Fig. 1. Pixel vectors generated from images under JPEG coding.Consider the case that there is a codebook containingcodewords , where is thenumber of classes into which we are going to classify thepixel vectors. For any given mean-removed input pixel vector,classification is carried out by performing a VQ encodingprocedure. That means one has to select a codeword which isclosest to the given mean-removed pixel vector. The selectionis based on the minimum Euclidean distance criterion, wherethe Euclidean distance measure is defined asand is the th element of codeword . The indexof the chosen codeword is then equivalentto the class number with which the pixel vector associates.The codewords have to be carefully chosen such that theyare representatives of their corresponding classes of pixelvectors. As usual, the codebook has to be designed with a set oftraining vectors prior to its application. Typical JPEG-encodedimages can be used to generate training vectors. In particular,all vertical and horizontal mean-removed pixel vectors of theimages, except those containing block corner pixels, are usedas training vectors. The reversed vector of each training vectoris also used as a training vector to double the size of thetraining set. Then, the LBG algorithm [13] is used to generatethe optimal codebook with the set of training vectors. To assurethe effectiveness of the postprocessing method, the training setshould be large enough and able to represent the statistics ofthe images to be coded.The design of the codebook and the prediction filters couldbe computation-intensive, but their realization can be off-line and no second realization is required once it is doneunless the nature of input images is greatly changed. In real-time application, the proposed method consists of two stages.Fig. 2 shows the block diagram of the proposed postprocessingmethod. After a JPEG-encoded image is decoded, pixel vectorsof the decoded image are mean-removed and then classifiedwith a VQ encoder. Based on the output of the VQ encoder, anappropriate prediction filter is selected for each mean-removedpixel vector. This vector is then fed into the filter to get aprediction value. The boundary pixel predicted will then beassigned the rounded result of the prediction value if it is nota block corner pixel. Note the reversed vector of the processingpixel vector is also a pixel vector and it gives another boundaryAuthorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 23:20 from IEEE Xplore.  Restrictions apply. 6 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 8, NO. 1, FEBRUARY 1998TABLE IRESTORATION PERFORMANCE OF VARIOUS JPEG POSTPROCESSING METHODS IN TERMS OF SNR IMPROVEMENTS (UNIT: dB)Fig. 2. Block diagram of the postprocessing method.pixel when it goes through the same procedures mentionedabove. As for those block corner pixels, they are restored at theend of the process. Since there will be two prediction values(One from a horizontal pixel vector and one from a verticalpixel vector), the reconstructed pixel value is assigned to bethe rounded result of the average value. Note that the pixelvectors used for restoring block corner pixels are composedof the restored boundary pixels.The robustness of the prediction filter is guaranteed by theclassification of pixel vectors and the following correspondingaction. In fact, the classification process introduces a nonlinearnature into the system. Theoretically, the greater the numberof classes, the better performance it can achieve. Naturally, thecomplexity gets higher too, but the burden can be significantlyreduced by using the technique proposed in [14].III. SIMULATIONSComputer simulations have been carried out to evaluatethe restoration performance of the proposed method. Thesimulations are based on a set of 256 level grayscale digitalimages of size 256 256 pixels. The blocky versions of fivestandard images “Lenna,” “House,” “Peppers,” “Airplane,”and “Sailboat” were used to generate a training set to ob-tain codebooks of size , where and .These blocky versions were generated by coding the originalimages with the JPEG scheme at a compression rate around0.27 b/pixel. Training pixel vectors were then classified intoclasses with the corresponding codebook. Prediction filtersof various classes were then designed with training vectors oftheir corresponding classes by using the approach mentionedin the previous section.The performance is evaluated in terms of the improvementin signal-to-noise ratio which is defined as(SNR) dB (3)where , , and are the restored, the JPEG-encoded, andthe original images, respectively. Some other postprocessingmethods [6]–[8] for blocking effect elimination are also eval-uated for comparison. The termination criterion for iterativemethods is chosen to be ,where is the restored image obtained at the th iteration.Table I and Fig. 3 show the restoration performance of vari-ous methods. One can see the superior restoration performanceof the proposed method compared with other postprocessingmethods [6]–[8]. In general, methods using low-pass filtering[6], [8] may reduce the SNR since they are concerned withthe enhancement of the subjective quality by smoothing theblock boundary. The modified JPEG decoders proposed in [15]and [16] are efficient and practical, but their performance wasmuch more sensitive than that of the proposed method to thechoice of the training set. This can be seen from the fact that,with the same small number of training images, its restorationperformance fluctuated from training images to nontrainingimages, while the proposed method could provide much morestable performance in our simulation. As a final remark, wenote that a few iterative methods such as [11] can occasionallyachieve a slightly higher SNR improvement than the proposedmethod can. However, due to their iterative nature, they cannotbe used for real-time applications.Table I also shows the performance of the proposed methodwith various numbers of classes. It was found that, thoughmore classes resulted in better performance, the marginalimprovement decreased as the number of classes increased.From our simulation result, 8 to 16 classes would be ap-propriate since it made the design simpler while maintaininga reasonable restoration performance compared with otherreported methods [6]–[11].Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 23:20 from IEEE Xplore.  Restrictions apply. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 8, NO. 1, FEBRUARY 1998 7(a) (b) (c)(d) (e) (f)(g) (h)Fig. 3. Reconstructed images with various JPEG-compatible deblocking decoding methods: (a) original (b) JPEG (0.2 bpp), (c) CLS [7], (d) NID0 [16], (e)proposed approach with no classification, (f) proposed approach with eight classes, (g) filtering [6], and (h) POCS [7] (testing image: girl).IV. CONCLUSIONBased on linear prediction and vector quantization concepts,an effective postprocessing technique for transform coding isproposed in this paper. This technique is fully compatible withthe existing JPEG standard and enables real-time application.Computer simulations show that the proposed method couldachieve a better restoration performance in terms of SNR im-provement compared with some other existing postprocessingmethods [6]–[9]. Moreover, the image quality could also beimproved subjectively by eliminating most of the blockingeffect.In this paper, we have considered the application of the pro-posed method in JPEG coding. However, the same techniquecan be applied to block-based transform video coding suchas MPEG standard. Since video sequence typically containsa series of images with similar statistics, even few classescan achieve a respectable and consistent improvement inimage quality when the proposed method is exploited. Thisis especially true in videophone application.Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 23:20 from IEEE Xplore.  Restrictions apply. 8 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 8, NO. 1, FEBRUARY 1998REFERENCES[1] W. B. Pennebaker and J. L. Mitchell, JPEG Still Image Data Compres-sion Standard. New York: Van Nostrand Reinhold, 1993.[2] H. S. Malvar and D. H. Staelin, “The LOT: Transform coding withoutblocking effects,” IEEE Trans. Acoust., Speech, Signal Processing, vol.37, pp. 553–559, Apr. 1989.[3] G. Aharoni, A. Averbuch, R. Coifman, and M. Israeli, “Local cosinetransform—A method for the reduction of the blocking effect in JPEG,”J. Mathematical Imaging and Vision, vol. 3, pp. 7–38, 1993.[4] Y. H. Chan and W. C. Siu, “An approach to subband DCT imagecoding,” J. Visual Commun., Image Represent., vol. 5, no. 1, pp. 95–106,Mar. 1994.[5] Y. Q. Zhang, R. L. Pickholtz, and M. H. Loew, “A new approach toreduce the blocking effect of transform coding,” IEEE Trans. Commun.,vol. 41, pp. 299–302, Feb. 1993.[6] H. C. Reeve and J. S. Lim, “Reduction of blocking effect in imagecoding,” Opt. Eng., vol. 23, no. 1, pp. 34–37, Jan./Feb. 1984.[7] Y. Yang, N. P. Galatsanos, and A. K. Katsaggelos, “Regularized recon-struction to reduce blocking artifacts of block discrete cosine transformcompressed images,” IEEE Trans. Circuits Syst. Video Technol., vol. 3,pp. 421–432, Dec. 1993.[8] A. Zakhor, “Iterative procedures for reduction of blocking effects intransform image coding,” IEEE Trans. Circuits Syst. Video Technol.,vol. 2, pp. 91–95, Mar. 1992.[9] T. P. O’Rourke and R. L. Stevenson, “Improved image decompressionfor reduced transform coding artifacts,” SPIE, 2182, Image & VideoProcessing II, pp. 90–101, 1994.[10] S. Minami and A. Zakhor, “An optimization approach for removingblocking effects in transform coding,” IEEE Trans. Circuits Syst. VideoTechnol., vol. 5, pp. 74–81, Apr. 1995.[11] S. W. Hong, Y. H. Chan, and W. C. Siu, “An adaptive constrained leastsquare approach for removing blocking effect,” in Proc. IEEE ISCAS’95,Apr./May 1995, vol. 2, pp. 773–776.[12] S. J. Orfanidis, Optimum Signal Processing, 2nd ed. New York:McGraw-Hill, 1988.[13] Y. Linde, A. Buzo, and R. M. Gray, “An algorithm for vector quantizerdesign,” IEEE Trans. Commun., vol. 28, pp. 84–95, 1980.[14] Y. H. Chan and W. C. Siu, “In search of the optimal searching sequencefor VQ encoding,” IEEE Trans. Commun., vol. 43, pp. 2891–2893, Dec.1995.[15] S. W. Wu and A. Gersho, “Improved decoder for transform coding withapplication to the JPEG baseline system,” IEEE Trans. Commun., vol.40, pp. 251–254, Feb. 1992.[16] , “Nonlinear interpolative decoding of standard transform codedimages and video,” in SPIE Conf. Image Processing Algorithms andTechniques III, San Jose, CA, Feb. 1992, pp. 88–99.Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on July 27, 2009 at 23:20 from IEEE Xplore.  Restrictions apply. 

A practical postprocessing technique for real-time block-based coding system

Abstract

Similar works

Full text

Available Versions

PolyU Institutional Repository