Chan, YL

Siu, WC

PolyU Institutional Repository

EFFICIENT INTERFRAME TRANSFORM CODING USING TEMPORAL CONTEXT Yui-Lam CHAN and Wan-Chi SIU Department of Electronic Engineering The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong Tel: (852)27666229 Fax: (852)23628439 email: enwcsiu@hkpucc.polyu.edu.hk ABSTRACT Three dimensional (3D) transform coding can reduce the interframe redundancy among a number of consecutive frames, while the motion compensation technique can only reduce the redundancy of at  most two frames. The former is very efficient when the correlation between interframe pix- els is high. However, the performance will be degraded for complex scenes with a large amount of motions. This paper presents a three dimensional discrete cosine transform (3D- DCT) coding with variable temporal lengths, which is based on the local temporal activity. Two scene change detectors are used to detect the local temporal activity. Our idea is to let the motion activity in each block (with variable tempo- ral length) be very low, while the efficiency of the 3D-DCT coding is increased. Through intensive computer simula- tions, the performance of the proposed 3D-DCT coding has been found to have substantial improvement over the con- ventional fixed length 3D-DCT coding. Furthermore, it is significant to point out that the performance of our pro- posed algorithm is better than that of the MPEG coding. 1. INTRODUCTION Three dimensional (3D) transform coding has the advan- tage of reducing the interframe redundancy for more than one frame. Besides, it is prevented from an infinite prop- agation of transmission errors. The price to be paid is a longer delay and a large memory requirements. Thus, it is an alternative approach to hybrid coding concepts used in today's standards. Adaptive three dimensional discrete cosine transform(3D-DCT) coding[l-4] has been reported to be comparable to hybrid coding (motion compensation prediction along the temporal axis[5-6] and DCT frequency coding in the spatial domain). Furthermore, it has an asym- metric property with the decoding being much faster than the encoding and the computation complexity is even lower than that required for the MPEG like coder [7 ] .  The 3D- DCT coding is very efficient when the amount of motions is small. In other words, the energy distribution in trans- form domain is more compact, therefore it is easier to code. The performance of the 3D-DCT coding will be affected in a complex scene with a large amount of motions. In this paper, we use a variable temporal length 3D- DCT instead of a fixed length 3D-DCT. The temporal length of the 3D-DCT varies with local temporal activities. Thus, Acknowledgment : This work was supported by The Croucher Foundation PolyU340/055. 0-7803-3073-0/96/$5 .OO O1996 IEEE 786 Figure 1: (a) Fixed length 3D-block (b) Variable temporal 3D-block (c) Pixel intensity which varies with time the motion activity in each block is still very low, while the coding efficiency becomes high. Our coding tests us- ing computer simulation show that this technique is indeed very efficient. It has great improvement as compared with the fixed length 3D-DCT. Furthermore, the performance of our proposed algorithm is better than that of the MPEG coding. 2. THE PROPOSED VARIABLE TEMPORAL LENGTH 3D-DCT CODING Theoretically, the most optimal transformation of the Markov-I type signals is the Karhunen-Lokve transform (KLT) [8]. The KLT can completely decorrelate the sig- nal in the transform domain, thus a high compression ratio can be achieved. It is well-known that the implementation of the KLT is difficult, while the DCT-I1 has been found asymptotically equivalent to the KLT for Markov-I signals as the correlation ( p )  between the pixels tends to 1 [8]. In conventional 3D-DCT coding, if a 3D-block has low frame- to-frame motion (the interframe pixels correlation is close to I), then only the coefficients having low temporal fre- quency need to be transmitted. The coding is very efficient when the amount of motions in the block is small. If the L x M x N pixel block is fixed, the coding performance will be degraded for complex scenes with a large amount of mo- tions. Figure l (a )  illustrates the problem of a fixed length 3D-DCT coding. The temporal motion activity is shown Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 30,2010 at 07:57:13 UTC from IEEE Xplore.  Restrictions apply. Figure 2: Block diagram of our proposed variable temporal length 3D-DCT coding system in figure I(c). In the figure, there is a scene change at the time tl and t z  in the image sequence. So, inside the three dimensional blocks B and C, there are high motion activi- ties and low interframe pixels correlations. This causes the high frequency coefficients in these transform blocks having significant values. Furthermore the distortion introduced by the coding process will probably spread over the whole 3D-block and be visible as they last for a long time on the decoded image sequence. In this case, the coding efficiency will be significantly decreased. In this paper, we propose a 3D-DCT coding with an adaptive adjustment of the length of the 3D-block in the time direction as shown in figure l(b).  It depends on the local activity in the image sequence. In other words, the temporal length, L ,  is varied instead of a fixed one. These blocks will remain in low motion activity and high interframe pixels correlation. Thus, this variable temporal length method could take the advantage of the 3D-DCT coding and achieve high coding efficiency. T- Widw. W (a) Time window 1001WOM10000L.. WO " Q F B  11 t2 b (b) Bloclr sequence with planes of s e n e  change and its mespoldlng bit sequence tl.l t,  tl.l 5 3 lo-' '0 (c) Vansblc temporal Ltnglh 3D-block Figure 3: Details of variable temporal length 3D-DCT cod- ing variable temporal length 3D-DCT coding system. Each im- age sequence is divided into a number of time windows, W ,  which is a fixed number of image frames from the original image sequence as shown in figure 3(a). In order to  de- termine the an appropriate cutting plane, a scene change detector has to be firstly used to detect the scene change before the interframe transformation can be done for the block. Two approaches are used in this paper. The first one is frame difference method. The frame difference ( F D k )  of the block in successive frames k + 1 and k is defined as : M-1 N-1 x=o y=o where I k ( z ,  y) is the pixel value in the image sequence at time k and spatial location (z, y). A block is called scene change if the value of F D k  in the block is greater than or equal to TO, where TO is a pre- defined threshold. The scene change detection using frame difference is very simple. However, its accuracy is affected in the case of a number of consecutive frames having frame difference just a little bit smaller than TO. Another design of scene change detector is motivated by Olstad [9]. In this paper, we refer it to  as adaptive tem- poral block filter. Assume that the block size in the spa- tial domain be M x N pixels. Let F = ( I ~ ( x , y ) , I ~ ( z , y ) ,  . . . , Iw-1 (x, y)) be any consecutive pixel intensity at loca- tion (x, y) within the spatial block in a time window of W samples as shown in figure 3(a). Let also ( Y P  represent a set of P planes of scene change within an image sequence with time window, W ,  and with the spatial dimensions of M x N .  The set (YP can then be written as a set of P indices, t ,  : 0 < t o  < tl < . . . < tp-1 < t p  = W as shown in fig- ure 3(b). The scene change detector generates the following subsequences of the time window: Fo = (Io(x,~),...,~t~-i(x,y)) FP-1 = ( L p 4 ( z ,  Y), . . . , Itp-l-l(z,  Y)) FP = (Itp_l(~,Y)h...,~w-i(~,Y)) In general, we can write F3 = ( I t ,  -, ( . , Y ) , . . . , I t t , - l ( " , Y ) )  In order to detect the scene change accurately, an error function ( E )  on the set ( ~ p  can be defined. Figure 2 illustrates the block diagram of our proposed j = o  x=o y=o 787 Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 30,2010 at 07:57:13 UTC from IEEE Xplore.  Restrictions apply. where e[(Io,I1, .  . . , I k - - l ) ]  = ~ ~ = 1 ( 1 1  - T ) 2 ,  and 7 de- notes the mean of lo, li,. . . , I k - i  Hence a scene change detector can then be developed for the minimization of the error function as shown above. The global minimum for the error function and the associated localization of the scene change planes can be efficiently computed with dynamic programming [9]. The number of scene change planes, P ,  in each 3D-block is chosen such that the error function E(ap)  < EO, where EO is a pre- defined threshold. In other words, the number of scene change planes, P ,  in each 3D-block is increased when there is a large amount of motions of the 3D-block in the image sequence. After the cutting planes are defined with the scene change detector in each block of sequence as shown in fig- ure 3(b), a variable temporal length 3D-DCT is performed on each 3D-block. Suppose that the scene change detector produces a scene of P planes. This implies that the block sequence is segmented into P + 1 3D-blocks with different temporal lengths. Each of the 3D-block is transformed by the 3D-DCT with different temporal lengths, L ,  by cascad- ing three 1D-DCTs, for L x M x N block size. The forward transforms are defined as follows: L-1 M-1 N-1 y C(%)C(V)C(W) k=O j = O  r = O  L x M x N  X(”,  II, w) = 26 ( 2 k  + 1)” ( 2 j  + 1)” l k  ( j ,  i) cos T ____ cos T-2L 2M (22 + l ) w  cos T------- 2 N  - fixed tempoml larplh 3D-DCT coding . . . . . . . . proposdd 3D-DCT coding udng frame QHfwmce - - -  proposed 3D-CCl coding udng odaplive blotk filter HPEG coding - - I I where C(n)  = % for Except for the DC coefficient of each 3D-block, which is coded using 8 bits, the quantization of coefficients is done according to their variances and the temporal lengths of the 3D-block. Since bits allocated to different temporal length 3D-blocks may be unequal, a different quantizer is required. For instance, the number of bits allocated to the 3D-block with the shorter temporal length would be less than that of the longer one. Let Bf be the total number of bits assigned to the 3D-blocks with temporal length I ,  and Bf can be evaluated using the following equation: = 0, otherwise C(n)  = 1. Bi = Bo $- IB, where Bo is the minimum number of bits for Bk and B, is the number of bits for each frame of the 3D-block. Then, the number of bits allocated to each individual transform coefficient is according to its variance [lo]. Af- ter the bit allocation has been found, the coefficients in each 3D-block could then be quantized. The quantized co- efficients in variable temporal length 3D-block are scanned in a zig-zag manner for the first frame, then followed by the successive frames. Then, the run-length coding and the huffman coding are employed to further reduce the bit rate. The decoding requires information about the scene changes from each 3D-block sequence such that the appro- priate temporal length inverse 3D-DCT can be performed. This side information can be recorded with a bit sequence 10010. .  . O l O  where the number of bits is equal to the num- ber of images in the current time window as shown in figure 3(b). A bit ‘1‘ indicates that this is the first frame in a new Figure 4: Plot of PSNR against bits per pixel for different algorithms c 4 Figure 5: Result for “salesman” sequence at  0.1 bits per pixel 3D-block. This bit sequence can be further compressed us- ing the Lempel-Ziv coding [ll] or the arithmetic coding [12]. For example, the temporal length is 3 for the first 3D-block in the above bit sequence. 3. RESULT A series of computer simulations have been conducted to evaluate the performance of the adaptive interframe trans- form coding. The 32 frames image sequence “salesman” with the size of 352 x 288 pixels has been used. The peak signal-to-noise ratio(PSNR) measure of image fidelity has been used to quantitatively evaluate the performance of the interframe coding systems. Figure 4 provides a comparison on the peak signal-to- noise ratios (PSNR) for different algorithms and for various bits per pixel. The PSNR performance of the proposed vari- able temporal length 3D-DCT coding using frame difference method as scene detector has a great improvement (about 788 Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 30,2010 at 07:57:13 UTC from IEEE Xplore.  Restrictions apply. 3.5dB) over the fixed temporal length (8 x 8 x 8 for each 3D-block) 3D-DCT coding at  low bit rate. But, there is an improvement of 1-2dB for cases using xbove 0.2 bits per pixel. The PSNR performance can be further improved by using an adaptive temporal block filter. As shown in figure 4, the proposed 3D-DCT coding using adaptive temporal block filter can achieve about 4dB and 3dB improvement as compared with the fixed temporal length 3D-DCT coding at low and high bit rates respectively. The results indicate that the proposed algorithms have a significant improve- ment as compared with the conventional 3D-DCT coding. To further evaluate our algorithm, we have compared it with a typical scheme according to the MPEG coding [13]. The comparisons presented in this paper are based on the MPEG coding with IPPPPPPPgroup of picture. The block size is 16 x 16 with the maximum displacement 8 for mo- tion estimation. The performance of the MPEG coding is better than that of the conventional fixed temporal length 3D-DCT coding for cases below 0.2 bit per pixel. But the PSNR performance of our proposed algorithm is better than that of MPEG coding, as depicted in figure 4. The MPEG coder lags behind by approximately 3-3.5dB at different bit rates. Figure 5 shows the performance of different algorithms at  0.1 bits per pixel against frame numbers. As expected, the values of the PSNR of MPEG coding for P-frames are worse than that of the I-frames. Thus, the PSNR perfor- mance of MPEG coding is very frustrated. The same is also true for the conventional 3D-DCT coding. But, the PSNR performance of our proposed algorithms is better, and the values of the PSNR of the reconstructed frames are very smooth. I t  implies that our proposed algorithms are less suffering from the “jerkiness” of the motion. Besides, The algorithms also have the advantage over the conventional 3D-DCT coding that they are free of annoying block dis- tortion. These results clearly verify the effectiveness of the proposed scheme. 4. CONCLUSIONS We propose a variable temporal length 3D-DCT coding which adaptively adjusts the temporal length of the 3D- block. The interframe correlation within each 3D-block is high such that the advantage of 3D-DCT can be fully uti- lized. The adjustment of the temporal length of each 3D- block depends on local activities in the image sequence. Two scene change detectors have been used to determine the local activity of the block sequence. A series of com- puter simulations show that the proposed variable temporal length 3D-DCT coding can achieve reconstructed image se- quences with good quality at  different bit rates. It has a significant improvement as compared with the fixed tem- poral length 3D-DCT coding using different scene change detectors. Besides, the PSNR performance of our proposed algorithm has 3-3.5dB improvement as compared with that of the MPEG coding. Results of our study indicate that significant gain over the standard hybrid coding techniques used nowadays is possible through the 3D-DCT coding with variable tem- poral length. However, many problems still remain to be resolved, including the high memory requirement and the design of perfect optimal scene change detector. Neverthe- less, three dimensional frequency coding is a fruitful direc- tion that should not be forgotten for studying techniques for video compression. 5. REFERENCES J.A.Roese and W.K.Pratt, “Interframe Cosine Trans- form Image Coding”, IEEE Transactions on Commu- nications, vol-25, Nov. 1977, pp.1329-1338. Olivier Chantelou, Christian Remus, “Adaptive Trans- form Coding of HDTV Picture”, International Work- shop on Signal Processing of HDTV, L‘Aguila, Italy, Elsevier Science Publishers, Feb-Mar.,1988, pp.231-8. Jong Won Kim, Shin Ho Lee and Sang Uk Lee, “On the adaptive three-dimensional transform coding tech- niques for moving images’’ Signal Processing: Image Communication, April, 1995, pp. 525-543. Yui-Lam Chan and Wan-Chi Siu, “A Mew Adaptive Interframe Transform Coding using Directional Clas- sification” Proceedings, IEEE Int. Conf. on Image Pro- cessing, No.2, pp.977-981, Nov. 1994, U.S.A. Yui-Lam Chan and Wan-Chi Siu, “An Adaptive Multiple-Candidate Hierarchical search for Block Matching Algorithm”, Electronics Letters, vol. 31, no.9, pp. 1637-1639, Sept. 1995, U.K. Yui-Lam Chan and Wan-Chi Siu, “A New Block Mo- tion Vector Estimation using Adaptive Pixel Decima- tion”, IEEE Transactions on Circuits and Systems for Video Technology, vo1.6, No.1, Feb.,1996 Yui-Lam Chan and Wan-Chi Siu, “Fast Interframe Transform Coding Based on Characteristic of Trans- form Coefficients and Frame Difference” Proceedings, IEEE Int. Sym. on Circuits and System, pp. 449-52, Vol. 1, April-May 1995, Seattle, U.S.A. K.R.Rao and R.Yip, Discrete Cosine Transform- Algorithms, Advantages and Applications. New York: Academic Press, 1990. B.Olstad, ‘(Adaptive Temporal Decimation for Video Compression Algorithms” Journal of Electronic Imag- ing, V01.2, Jan., 1993, pp.5-18. A.Segall, “Bit Allocation and Encoding for Vector Sources” IEEE Transactions on Information Theory, V01.22, No.2, Mar., 1976, pp.162-169. Jacob Ziv and Abraham Lempel, “A Universal Algo- rithm for Sequential Data Compression” IEEE Trans. on Inf. Theory, vo1.23, No.3, May, 1977, pp.337-43. Paul G. Howard and Jeffrey Scott Vitter “Arithmetic Coding for Data Compression” Proceedings of the IEEE, Vol. 82, No.6, June 1994, pp.857-865 D. J. LeGall, ”MPEG: A Video Compression Standard for Multimedia Applications”, Communications of the ACM, vol-34, April 1991, pp.47-58. 789 Authorized licensed use limited to: Hong Kong Polytechnic University. Downloaded on June 30,2010 at 07:57:13 UTC from IEEE Xplore.  Restrictions apply. 

Efficient interframe transform coding using temporal context

Abstract

Similar works

Full text

Available Versions

PolyU Institutional Repository