Inclusion of the latest research work to a real life applications and achieving adaptive video signal postprocessing is a complex and timely task. The proposed framework breaks the filtering process into several distinct phases and enables a researcher to move at a faster pace by narrowing the scope of the research and enabling seamless integration of this work into inherently adaptive filtering process. Some of the benefits of this modular and adaptive framework have been presented on the example of the MPEG-4 postprocessing improvements.Uključivanje najnovijih istraživačkih dostignuća u svakodnevne primjene i postizanje adaptivnog postprocesiranja videosignala zahtjevan je zadatak za koji je potrebno mnogo vremena. Predloženi sustav dijeli postupak filtriranja u nekoliko jasno odvojenih faza te na taj način sužava opseg svakog pojedinog istraživanja. Time je omogućena jednostavna integracija novih dostignuća u inherentno adaptivan proces filtriranja, što omogućava osjetno brži napredak u istraživanju. Neke prednosti ovog modularnog i adaptivnog sustava prikazane su na primjeru poboljšanja procesa postprocesiranja videosignala kodiranog prema MPEG-4 normi

Mladen Tomić

Nino Stojković

Hrčak - Portal of scientific journals of Croatia

ISSN 0005—1144ATKAAF 48(1—2), 21—25 (2007)Mladen Tomić, Nino StojkovićFramework for Adaptive Postprocessing of Video SignalsUDK 621.397:004.932IFAC 5.8.4Preliminary communicationInclusion of the latest research work to a real life applications and achieving adaptive video signal post-processing is a complex and timely task. The proposed framework breaks the filtering process into severaldistinct phases and enables a researcher to move at a faster pace by narrowing the scope of the research andenabling seamless integration of this work into inherently adaptive filtering process. Some of the benefits ofthis modular and adaptive framework have been presented on the example of the MPEG-4 postprocessingimprovements.Key words: signal processing, video compression, MPEG-4, adaptive filtering, filtering framework, postpro-cessing1 INTRODUCTIONAs video signals became common in the mostaspects of human activity, a lot of effort has beenput in ensuring as high reproduced video qualityas possible. This was vitally important in the caseof lossy coded video signals, but it is also impor-tant for the other types of video signals. For thatpurpose, many different filtering techniques andfilters were developed. Averbuch et al. in [1] pro-posed usage of weighted sums of symmetricallyaligned pixels for deblocking of block-transformcompressed images. They define the basic weights,to maximize efficiency at lower bit-rates but alsoadapt them at higher bit-rates to avoid edge ghost-ing effect. The masking effect of the Human VisualSystem (HVS) is taken into consideration by Chenet al. in [2]. They use large processing window ina low-activity areas to ensure proper image smoot-hing where is no background masking ability, butin a high-activity areas the window is reduced andthe greatest weight is given to central pixel, there-fore maintaining image details. Alter et al. in [3]proposed technique based on a weighted total vari-ation minimization which allows them to efficient-ly deblock the image without too extensive over-smoothing. Different approach is taken by Liew etal. in [4] who proposed the method based on pro-jection onto convex sets (POSC). Instead of usingstrict 8 × 8 blocks of pixels, their algorithm enfor-ces smoothness in a more general region that mightspan several blocks. The thin-plate spline surfacemodeling is used for that purpose. However, there are two basic problems that stillshow up with current research approach — it is verydifficult to derive a method that will perform ex-tremely well at the whole range of different ima-ges contained in a video signal, and also, the pro-cess of integration of research achievements in reallife applications is too complex. The proposed fra-mework is trying to address both of the issues bybreaking the process of filtering into several dis-tinct phases/processes. This way we facilitate adap-tation to a specific video signal and enable the en-hancements in some part of filtering process notto affect the other parts. In effect, research areasare narrowed to more specific topics where betterresults could be achieved, and integration with theexisting applications is greatly simplified and mademore transparent. This is all possible because ofthe clear and distinct rules for each process func-tion and the way in which parameters are passedand used during different phases of the video sig-nal processing. The example of MPEG-4 video sig-nal postprocessing using the proposed frameworkwill be presented.2 FRAMEWORK FOR ADAPTIVE POSTPROCESSINGPostprocessing of the video signal is performedon every frame by any number of filters. Filterpool may contain more than one filter for the samepurpose. For instance, there may be a single filterfor the contrast enhancement and three filters forblocking artifacts reduction. To make postprocess-AUTOMATIKA 48(2007) 1—2, 21—25 21tomic:clanak master.qxd  27.6.2007  12:12  Page 21ing adaptive and independent of the video signal,the process of filtering is broken into six phases:Postprocessing Parameters Extraction (PPE), Sig-nal Quality Estimation (SQE), Filtering Level Esti-mation (FLE), Filter Selection (FS), Filtering (F)and, finally, Filter Efficiency Verification (FEV).Each phase is implemented by means of respec-tive Algorithms (PPEA, SQEA, FLEA, FSA, FAand FEVA, respectively) and in as many steps asneeded, in different phases of the video signal pro-cessing. Informations between these steps are inter-changed. The whole framework is shown in Fi-gure 1. During the PPE phase, video signal is analyzedand parameters for every filter are extracted. Theseparameters could mark the portions of the frameto be filtered, filtering mode to be performed orany other filter specific information. This processis influenced by filtering level, as determined bythe FLE.SQE should provide at least rough estimation ofthe current frame quality, in aspects that are mean-ingful for each of the filters.FLE is the core phase for the adaptive postpro-cessing. By using informations from the SQE andother appropriate parameters, FLE derives optimallevel of filtering for each filter. Apart from theSQE information, FLE takes as its input the meas-ure of available system resources, output videobuffer level, user preferences and previous levelfor each of the filters. Other inputs, such as theFEV output, could be added if necessary. The de-rived filtering level should provide the highest pos-22 AUTOMATIKA 48(2007) 1—2, 21—25Framework for Adaptive Postprocessing of Video Signals M. Tomić, N. StojkovićFig. 1 Proposed framework for adaptive postprocessingtomic:clanak master.qxd  27.6.2007  12:12  Page 22sible reproduced video quality that could beachieved with the available system resources andthe current filter pool. In case of the lowered fil-tering levels, because of the insufficient system re-sources, FLE considers the priority of filters, asdetermined by the Human Visual System (HVS)characteristics or other factors included in the FLEalgorithms.With the FLE and SQE information available,the FS process is able to select the most appropri-ate filter from the filter pool, for each of the filtertypes. After the filtering, the FEV may be imple-mented for each filter type, which will verify thatthe filter efficiency was acceptable. Informationsfrom the FEV may be used to request from theFLE increase or decrease of the filtering level forthe specific filters, but also for the analysis of theentire process.3 IMPLEMENTATION OF MPEG-4 POSTPROCESSING FILTERSMPEG-4 uses deblocking and deringing filtersto reduce the effect of blocking artifacts and mos-quito noise, respectively. The filtering process iscomputationally very intensive and to achieve thespeed gains with reasonable quality loss, MPEG-4introduced blocking and ringing semaphores [5],which lead to the application of strong or weakdeblocking or deringing filter in the case of therespective semaphore being 1 or 0. The efficiencyof this approach is widely limited by the video sig-nals bit-rate. By modifying this filtering processand implementing it through the proposed frame-work, the adaptive filtering will be achieved thatwill provide the highest possible reproduced videoquality, with available system resources, at thebroad range of bit-rates and with significant speedimprovements at higher bit-rates.PPE is implemented in two steps for both thedeblocking and the deringing filter. One step isused to extract the semaphores of the intra frame,while the other is used to extract the semaphoresof the inter frame. MPEG-4 defines that the hori-zontal blocking semaphore of intra-frame shouldbe set to 0 if there is at least one non-zero coeffi-cient in the top row of 8 × 8 matrix of the inversequantized DCT coefficients. The same way, verti-cal blocking semaphore of the intra-frame shouldbe set to 0 if there is at least one non-zero coeffi-cient in the far left column of the coefficients ma-trix. Blocking semaphores for inter-frames are ex-tracted in a similar way by using not only theresidual signal but also the blocking semaphoresof the reference block (as indicated by the motionvector). These rules are altered in a way that thenumber of coefficients needed to set the blockingsemaphore to 0 becomes variable, and that vari-able will be called DeblockingLevel, with therange of valid values [0, 8]. For the value of 0, de-blocking filter will be completely disabled, whilefor the value of 1 the blocking semaphores will beset in the same way as in the MPEG-4 standard.The remaining values are used to fine-tune the ap-propriate level of filtering, with the value of 8meaning that all block boundaries in the frame willbe processed.The ringing semaphore should be set to 0 if the-re is at least one non-zero coefficient in the 8 × 8DCT coefficients matrix in the positions other thenthe DC component or the first vertical, and thefirst horizontal AC component. The same way as inthe case of the blocking semaphore this number isset to be variable and this variable will be calledDeringingLevel, with a range of valid values[0, 62]. The value of 0 means that all blocks willbe processed, while the value of 1 results in theringing semaphores being set the same way as de-fined in the MPEG-4 standard. Again, the remain-ing values are used to fine-tune the level of pro-cessing, with the value of 62 being used to com-pletely disable deringing filter. Both the Debloc-kingLevel and the DeringingLevel are derived bythe FLE and input to the PPE.SQE for this purpose is very simple and it pure-ly extracts the quantization parameter (QP) of thecurrent frame, but it still provides a solid estima-tion of the level of blocking artifacts and mosqui-to noise that will arise. FLE takes three inputs:SQE quality estimation (QP of the current frame),available system resources measure and the outputvideo buffer level. Considering latter two inputs,FLE will select appropriate values for Debloc-kingLevel and DeringingLevel to achieve the high-est possible quality. A number of tests, using subjective quality as-sessments, were carried out to determine the ap-propriate levels for these variables and the resultsobtained were subject to objective assessmentthrough the FEV to further confirm its validity.Since there was only one filter for each filter type,the FS needed not to be implemented. FEV wasused to provide objective assessment of the filter-ing process. It was implemented using the recent-ly proposed Structural Similarity (SSIM) index [6,7] which proved to provide a much better correla-tion between the calculated and the perceived qual-ity than the usual PSNR measure.AUTOMATIKA 48(2007) 1—2, 21—25 23M. Tomić, N. Stojković Framework for Adaptive Postprocessing of Video Signalstomic:clanak master.qxd  27.6.2007  12:12  Page 23Figure 2 shows SSIM values for the CIF formatForeman sequence for the standard implementa-tion and the framework implementation, relativeto the full filtering of every block and block boun-darywhile Figure 3 shows processing times for differ-ent CIF format sequences by using the frameworkimplementation, relative to the standard implemen-tationBecause of the adaptive nature of the frameworkimplementation, the highest quality is retained at awide range of bit-rates, and still in the worst caseprocessing is just over 10 % slower than the stan-dard implementation, but as the bit-rates rise, insome situations, it becomes more than 50 % faster.More benefits occur in the case of very limitedsystem resources. In the most current applicationsthe lack of available resources is solved by com-pletely disabling certain filters, which inevitablycauses the loss of quality. Framework implementa-tion introduced adaptive fine-grained tuning of fil-tering level and made it possible to use virtuallyall of the available resources to achieve as higherquality as possible in the case of either temporaryor permanent lack of system resources.4 CONCLUSIONThe proposed powerful and modular frameworkopens many research possibilities for all of thebasic building blocks — PPE, SQE, FLE, FS andFEV. By strictly separating these research areas itbecomes possible to form a pool of postprocessingfilters, each of them being superior in a certainarea of application and being selected by the FSdecision and its application level being determinedby the FLE. This way, the framework may be ap-plied to a video signal of any given quality, to beenhanced in a number of ways. Example present-ed showed how seamless integration of researchwork to a real life application can be and what im-provements this framework can introduce. Beinginherently adaptive, it crosses the boundary of be-ing applicable only to a specific video signals andthe innovative approach that was taken makes itpossible to use it for a wide range of different vi-deo signals and applications, which promotes it toa valuable tool for the researchers and the deve-lopers in the extensive area of video signal pro-cessing and reproduction.ACKNOWLEDGEMENTThis work is supported by Croatian Ministry ofScience and Technology under grant 0069015.REFERENCES[1] A. Z. Averbuch, A. Schlar, D. L. Donoho, Deblockingof Block-Transform Compressed Images UsingWeighted Sums of Symmetrically Aligned Pixels.IEEE Transactions on Image Processing, vol. 14, no.2, pp. 200—212, Feb. 2005.[2] Tao Chen, Hong R. Wu, Bin Qiu, Adaptive Postfilte-ring of Transform Coefficients for the Reductionof Blocking Artifacts. IEEE Transactions on CircuitsSystem Video Technique, vol. 11, no. 5, pp. 594—602,2001.24 AUTOMATIKA 48(2007) 1—2, 21—25Framework for Adaptive Postprocessing of Video Signals M. Tomić, N. StojkovićFig. 2 Comparison of SSIM values, relative to the full filter-ing, for the CIF format Foreman sequenceFig. 3 Comparison of framework implementation's processingtimes, relative to the standard approachtomic:clanak master.qxd  27.6.2007  12:12  Page 24[3] F. Alter, S. Durand, J. Froment, Deblocking DCT--Based Compressed Images with Weighted TotalVariation. In Proceedings of International Conferenceon Acoustics, Speach and Signal Processing ICASSP2004, Montreal, Canada, vol. 3, pp. 221—224, 2004.[4] A. W. C. Liew, H. Yan, N. F. Law, POCS-BasedBlocking Artifacts Suppression Using a Smooth-ness Constraint Set with Explicit Region Modeling.IEEE Transactions on Circuits and Systems for VideoTechnology, vol. 15, no.6, June 2005, pp. 795—800.[5] ..., Information Technology — Coding of Audio--Visual objects — Part 2: Visual, International Or-ganisation for Standardisation, ISO/IEC 14496-2:1999/Amd. 1:2000(E), July 2000.[6] Z. Wang, L. Lu, A. C. Bovik, Video Quality Assess-ment Based on Structural Distortion Measure-ment, Signal Processing: Image Communication.special issue on Objective video quality metrics, vol.19, no. 2, pp. 121—132, February 2004.[7] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli,Image Quality Assessment: From Error Visibilityto Structural Similarity. IEEE Transactions on Ima-ge Processing, vol. 13, no. 4, pp. 600—612, Apr. 2004.AUTOMATIKA 48(2007) 1—2, 21—25 25M. Tomić, N. Stojković Framework for Adaptive Postprocessing of Video SignalsSustav za adaptivno postprocesiranje videosignala. Uključivanje najnovijih istraživačkih dostignuća usvakodnevne primjene i postizanje adaptivnog postprocesiranja videosignala zahtjevan je zadatak za koji jepotrebno mnogo vremena. Predloženi sustav dijeli postupak filtriranja u nekoliko jasno odvojenih faza te nataj način sužava opseg svakog pojedinog istraživanja. Time je omogućena jednostavna integracija novihdostignuća u inherentno adaptivan proces filtriranja, što omogućava osjetno brži napredak u istraživanju.Neke prednosti ovog modularnog i adaptivnog sustava prikazane su na primjeru poboljšanja procesa postpro-cesiranja videosignala kodiranog prema MPEG-4 normi.Klju~ne rije~i: obrada signala, kompresija videa, MPEG-4, adaptivno filtriranje, filtarski sustav, postproce-siranjeAUTHORS’ ADDRESSESmr. sc. Mladen TomićUniversity of Rijeka, Faculty of EngineeringVukovarska 58, 51000 RijekaHR — CroatiaNino Stojković, associate professorUniversity of Rijeka, Faculty of EngineeringVukovarska 5851000 RijekaHR — CroatiaReceived: 2007-02-15tomic:clanak master.qxd  27.6.2007  12:12  Page 25

Sustav za adaptivno postprocesiranje videosignala

Abstract

Similar works

Full text

Available Versions

Hrčak - Portal of scientific journals of Croatia