Crnojević, Vladimir

Panic, Marko

Sladojević, Srdjan

Ćulibrk, Dubravko

Ghent University Academic Bibliography

Local Binary Patterns and Neural Networks forNo-Reference Image and Video QualityAssessmentMarko Panic´, Dubravko C´ulibrk, Srdjan Sladojevic´, and Vladimir Crnojevic´University of Novi Sad, Faculty of Technical Sciences,Trg Dositeja Obradovica 6, 21000 Novi Sad, Serbia{mpanic,dculibrk,sladojevic,crnojevic}@uns.ac.rshttp://www.ftn.uns.ac.rsAbstract. In the modern world where multimedia is predicted to form86% of traffic transmitted over the telecommunication networks in thenear future and providers are looking to shift towards Quality of Experi-ence, rather than Quality of Service in multimedia delivery, no-referenceimage quality assessment and the related video quality assessment re-maining open research problem, with significant market potential. In thispaper we describe a study focused on evaluating the applicability of LocalBinary Patterns (LBP) as features and neural networks as estimators forimage quality assessment. We focus on blockiness artifacts, as a promi-nent effect in all block-based coding approaches and the dominant artifactin occurring in videos coded with state-of-the-art video codecs (MPEG-4,H.264, HVEC). In this initial study we show how an LBP-inspired ap-proach, tuned to this particular effect can be efficiently used to predict theMOS of JPEG coded images. The proposed approach is evaluated on awell-known public database and against widely-used features. The resultspresented in the paper show that the approach achieves superior perfor-mance, which forms a sound basis for future research aimed at videoquality assessment and precise blocking artifact detection with sub-frameprecision.Keywords: Local Binary Patterns, Neural Networks, Multimedia Qual-ity, Image Quality, Assessment1 IntroductionVideo traffic is predicted to form approximately 86 percent of global consumertraffic by 2016 [7]. Every second, 1.2 million minutes of video content will crossthe network in 2016. Video traffic is becoming the dominant application over3G/4G mobile systems. Fuelled by proliferation of smartphones, netbooks andtablets, mobile data traffic is estimated to grow 18 times from 2011-2016, withthe surge caused mainly by mobile video services. Within this landscape animportant question for service providers is the quality assurance of the serviceprovided to the consumers of multimedia content.2 LBPs and NNs for Image and Video QAVideo Quality Assessment (VQA) algorithms attempt to automatically assessperceptual degradations introduced by signal processing and transmission oper-ations performed on video sequences. Unfortunately, despite recent advances invideo processing and communication technology, their performance leaves some-thing to be desired and there is considerable room for improvement [12]. This isespecially true for no-reference methods, which have no notion of the original,non-compressed multimedia content.To estimate video quality one usually first needs to be able to derive metrics ofimpairments at the level of single frames. Once the quality is estimated at framelevel, diverse methods exist to aggregate this information to create a qualityscore for the whole video. Overall degradation of the quality of frames(images)is a compound effect of different coding artifacts. Three types of artifacts aretypically considered pertinent to DCT block (JPEG, MPEG and H.264) codeddata: blocking, ringing and blurring. Blocking appears in all block-based com-pression techniques due to coarse quantization of frequency components [16]. Itcan be observed as surface discontinuity (edge) at block boundaries. These edgesare perceived as abnormal high-frequency components in the spectrum. Ring-ing is observed as periodic pseudo edges around original edges [9]. It is due toimproper truncation of high frequency components. This artifact is also knownas the Gibbs phenomenon or Gibbs effect. In the worst case, the edges can beshifted far away from the original edge locations, observed as false edge. Blur-ring, which appears as edge smoothness or texture blur, is due to the loss of highfrequency components when compared with the original image. Blurring causesthe received image to be smoother than the original one [5].Broadly, the different artifacts can be considered to add texture not presentin original content, to the images. However, general-purpose texture descriptorsare rarely, if ever, used as basis for deriving the metrics of the level of artifactsintroduced into multimedia content. In the study presented here, we explore theapplicability of Local Binary Patterns (LBP) to the problem of creating qualitymetrics. LBPs represent texture descriptors which have been successfully usedin a number of computer vision applications. They have, so far as we know, notbeen considered for no-reference image and video quality assessment.The goal of each no-reference approach is to create an estimator based onthe proposed features that would predict the Mean Opinion Score (MOS)[8] ofhuman observers, without using the original (not-degraded) image or sequencedata. In the study presented here, we evaluated the applicability of LBPs asfeatures on which to base no-reference MOS estimation and the Multilayer Per-ceptron (MLP) [6] neural network as an estimator. To evaluate both the proposedapproach images from a public and commonly LIVE Image Quality AssessmentDatabase [13] were used. The experiments conducted show that the proposedapproach is able to achieve no-reference image quality assessment beyond thatof a widely used state-of-the art approach.LBPs and NNs for Image and Video QA 32 Related WorkWhen objective image and video quality is concerned, most studies focus on full-reference approaches, which assume the availability of pristine, uncompressedoriginal content at the time of assessment. This is a fairly well researched andsolved problem. Evaluation of a number of such approaches on the LIVE databaseis presented in [14].In a realistic multimedia content delivery over telecommunication networkscenario, these algorithms can only be used at server side, or in specific caseswhere the pristine content has been previously delivered to client side. The prob-lem of no-reference quality assessment is more complex and the need for bettersolutions still exists.Wang et al.[16] proposed an early no-reference approach to quality assessmentin JPEG coded images. Their final measure is derived as a non-linear combina-tion of a blockiness, local activity and a so-called zero-crossing measure. Thecombination is supposed to provide information regarding both blockiness andblurring (via the two latter measures) in JPEG coded images. Their approachremains to this date the one usually compared against, when no-reference qual-ity assessment is concerned. Therefore, we evaluated the results of the approachproposed here against that of the approach proposed in [16].More recently Culibrk et al. proposed a VQA approach that used differentpreviously proposed artifact and quality metrics as basic features and machinelearning algorithms to create a no-reference video quality (MOS) estimator [3].The study demonstrated the viability of a machine approach and the abilityto achieve superior quality estimation this way. In fact, one of the algorithmsevaluated was MLP. In addition, the authors provided evidence that the basicfeatures used by Wang et al. are among best predictors selected from the set of35 classic measures and saliency-related features evaluated.LBPs were originally proposed as a texture descriptor [11]. They have sincethen been employed as dynamic texture descriptors [18], as well as found anumber of successful applications such as face recognition [1] and object detection[15]. To the best of our knowledge it has never been applied to the problem ofimage and video quality assessment.3 LBP-VQA approachThe LBP operator is primarily used as an unifying approach to structural andstatistical texture analysis. A detailed explanation of LBP derivation and itsextensions is provided in [10]. Equation (1) describes the process of obtainingLBP code for a local area in a image, defined by the location of the central pixel(xc, yc), radius from it R and number of pixels that surround it P at the distanceof the radius R.LBPP,R(xc, yc) =P∑p=0s(gp − gc)2p (1)4 LBPs and NNs for Image and Video QAFig. 1. Referent image (left) and images compressed with bitrates 0.45313, 0.3263, and0.15771, respectively (right).If the location of surrounding pixels (xp, yp), where p denotes the index ofneighboring pixel does not match the pixel grid exactly, the intensity of thatpixel is obtained through bilinear interpolation. The differences in the valuesbetween the central pixel and those surrounding it, as defined by a local area(mask) with parameters R and P , are mapped to zero or one using the functions(gp − gc). Where gc and gp denote gray value of central pixel and surroundingpixel respectively:s(x) ={1 if x ≥ 00 if x < 0(2)The LBP value is obtained as a sum of ones weighted by 2p, where p corre-sponds to the index of the pixel in the mask. Thus, the LBP value is a decimalrepresentation of the binary number extracted from the predefined mask. Thestart of the pixel counting sequence within the mask is chosen arbitrarily thefirst time, and then must remain unchanged.Finally, the feature vector is a histogram of values of LBP codes calculatedfor the whole image. Figure 1 shows referent and images compressed with dif-ferent bitrates. Figure 2 shows the LBP codes for the same images. As artifactsintroduced by JPEG compression become more visible, the distribution of LBPcodes changes in such a way that only a few codes remain, representing blocki-ness and uniform regions. Therefore, histograms of LBP codes seem to be a goodrepresentation of image quality, as one can easily distinguish between those per-taining to compressed and non-compressed images (Figure 3). It should be notedthat all histograms were normalized using√‖h‖1 norm, where h represents thehistogram.4 Experiments and ResultsThe evaluation was performed using images from the LIVE Image Quality As-sessment Database [13]. The images were compressed using JPEG and formLBPs and NNs for Image and Video QA 5Fig. 2. LBP codes for images in Figure 1.Fig. 3. Histograms of LBP codes for images in Figure 1.Fig. 4. Correlation coefficients and RMSE values for different MLP ridge parametervalues and LBP codes generated for diverse neighborhoods: R = 1, N = 8 - circles,R = 2, N = 16 - crosses, and R = 3, N = 24 - triangles.6 LBPs and NNs for Image and Video QAthe release 1 of the database. The database was created from twenty-nine high-resolution 24-bits/pixel RGB color images (typically 768 x 512) using differentcompression ratios. This yielded 204 images. For all 233 images, subjective MOSis available.Feature extraction was implemented in C++ using OpenCV library [2]. Weevaluated the MLP neural network as a MOS predictor. The network was trainedand its performance tested using the machine learning package Weka [17]. Theneural network had one hidden layer and was trained by minimizing the squarederror plus a quadratic penalty with the BroydenFletcherGoldfarbShanno (BFGS)method [4].To provide for better evaluation we considered different values for the keyparameters. The estimator proved to be most sensitive to the value of the ridgeparameter, which controls the overfitting. Figure 4 shows how the performanceof model changes for different ridge parameter values. We considered three sizesfor the neighborhoods used to extract the LBP codes. The results shown wereobtained using 10-fold cross-validation and the network that had just 2 neuronsin the hidden layer. Due to the complexity of the BFGS model, 2 neurons in thehidden layer are the default in Weka.As Figure 4 shows, good correlation and low RMSE can be achieved us-ing the proposed approach. For the ridge parameter value of 0.7, correlation of0.9767 and a RMSE of 0.4890 is achieved between the predicted and subjectiveMOS. When tested on the train set, slightly better result are obtained, 0.9867for the correlation coefficient and 0.3740 for RMSE, indicating that the modelgeneralizes well and is stable.Figure 5 shows the scatter plots of predictions of the proposed approach andthe well-known model proposed Wang and Bovik [16]. The scatters show thecomparison of model predictions, using both 10-fold cross-validation and whentested on the train set. The evaluation on the training set was included as thefinal model of Wang and Bovik was fitted using all the images in the dataset,making the evaluation of their model, as conducted here, equivalent to testingon the training set.As the scatters show, the performance of the proposed approach is signifi-cantly better than that reported by Wang et al. in [16], as their model achievedan average RMSE of 0.7256. In addition, proposed model correlates with MOSvery well .5 ConclusionThe paper proposes an approach to image and video quality assessment basedon LBPs as features and an MLP neural network estimator.The proposed approach was evaluated on a standard database of JPEG im-ages and achieved superior performance when compared to a classic no-referenceimage quality assessment methodology.The study demonstrated the usefulness of LBPs and neural networks for qual-ity assessment. Something that has not been explored before. In the future theLBPs and NNs for Image and Video QA 7Fig. 5. MOS predictions for the proposed approach (crosses) and Wang and Bovikmodel (circles), when cross-validated (top) and evaluated on the training set (bottom)8 LBPs and NNs for Image and Video QAstudy will be extended to evaluate the proposed approach on H.264 videos and tocreate a blockiness artifact detector which will be able to detect isolated artifactsthat appear due to network-introduced errors in multimedia transmission.AcknowledgmentThis research is supported in part by the FP7 project QoSTREAM.References1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns:Application to face recognition. Pattern Analysis and Machine Intelligence, IEEETransactions on 28(12), 2037–2041 (2006)2. Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)3. Culibrk, D., Mirkovic, M., Zlokolica, V., Pokric, M., Crnojevic, V., Kukolj, D.:Salient Motion Features for Video Quality Assessment. IEEE Trans. on ImageProcessing 20, 948–958 (2010)4. Dennis, J.E., Schnabel, R.B.: Numerical methods for unconstrained optimizationand nonlinear equations, vol. 16. Society for Industrial and Applied Mathematics(1987)5. Ferzli, R., Karam, L.: A no-reference objective image sharpness metric based onjust-noticeable blur and probability summation. Proceedings of IEEE 2007 Inter-national Conference on Image Processing 3, III –445–III –448 (16 2007-Oct 192007)6. Haykin, S.: Neural Networks: A Comprehensive Foundation. Macmillan, New York(1994)7. Index, C.: Forecast and methodology, 2011-2016. White paper, CISCO, December(2012)8. ITU-R BT.500: Methodology for the Subjective Assessment of the Quality of Tele-vision Pictures. Video Quality Experts Group (2002)9. Kirenko, I.: Reduction of coding artifacts using chrominance and luminance spatialanalysis. Consumer Electronics, 2006. ICCE ’06. 2006 Digest of Technical Papers.International Conference on pp. 209–210 (Jan 2006)10. Ma¨enpa¨a¨, T.: The Local Binary Pattern Approach to Texture Analysis: Extenxionsand Applications. Oulun yliopisto (2003)11. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotationinvariant texture classification with local binary patterns. Pattern Analysis andMachine Intelligence, IEEE Transactions on 24(7), 971–987 (2002)12. Seshadrinathan, K., Bovik, A.: An information theoretic video quality metric basedon motion models. In: Third International Workshop on Video Processing andQuality Metrics for Consumer Electronics. pp. 25–26. Citeseer (2007)13. Seshadrinathan, K., Soundararajan, R., Bovik, A., Cormack, L.: Study of subjec-tive and objective quality assessment of video. Image Processing, IEEE Transac-tions on 19(6), 1427–1441 (2010)14. Seshadrinathan, K., Soundararajan, R., Bovik, A., Cormack, L.: A subjective studyto evaluate video quality assessment algorithms. In: SPIE Proceedings HumanVision and Electronic Imaging. vol. 7527. Citeseer (2010)LBPs and NNs for Image and Video QA 915. Wang, X., Han, T., Yan, S.: An hog-lbp human detector with partial occlusionhandling. In: Computer Vision, 2009 IEEE 12th International Conference on. pp.32–39. IEEE (2009)16. Wang, Z., Sheikh, H.R., Bovik, A.C.: No-reference perceptual quality assessment ofjpeg compressed images. In: Proceedings of IEEE 2002 International Conferencingon Image Processing. pp. 477–480 (2002)17. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and tech-niques, 2nd Edition. Morgan Kaufmann, San Francisco (2005)18. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patternswith an application to facial expressions. Pattern Analysis and Machine Intelli-gence, IEEE Transactions on 29(6), 915–928 (2007)

Local binary patterns and neural networks for no-reference image and video quality assessment

https://biblio.ugent.be/publication/8529685/file/8535714

Local binary patterns and neural networks for no-reference image and video quality assessment

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography