Staelens, Nicolas

English

Ghent University Academic Bibliography

Quality of Experience Assessment of Video inReal-life EnvironmentsNicolas StaelensSupervisor(s): Brecht Vermeulen, Piet DemeesterI. INTRODUCTIONAccording to ITU-T Rec. P.10/G.100 Amd.2, Quality of Experience (QoE) includes thecomplete end-to-end system effects and can beinfluenced by user expectations and context.Therefore, the QoE of IPTV and Video on De-mand (VoD) services should ideally be mea-sured in the home environment. Unfortunately,current existing subjective video quality assess-ment methodologies require a controlled envi-ronment where human observers rate the visualquality of a series of short video clips (10 - 15sec). In this article, we compare the results ofa subjective test conducted in real-life environ-ments with the results obtained using a stan-dardized methodology.II. REAL-LIFE QUALITY OF EXPERIENCEASSESSMENTExisting subjective quality assessment meth-odologies, specified in ITU-T Rec. P.910 andITU-R Rec. BT.500-11, pose stringent de-mands on the way subjective experiments areset up and conducted. As a consequence, theycannot be used for the quality assessment oflong video sequences in realistic environments.In order to enable real-life QoE assess-ment of IPTV and VoD services, we used anovel methodology based on full length DVDmovies. This new methodology tries to mimicthe typical lean-backward TV-viewing expe-rience by letting the subjects take the movie—————————————————–N. Staelens is with the Department of InformationTechnology, Ghent University - IBBT, Belgium. E-mail: nicolas.staelens@intec.ugent.be .home to watch it in the same environment theyusually watch television. Using the toolchaindescribed in [1], controlled visual degradationscan be inserted in the movie prior to writing iton a compliant DVD disc.III. SUBJECTIVE TESTTwo subjective tests were conducted usingour new methodology described in the previ-ous section. In [1], we used movies which con-tained up to three visual impairments to studythe tolerance towards random blockiness andframe freezes. A second subjective test wasconducted in [2] in order to assess the influ-ence of H.264 SVC scalability on end-users’QoE. During this test, a movie was used inwhich six fragments were temporally and qual-ity (SNR) scaled down. In both tests, subjectswere not informed about any possible degra-dations.Feedback was collected by means of aquestionnaire which was provided to the sub-jects in a sealed envelope.In order to highlight the importance of mea-suring QoE in real-life environments, we com-pared the results from [1] and [2] with the re-sults obtained using a standardized assessmentmethodology1. For this standard test, we ex-tracted the fragments containing a visual im-pairment from the DVDs in order to create shortvideo sequences of 15 seconds long.—————————————————–1based on the Single Stimulus Absolute CategoryRating (SS ACR) method from ITU-R Rec. BT.500-11.IV. RESULTSAs shown in Figure 1, visual impairmentsare much more detected during a standard sub-jective test. Before such tests, subjects re-ceive specific instructions on how to evaluatethe video sequences. As a consequence, sub-jects’ primary focus is on visual quality eval-uation. While watching a movie, subjects areconcentrated on content.20%30%40%50%60%70%80%90%100%%  pe rc ei ve dReal‐life QoE assessmentSS ACR method0%10%Frame freeze BlockinessVisual impairmentFigure 1. Impairment detection during real-life QoEassessment and a standard subjective test.Despite the fact that frame freezes are lessdetected, they were rated more annoying dur-ing movies as they break the natural flow andthe fluidity of the playback. However, dur-ing the standard test, quality scores for framefreezes were significantly higher compared tothe scores for blockiness.A similar result is obtained regarding the vis-ibility of temporal and quality scalability, de-picted in Figure 2. The graph indicates that im-pairment visibility is also content dependent.Overall quality ratings in [1] showed thatsubjects tolerate up to two visual impairmentsduring playback. It is important to mention thatthis quality rating is higher compared to the rat-ings subjects gave to the short clips, which onlycontained one visual degradation.V. CONCLUSIONSOur results show significant differences con-cerning impairment visibility and tolerancewhen conducting subjective video quality testsin real-life environments. Furthermore, our re-search indicates that results obtained using a20%30%40%50%60%70%80%90%100%%  pe rc ei ve dTemporal ScalabilitySNR Scalability0%10%1 2 3 4 5 6Scene(a)Real-life QoE assessment20%30%40%50%60%70%80%90%100%%  pe rc ei ve dTemporal ScalabilitySNR Scalability0%10%1 2 3 4 5 6Scene(b)SS ACR methodFigure 2. Downscaling visibility during real-life as-sessment (a) and during a standard test (b).standard video quality assessment methodol-ogy do not always hold in the case of real-lifeQoE assessment.ACKNOWLEDGMENTSThis paper is the result of research carriedout as part of the Video Q-SAC project, partlyfunded by the IBBT.REFERENCES[1] N. Staelens, B. Vermeulen, S. Moens, J.-F. Macq,P. Lambert, R. Van de Walle, and P. Demeester,“Assessing the Influence of Packet Loss andFrame Freezes on the Perceptual Quality of FullLength Movies,” Fourth International Workshopon Video Processing and Quality Metrics for Con-sumer Electronics, January 2009.[2] N. Staelens, S. Moens, W. Van den Broeck,I. Marie¨n, B. Vermeulen, P. Lambert, R. Van deWalle, and P. Demeester, “Assessing the Per-ceptual Influence of H.264/SVC Signal-to-NoiseRatio and Temporal Scalability on Full LengthMovies,” First International Workshop on Qual-ity of Multimedia Experience, July 2009.

Quality of experience assessment of video in real-life environments

https://biblio.ugent.be/publication/891000/file/920385.pdf

Quality of experience assessment of video in real-life environments

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography