Adaptive Video Streaming Testbed Design for Performance Study and Assessment of QoE

[EN] Hypertext Transfer Protocol adaptive streaming switches between different video qualities, adapting to the network conditions, and avoids stalling streamed frames over high¿oscillation client's throughput improving the users' quality of experience (QoE). Quality of experience has become the most important parameter to lead the service providers to know about the end¿user feedback. Implementing Hypertext Transfer Protocol adaptive streaming applications to find out QoE in real¿life scenarios of vast networks becomes more challenging and complex task regarding to cost, agile, time, and decisions. In this paper, a virtualized network testbed to virtualize various machines to support implementing experiments of adaptive video streaming has been developed. Within the test study, the metrics which demonstrate performance of QoE are investigated, respectively, including initial delay (ie, startup delay at the beginning of playback a video), frequency switches (ie, number of times the quality is changed), accumulative video time (ie, number and length of stalls), CPU usage, and battery energy consumption. Furthermore, the relation between effective parameters of QoS on the aforementioned metrics for different segment length is investigated. Experimental results show that the proposed virtualized system is agile, easy to install and use, and costs less than real testbeds. Moreover, the subjective and objective performance studies of QoE evaluation in the system have proven that the segment lengths of 6 to 8 seconds were faired and more efficient than others according to the investigated parameters.Ministerio de Economia y Competitividad, Grant/Award Number: TIN2014-57991-C3-1-PAbdullah, MTA.; Lloret, J.; Ali, A.; García-García, L. (2018). Adaptive Video Streaming Testbed Design for Performance Study and Assessment of QoE. International Journal of Communication Systems. 1-15. https://doi.org/10.1002/dac.3551S11

A Comparative Case Study of HTTP Adaptive Streaming Algorithms in Mobile Networks

HTTP Adaptive Streaming (HAS) techniques are now the dominant solution for video delivery in mobile networks. Over the past few years, several HAS algorithms have been introduced in order to improve user quality-of-experience (QoE) by bit-rate adaptation. Their difference is mainly the required input information, ranging from network characteristics to application-layer parameters such as the playback buffer. Interestingly, despite the recent outburst in scientific papers on the topic, a comprehensive comparative study of the main algorithm classes is still missing. In this paper we provide such comparison by evaluating the performance of the state-of-the-art HAS algorithms per class, based on data from field measurements. We provide a systematic study of the main QoE factors and the impact of the target buffer level. We conclude that this target buffer level is a critical classifier for the studied HAS algorithms. While buffer-based algorithms show superior QoE in most of the cases, their performance may differ at the low target buffer levels of live streaming services. Overall, we believe that our findings provide valuable insight for the design and choice of HAS algorithms according to networks conditions and service requirements.Comment: 6 page

Measurement And Improvement of Quality-of-Experience For Online Video Streaming Services

Title from PDF of title page, viewed on September 4, 2015Dissertation advisor: Deep MedhiVitaIncludes bibliographic references (pages 126-141)Thesis (Ph.D.)--School of Computing and Engineering. University of Missouri--Kansas City, 2015HTTP based online video streaming services have been consistently dominating the online traffic for the past few years. Measuring and improving the performance of these services is an important challenge. Traditional Quality-of-Service (QoS) metrics such as packet loss, jitter and delay which were used for networked services are not easily understood by the users. Instead, Quality-of-Experience (QoE) metrics which capture the overall satisfaction are more suitable for measuring the quality as perceived by the users. However, these QoE metrics have not yet been standardized and their measurement and improvement poses unique challenges. In this work we first present a comprehensive survey of the different set of QoE metrics and the measurement methodologies suitable for HTTP based online video streaming services. We then present our active QoE measurement tool Pytomo that measures the QoE of YouTube videos. A case study on the measurement of QoE of YouTube videos when accessed by residential users from three different Internet Service Providers (ISP) in a metropolitan area is discussed. This is the first work that has collected QoE data from actual residential users using active measurements for YouTube videos. Based on these measurements we were able to study and compare the QoE of YouTube videos across multiple ISPs. We also were able to correlate the QoE observed with the server clusters used for the different users. Based on this correlation we were able to identify the server clusters that were experiencing diminished QoE. DynamicAdaptive Streaming overHTTP (DASH) is an HTTP based video streaming that enables the video players to adapt the video quality based on the network conditions. We next present a rate adaptation algorithm that improves the QoE of DASH video streaming services that selects the most optimum video quality. With DASH the video server hosts multiple representation of the same video and each representation is divided into small segments of constant playback duration. The DASH player downloads the appropriate representation based on the network conditions, thus, adapting the video quality to match the conditions. Currently deployed Adaptive Bitrate (ABR) algorithms use throughput and buffer occupancy to predict segment fetch times. These algorithms assume that the segments are of equal size. However, due to the encoding schemes employed this assumption does not hold. In order to overcome these limitations, we propose a novel Segment Aware Rate Adaptation algorithm (SARA) that leverages the knowledge of the segment size variations to improve the prediction of segment fetch times. Using an emulated player in a geographically distributed virtual network setup, we compare the performance of SARA with existing ABR algorithms. We demonstrate that SARA helps to improve the QoE of the DASH video streaming with improved convergence time, better bitrate switching performance and better video quality. We also show that unlike the existing adaptation schemes, SARA provides a consistent QoE irrespective of the segment size distributions.Introduction -- Measurement of QoE for Online Video Streaming Services: A Literature Survey -- Pytomo: A Tool for measuring QoE of YouTube Videos -- Case Study: QoE across three Internet Service Providers in a Metropolitan Area -- Adaptive Bitrate Algorithms for DASH -- Segment Aware Rate Adaptation for DASH -- Performance Evaluation of SARA -- Conclusion and Future Research --Appendix A. Sample MPD Fil

Look ahead to improve QoE in DASH streaming

[EN] When a video is encoded with constant quality, the resulting bitstream will have variable bitrate due to the inherent nature of the video encoding process. This paper proposes a video Adaptive Bitrate Streaming (ABR) algorithm, called Look Ahead, which takes into account this bitrate variability in order to calculate, in real time, the appropriate quality level that minimizes the number of interruptions during the playback. The algorithm is based on the Dynamic Adaptive Streaming over HTTP (DASH) standard for on-demand video services. In fact, it has been implemented and integrated into ExoPlayer v2, the latest version of the library developed by Google to play DASH contents. The proposed algorithm is compared to the Müller and Segment Aware Rate Adaptation (SARA) algorithms as well as to the default ABR algorithm integrated into ExoPlayer. The comparison is carried out by using the most relevant parameters that affect the Quality of Experience (QoE) in video playback services, that is, number and duration of stalls, average quality of the video playback and number of representation switches. These parameters can be combined to define a QoE model. In this sense, this paper also proposes two new QoE models for the evaluation of ABR algorithms. One of them considers the bitrate of every segment of each representation, and the second is based on VMAF (Video Multimethod Assessment Fusion), a Video Quality Assessment (VQA) method developed by Netflix. The evaluations presented in the paper reflect: first, that Look Ahead outperforms the Müller, SARA and the ExoPlayer ABR algorithms in terms of number and duration of video playback stalls, with hardly decreasing the average video quality; and second, that the two QoE models proposed are more accurate than other similar models existing in the literature.This work is supported by the PAID-10-18 Program of the Universitat Politecnica de Valencia (Ayudas para contratos de acceso al sistema espanol de Ciencia, Tecnologia e Innovacion, en estructuras de investigacion de la Universitat Politecnica de Valencia) and by the Project 20180810 from the Universitat Politecnica de Valencia ("Tecnologias de distribucion y procesado de informacion multimedia y QoE").Belda Ortega, R.; De Fez Lava, I.; Arce Vila, P.; Guerri Cebollada, JC. (2020). Look ahead to improve QoE in DASH streaming. Multimedia Tools and Applications. 79(33-34):25143-25170. https://doi.org/10.1007/s11042-020-09214-9S25143251707933-34Akhshabi S, Narayanaswamy S, Begen AC, Dovrolis C (2012) An experimental evaluation of rate-adaptive video players over HTTP. Signal process. Image Commun 27(4):271–287. https://doi.org/10.1016/j.image.2011.10.003Android Developers webpage, ExoPlayer. Available online at: https://developer.android.com/guide/topics/media/exoplayer.html . Accessed: Jun. (2019)Bampis CG, Li Z, Bovik AC (2018) SpatioTemporal feature integration and model fusion for full reference video quality assessment. IEEE Trans on Circuits and Syst for Video Tech 29:2256–2270. https://doi.org/10.1109/TCSVT.2018.2868262Barman N, Martini MG (2019) QoE modeling for HTTP adaptive video streaming - a survey and open challenges. IEEE Access 7:30831–30859. https://doi.org/10.1109/ACCESS.2019.2901778Belda R (2013) Algoritmo de adaptación DASH: Look Ahead. Master Thesis. Universitat Politècnica de València. http://hdl.handle.net/10251/33359 .Belda R, de Fez I, Arce P, Guerri J C (2018) Look ahead: a DASH adaptation algorithm. Proc. of the IEEE Int. Symp. On broadband multimed. Syst. And broadcast., Valencia, Spain: article no. 158. https://doi.org/10.1109/BMSB.2018.8436718 .Blender Foundation webpage. Available online at: https://www.blender.org/foundation . Accessed: Jun. (2019).Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20-3:273–297. https://doi.org/10.1023/A:1022627411411DASH Industry forum webpage. Available online at: http://dashif.org . Accessed: Jun. (2019)Ghadiyaram D, Pan J, Bovik AC (2019) A subjective and objective study of stalling events in mobile streaming videos. IEEE Trans on Circuits and Syst for Video Technol 29(1):183–197. https://doi.org/10.1109/TCSVT.2017.2768542Ghent University. 4G/LTE bandwidth logs. Available online at: http://users.ugent.be/~jvdrhoof/dataset-4g . Accessed: Jun. (2019).Github webpage. A DASH segment size aware rate adaptation model for DASH. Available online at: https://github.com/pari685/AStream . Accessed: Jun. (2019)GitHub website. Dashgen, Multimedia Communications Group. Available online at: https://github.com/comm-iteam/dashgen . Accessed: Jun. (2019).van der Hooft J, Petrangeli S, Wauters T, Huysegems R, Alface PR, Bostoen T, De Turck F (2016) HTTP/2-based adaptive streaming of HEVC video over 4G/LTE networks. IEEE Commun Lett 20(1):2177–2180. https://doi.org/10.1109/LCOMM.2016.2601087Huang TY, Johari R, McKeown N, Trunnell M, Watson M (2014) A buffer-based approach to rate adaptation: evidence from a large video streaming service. Proc. of the 2014 ACM Conf. On SIGCOMM, Chicago, IL, USA: 187-198. https://doi.org/10.1145/2619239.2626296Institute of Telecommunications and Multimedia Applications website. Look Ahead Demo. Available online at: https://lookahead.iteam.upv.es . Accessed: Jun. (2019)ISO/IEC 23009–1:2014 (2014) Dynamic adaptive streaming over HTTP (DASH) - Part 1: media presentation description and segment formats.Juluri P, Tamarapalli V, Medhi D (2015) SARA: segment aware rate adaptation algorithm for dynamic adaptive streaming over HTTP. Proc. of the IEEE Int. Conf. On Commun. Workshop (ICCW), London, UK: 1765-1770. https://doi.org/10.1109/ICCW.2015.7247436 .Juluri P, Tamarapalli V, Medhi D (2016) QoE management in DASH systems using the segment aware rate adaptation algorithm. Proc. of the IEEE/IFIP Netw. Oper. And Manag. Symp. (NOMS), Istanbul, Turkey: 129-136. https://doi.org/10.1109/NOMS.2016.7502805 .Kua J, Armitage G, Branch P (2017) A survey of rate adaptation techniques for dynamic adaptive streaming over HTTP. IEEE Commun Surv & Tutor 19(3):1842–1866. https://doi.org/10.1109/COMST.2017.2685630Lee S, Youn K, Chung K (2015) Adaptive video quality control scheme to improve QoE of MPEG DASH. Proc. of IEEE Int. Conf. On Consum. Electron. (ICCE), Las Vegas, NV, USA: 126-127. https://doi.org/10.1109/ICCE.2015.7066348 .Li S, Zhang F, Ma L, Ngan K (2011) Image quality assessment by separately evaluating detail losses and additive impairments. IEEE Trans. on Multimed. 13-5:935–949. https://doi.org/10.1109/TMM.2011.2152382Liu C, Bouazizi I, Gabbouj M (2011) Rate adaptation for adaptive HTTP streaming. Proc. of the second annual ACM Conf. On multimed. Syst. (MMSys), San Jose, CA, USA: 169-174. https://doi.org/10.1145/1943552.1943575 .Medium webpage (2016) Toward a practical perceptual video quality metric. Available online at: https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652 . Accessed: Jun. 2019.Mobile Video Service Performance Study (2015) HUAWEI white paper. Available online at: http://www.ctiforum.com/uploadfile/2015/0701/20150701091255294.pdf .Mok RKP, Luo X, Chan EWW, Chang RKC (2012) QDASH: a QoE-aware DASH system. Proc. of multim. Syst. Conf. (MMSys), Chapel Hill, NC, USA: 11-22. https://doi.org/10.1145/2155555.2155558Moldovan C, Hagn K, Sieber C, Kellerer W, Hoßfeld T (2017) Keep calm and don’t switch: about the relationship between switches and quality in HAS. Proc. of the Int. Teletraffic Congr. (ITC), Genoa, Italy: pp. 1-6. https://doi.org/10.23919/ITC.2017.8065802Müller C, Lederer S, Timmerer C (2012) An evaluation of dynamic adaptive streaming over HTTP in vehicular environments. Proc. of the 4th workshop on mob. Video (MoVid), Chapel Hill, NC, USA: 37-42. https://doi.org/10.1145/2151677.2151686Nguyen T, Vu T, Nguyen DV, Ngoc NP, and Thang TC (2015) QoE optimization for adaptive streaming with multiple VBR videos. Proc. of the Int. Conf. On comp., Manag. And Telecommun. (ComManTel), DaNang, Vietnam: 189-193. https://doi.org/10.1109/ComManTel.2015.7394285 .Qin Y, H. Shuai, Pattipati K R, Qian F, Sen S, Wang B, Yue C (2018) ABR Streaming of VBR-encoded videos: characterization, challenges, and solutions. Proc. of ACM CoNext 2018, Heraklion, Greece: 366–378. https://doi.org/10.1145/3281411.3281439 .Samain J, Carofiglio G, Muscariello L, Papalini M, Sardara M, Tortelli M, Rossi D (2017) Dynamic adaptive video streaming: towards a systematic comparison of ICN and TCP/IP. IEEE Trans on Multimed 19(10):2166–2181. https://doi.org/10.1109/TMM.2017.2733340Sheikh H, Bovik A (2006) Image information and visual quality. IEEE Trans on Image Process 15(2):430–444. https://doi.org/10.1109/TIP.2005.859378Shuai Y, Herfet T (2016). A buffer dynamic stabilizer for low-latency adaptive video streaming. Proc. of the Int. Conf. on Consum. Electron., Berlin: 1–5. https://doi.org/10.1109/ICCE-Berlin.2016.7684742 .Tavakoli S, Egger S, Seufert M, Schatz R, Brunnström K, García N (2016) Perceptual quality of HTTP adaptive streaming strategies: cross-experimental analysis of multi-laboratory and crowdsourced subjective studies. IEEE Journal on Select Areas in Commun 34-8:2141–2153. https://doi.org/10.1109/JSAC.2016.2577361Yarnagula H K, Juluri P, Mehr S K, Tamarapalli V, Medhi D (2019) QoE for Mobile clients with segment-aware rate adaptation algorithm (SARA) for DASH video streaming. ACM trans. On multimed. Comput., Commun., and Appl. (TOMM) 15(2):article no. 36 https://doi.org/10.1145/3311749 .Yin X, Sekar V, Sinopoli B (2014) Toward a principled framework to design dynamic adaptive streaming algorithms over HTTP. Proc. of the 13th ACM workshop on hot topics in Netw. (HotNets), Los Angeles, CA, USA: 1-7. https://doi.org/10.1145/2670518.2673877 .YouTube webpage (2019) Youtube press. Available online at: https://www.youtube.com/yt/about/press . Accessed: Jun. 2019.Youtube webpage, Google I/O ‘18: Building feature-rich media apps with ExoPlayer. Available online at: https://youtu.be/svdq1BWl4r8?t=2m . Published: May (2018)Yu L, Tillo T, Xiao J (2017) QoE-driven dynamic adaptive video streaming strategy with future information. IEEE Trans on Broadcast 63-3:523–534. https://doi.org/10.1109/TBC.2017.2687698Zhao S, Li Z, Medhi D, Lai P, Liu S (2017) Study of user QoE improvement for dynamic adaptive streaming over HTTP (MPEG-DASH). Proc. of the Int. Conf. On Comput., network. And Commun. (ICNC): multimed. Comput. And Commun., Santa Clara, CA, USA: 566-570. https://doi.org/10.1109/ICCNC.2017.7876191 .Zhou Y, Duan Y, Sun J, Guo Z (2014) Towards a simple and smooth rate adaption for VBR video in DASH. Proc. of the IEEE Vis. Commun. and Image Process. Conf, Valletta, pp 9–12. https://doi.org/10.1109/VCIP.2014.7051491Zhou C, Lin C-W, Guo Z (2016) mDASH: a Markov decision-based rate adaptation approach for dynamic HTTP streaming. IEEE Trans. on Multimed 18(4):738–751. https://doi.org/10.1109/TMM.2016.252265

QoE-Based Low-Delay Live Streaming Using Throughput Predictions

Recently, HTTP-based adaptive streaming has become the de facto standard for video streaming over the Internet. It allows clients to dynamically adapt media characteristics to network conditions in order to ensure a high quality of experience, that is, minimize playback interruptions, while maximizing video quality at a reasonable level of quality changes. In the case of live streaming, this task becomes particularly challenging due to the latency constraints. The challenge further increases if a client uses a wireless network, where the throughput is subject to considerable fluctuations. Consequently, live streams often exhibit latencies of up to 30 seconds. In the present work, we introduce an adaptation algorithm for HTTP-based live streaming called LOLYPOP (Low-Latency Prediction-Based Adaptation) that is designed to operate with a transport latency of few seconds. To reach this goal, LOLYPOP leverages TCP throughput predictions on multiple time scales, from 1 to 10 seconds, along with an estimate of the prediction error distribution. In addition to satisfying the latency constraint, the algorithm heuristically maximizes the quality of experience by maximizing the average video quality as a function of the number of skipped segments and quality transitions. In order to select an efficient prediction method, we studied the performance of several time series prediction methods in IEEE 802.11 wireless access networks. We evaluated LOLYPOP under a large set of experimental conditions limiting the transport latency to 3 seconds, against a state-of-the-art adaptation algorithm from the literature, called FESTIVE. We observed that the average video quality is by up to a factor of 3 higher than with FESTIVE. We also observed that LOLYPOP is able to reach a broader region in the quality of experience space, and thus it is better adjustable to the user profile or service provider requirements.Comment: Technical Report TKN-16-001, Telecommunication Networks Group, Technische Universitaet Berlin. This TR updated TR TKN-15-00

Understanding user experience of mobile video: Framework, measurement, and optimization

Since users have become the focus of product/service design in last decade, the term User eXperience (UX) has been frequently used in the field of Human-Computer-Interaction (HCI). Research on UX facilitates a better understanding of the various aspects of the user’s interaction with the product or service. Mobile video, as a new and promising service and research field, has attracted great attention. Due to the significance of UX in the success of mobile video (Jordan, 2002), many researchers have centered on this area, examining users’ expectations, motivations, requirements, and usage context. As a result, many influencing factors have been explored (Buchinger, Kriglstein, Brandt & Hlavacs, 2011; Buchinger, Kriglstein & Hlavacs, 2009). However, a general framework for specific mobile video service is lacking for structuring such a great number of factors. To measure user experience of multimedia services such as mobile video, quality of experience (QoE) has recently become a prominent concept. In contrast to the traditionally used concept quality of service (QoS), QoE not only involves objectively measuring the delivered service but also takes into account user’s needs and desires when using the service, emphasizing the user’s overall acceptability on the service. Many QoE metrics are able to estimate the user perceived quality or acceptability of mobile video, but may be not enough accurate for the overall UX prediction due to the complexity of UX. Only a few frameworks of QoE have addressed more aspects of UX for mobile multimedia applications but need be transformed into practical measures. The challenge of optimizing UX remains adaptations to the resource constrains (e.g., network conditions, mobile device capabilities, and heterogeneous usage contexts) as well as meeting complicated user requirements (e.g., usage purposes and personal preferences). In this chapter, we investigate the existing important UX frameworks, compare their similarities and discuss some important features that fit in the mobile video service. Based on the previous research, we propose a simple UX framework for mobile video application by mapping a variety of influencing factors of UX upon a typical mobile video delivery system. Each component and its factors are explored with comprehensive literature reviews. The proposed framework may benefit in user-centred design of mobile video through taking a complete consideration of UX influences and in improvement of mobile videoservice quality by adjusting the values of certain factors to produce a positive user experience. It may also facilitate relative research in the way of locating important issues to study, clarifying research scopes, and setting up proper study procedures. We then review a great deal of research on UX measurement, including QoE metrics and QoE frameworks of mobile multimedia. Finally, we discuss how to achieve an optimal quality of user experience by focusing on the issues of various aspects of UX of mobile video. In the conclusion, we suggest some open issues for future study