A Survey of Machine Learning Techniques for Video Quality Prediction from Quality of Delivery Metrics

A growing number of video streaming networks are incorporating machine learning (ML) applications. The growth of video streaming services places enormous pressure on network and video content providers who need to proactively maintain high levels of video quality. ML has been applied to predict the quality of video streams. Quality of delivery (QoD) measurements, which capture the end-to-end performances of network services, have been leveraged in video quality prediction. The drive for end-to-end encryption, for privacy and digital rights management, has brought about a lack of visibility for operators who desire insights from video quality metrics. In response, numerous solutions have been proposed to tackle the challenge of video quality prediction from QoD-derived metrics. This survey provides a review of studies that focus on ML techniques for predicting the QoD metrics in video streaming services. In the context of video quality measurements, we focus on QoD metrics, which are not tied to a particular type of video streaming service. Unlike previous reviews in the area, this contribution considers papers published between 2016 and 2021. Approaches for predicting QoD for video are grouped under the following headings: (1) video quality prediction under QoD impairments, (2) prediction of video quality from encrypted video streaming traffic, (3) predicting the video quality in HAS applications, (4) predicting the video quality in SDN applications, (5) predicting the video quality in wireless settings, and (6) predicting the video quality in WebRTC applications. Throughout the survey, some research challenges and directions in this area are discussed, including (1) machine learning over deep learning; (2) adaptive deep learning for improved video delivery; (3) computational cost and interpretability; (4) self-healing networks and failure recovery. The survey findings reveal that traditional ML algorithms are the most widely adopted models for solving video quality prediction problems. This family of algorithms has a lot of potential because they are well understood, easy to deploy, and have lower computational requirements than deep learning techniques

DaI: Decrypt and Infer the Quality of Real-Time Video Streaming

Inferring the quality of network services is the vital basis of optimization for network operators. However, prevailing real-time video streaming applications adopt encryption for security, leaving it a problem to extract Quality of Service (QoS) indicators of real-time video. In this paper, we propose DaI, a traffic-based real-time video quality estimator. DaI can partially decrypt the encrypted real-time video data and applies machine learning methods to estimate key objective Quality of Experience (QoE) metrics of real-time video. According to the experimental results, DaI can estimate objective QoE metrics with an average accuracy of 79%

SAP: Stall-aware pacing for improved DASH video experience in cellular networks

The dramatic growth of cellular video traffic represents a practical challenge for cellular network operators in providing a consistent streaming Quality of Experience (QoE) to their users. Satisfying this objective has so-far proved elusive, due to the inherent system complexities that degrade streaming performance, such as variability in both video bitrate and network conditions. In this paper, we present SAP as a DASH video traffic management solution that reduces playback stalls and seeks to maintain a consistent QoE for cellular users, even those with diverse channel conditions. SAP achieves this by leveraging both network and client state information to optimize the pacing of individual video flows. We extensively evaluate SAP performance using real video content and clients, operating over a simulated LTE network. We implement state-of-the-art client adaptation and traffic management strategies for direct comparison. Our results, using a heavily loaded base station, show that SAP reduces the number of stalls and the average stall duration per session by up to 95%. Additionally, SAP ensures that clients with good channel conditions do not dominate available wireless resources, evidenced by a reduction of up to 40% in the standard deviation of the QoE metric

Video QoE Estimation using Network Measurement Data

More than even before, last-mile Internet Service Providers (ISPs) need to efficiently provision and manage their networks to meet the growing demand for Internet video (expected to be 82% of the global IP traffic in 2022). This network optimization requires ISPs to have an in-depth understanding of end-user video Quality of Experience (QoE). Understanding video QoE, however, is challenging for ISPs as they generally do not have access to applications at end user devices to observe key objective metrics impacting QoE. Instead, they have to rely on measurement of network traffic to estimate objective QoE metrics and use it for troubleshooting QoE issues. However, this can be challenging for HTTP-based Adaptive Streaming (HAS) video, the de facto standard for streaming over the Internet, because of the complex relationship between the network observable metrics and the video QoE metrics. This largely results from its robustness to short-term variations in the underlying network conditions due to the use of the video buffer and bitrate adaptation. In this thesis, we develop approaches that use network measurement to infer video QoE. In developing inference approaches, we provide a toolbox of techniques suitable for a diversity of streaming contexts as well as different types of network measurement data. We first develop two approaches for QoE estimation that model video sessions based on the network traffic dynamics of the HAS protocol under two different streaming contexts. Our first approach, MIMIC, estimates unencrypted video QoE using HTTP logs. We do a large-scale validation of MIMIC using ground truth QoE metrics from a popular video streaming service. We also deploy MIMIC in a real-world cellular network and demonstrate some preliminary use cases of QoE estimation for ISPs. Our second approach is called eMIMIC that estimates QoE metrics for encrypted video using packet-level traces. We evaluate eMIMIC using an automated experimental framework under realistic network conditions and show that it outperforms state-of-the-art QoE estimation approaches. Finally, we develop an approach to address the scalability challenges of QoE inference. We leverage machine learning to infer QoE from coarse-granular but light-weight network data in the form of Transport Layer Security (TLS) transactions. We analyze the scalability and accuracy trade-off in using such data for inference. Our evaluation shows that that the TLS transaction data can be used for detecting video performance issues with a reasonable accuracy and significantly lower computation overhead as compared to packet-level traces.Ph.D

On the analysis of youTube QoE in cellular networks through in-smartphone measurements

International audienceCellular-network operators are becoming increasingly interested in knowing the Quality of Experience (QoE) of their customers. QoE measurements represent today a main source of information to monitor, analyze, and subsequently manage operational networks. In this paper, we focus on the analysis of YouTube QoE in cellular networks, using QoE and distributed network measurements collected in real users' smart-phones. Relying on YoMoApp, a well-known tool for collecting YouTube smartphone measurements and QoE feedback in a crowdsourcing fashion, we have built a dataset covering about 360 different cellular users around the globe, throughout the past five years. Using this dataset, we study the characteristics of different QoE-relevant features for YouTube in smartphones. Measurements reveal a constant improvement of YouTube QoE in cellular networks over time, as well as an enhancement of the YouTube video streaming functioning in smartphones. Using the gathered measurements, we additionally investigate two case studies for YouTube QoE monitoring and analysis in cellular networks: the machine-learning-based prediction of QoE-relevant metrics from network-level measurements, and the modeling and assessment of YouTube QoE and user engagement in cellular networks and smartphone devices. Last but not least, we introduce the YoMoApp cloud dashboard to openly share smartphone YouTube QoE measurements, which allows anyone using the YoMoApp smartphone app to get immediate access to all the raw measurements collected at her devices

FlexStream: SDN-Based Framework for Programmable and Flexible Adaptive Video Streaming

With the tremendous increase in video traffic fueled by smartphones, tablets, 4G LTE networks, and other mobile devices and technologies, providing satisfactory services to end users in terms of playback quality and a fair share of network resources become challenging. As a result, an HTTP video streaming protocol was invented and widely adopted by most video providers today with the goal of maximizing the user’s quality of experience. However, despite the intensive efforts of major video providers such as YouTube and Netflix to improve their players, several studies as well as our measurements indicate that the players still suffer from several performance issues including instability and sub-optimality in the video bitrate, stalls in the playback, unfairness in sharing the available bandwidth, and inefficiency with regard to network utilization, considerably degrading the user’s QoE. These issues are frequently experienced when several players start competing over a common bottleneck. Interestingly, the root cause of these issues is the intermittent traffic pattern of the HTTP adaptive protocol that causes the players to over-estimate the available bandwidth and stream unsustainable video bitrates. In addition, the wireless network standards today do not allow the network to have a fine-grain control over individual devices which is necessary for providing resource usage coordination and global policy enforcement. We show that enabling such a network-side control would drive each device to fairly and efficiently utilize the network resources based on its current context, which would result in maximizing the overall viewing experience in the network and optimizing the bandwidth utilization. In this dissertation, we propose FlexStream, a flexible and programmable Software-Defined Network (SDN) based framework that solves all the adaptive streaming problems mentioned above. We develop FlexStream on top of the SDN-based framework that extends SDN functionality to mobile end devices, allowing for a fine-grained control and management of bandwidth based on real time context-awareness and specified policy. We demonstrate that FlexStream can be used to manage video delivery for a set of end devices over WiFi and cellular links and can effectively alleviate common problems such as player instability, playback stalls, large startup delay, and inappropriate bandwidth allocation. FlexStream offloads several tasks such as monitoring and policy enforcement to end-devices, while a network element (i.e., Global Controller), which has a global view of a network condition, is primarily employed to manage the resource allocation. This also alleviates the need for intrusive, large and costly traffic management solutions within the network, or modifications to servers that are not feasible in practice. We define an optimization method within the global controller for resource allocation to maximize video QoE considering context information, such as screen size and user priority. All features of FlexStream are implemented and validated on real mobile devices over real Wi-Fi and cellular networks. To the best of our knowledge, FlexStream is the first implementation of SDN-based control in a live cellular network that does not require any internal network support for SDN functionality

Inferring Streaming Video Quality from Encrypted Traffic: Practical Models and Deployment Experience

Inferring the quality of streaming video applications is important for Internet service providers, but the fact that most video streams are encrypted makes it difficult to do so. We develop models that infer quality metrics (\ie, startup delay and resolution) for encrypted streaming video services. Our paper builds on previous work, but extends it in several ways. First, the model works in deployment settings where the video sessions and segments must be identified from a mix of traffic and the time precision of the collected traffic statistics is more coarse (\eg, due to aggregation). Second, we develop a single composite model that works for a range of different services (i.e., Netflix, YouTube, Amazon, and Twitch), as opposed to just a single service. Third, unlike many previous models, the model performs predictions at finer granularity (\eg, the precise startup delay instead of just detecting short versus long delays) allowing to draw better conclusions on the ongoing streaming quality. Fourth, we demonstrate the model is practical through a 16-month deployment in 66 homes and provide new insights about the relationships between Internet "speed" and the quality of the corresponding video streams, for a variety of services; we find that higher speeds provide only minimal improvements to startup delay and resolution

From Network Traffic Measurements to QoE for Internet Video

International audienceVideo streaming is a dominant contributor to the global Internet traffic. Consequently, monitoring video streaming Quality of Experience (QoE) is of paramount importance to network providers. Monitoring QoE of video is a challenge as most of the video traffic of today is encrypted. In this paper, we consider this challenge and present an approach based on controlled experimentation and machine learning to estimate QoE from encrypted video traces using network level measurements only. We consider a case of YouTube and play out a wide range of videos under realistic network conditions to build ML models (classification and regression) that predict the subjective MOS (Mean Opinion Score) based on the ITU P.1203 model along with the QoE metrics of startup delay, quality (spatial resolution) of playout and quality variations, and this is using only the underlying network Quality of Service (QoS) features. We comprehensively evaluate our approach with different sets of input network features and output QoE metrics. Overall, our classification models predict the QoE metrics and the ITU MOS with an accuracy of 63-90% while the regression models show low error; the ITU MOS (1-5) and the startup delay (in seconds) are predicted with a root mean square error of 0.33 and 2.66 respectively

Investigating the Effects of Network Dynamics on Quality of Delivery Prediction and Monitoring for Video Delivery Networks

Video streaming over the Internet requires an optimized delivery system given the advances in network architecture, for example, Software Defined Networks. Machine Learning (ML) models have been deployed in an attempt to predict the quality of the video streams. Some of these efforts have considered the prediction of Quality of Delivery (QoD) metrics of the video stream in an effort to measure the quality of the video stream from the network perspective. In most cases, these models have either treated the ML algorithms as black-boxes or failed to capture the network dynamics of the associated video streams. This PhD investigates the effects of network dynamics in QoD prediction using ML techniques. The hypothesis that this thesis investigates is that ML techniques that model the underlying network dynamics achieve accurate QoD and video quality predictions and measurements. The thesis results demonstrate that the proposed techniques offer performance gains over approaches that fail to consider network dynamics. This thesis results highlight that adopting the correct model by modelling the dynamics of the network infrastructure is crucial to the accuracy of the ML predictions. These results are significant as they demonstrate that improved performance is achieved at no additional computational or storage cost. These techniques can help the network manager, data center operatives and video service providers take proactive and corrective actions for improved network efficiency and effectiveness

Identifying and diagnosing video streaming performance issues

On-line video streaming is an ever evolving ecosystem of services and technologies, where content providers are on a constant race to satisfy the users' demand for richer content and higher bitrate streams, updated set of features and cross-platform compatibility. At the same time, network operators are required to ensure that the requested video streams are delivered through the network with a satisfactory quality in accordance with the existing Service Level Agreements (SLA). However, tracking and maintaining satisfactory video Quality of Experience (QoE) has become a greater challenge for operators than ever before. With the growing popularity of content engagement on handheld devices and over wireless connections, new points-of-failure have added to the list of failures that can affect the video quality. Moreover, the adoption of end-to-end encryption by major streaming services has rendered previously used QoE diagnosis methods obsolete. In this thesis, we identify the current challenges in identifying and diagnosing video streaming issues and we propose novel approaches in order to address them. More specifically, the thesis initially presents methods and tools to identify a wide array of QoE problems and the severity with which they affect the users' experience. The next part of the thesis deals with the investigation of methods to locate under-performing parts of the network that lead to drop of the delivered quality of a service. In this context, we propose a data-driven methodology for detecting the under performing areas of cellular network with sub-optimal Quality of Service (QoS) and video QoE. Moreover, we develop and evaluate a multi-vantage point framework that is capable of diagnosing the underlying faults that cause the disruption of the user's experience. The last part of this work, further explores the detection of network performance anomalies and introduces a novel method for detecting such issues using contextual information. This approach provides higher accuracy when detecting network faults in the presence of high variation and can benefit providers to perform early detection of anomalies before they result in QoE issues.La distribución de vídeo online es un ecosistema de servicios y tecnologías, donde los proveedores de contenidos se encuentran en una carrera continua para satisfacer las demandas crecientes de los usuarios de más riqueza de contenido, velocidad de transmisión, funcionalidad y compatibilidad entre diferentes plataformas. Asimismo, los operadores de red deben asegurar que los contenidos demandados son entregados a través de la red con una calidad satisfactoria según los acuerdos existentes de nivel de servicio (en inglés Service Level Agreement o SLA). Sin embargo, la monitorización y el mantenimiento de un nivel satisfactorio de la calidad de experiencia (en inglés Quality of Experience o QoE) del vídeo online se ha convertido en un reto mayor que nunca para los operadores. Dada la creciente popularidad del consumo de contenido con dispositivos móviles y a través de redes inalámbricas, han aparecido nuevos puntos de fallo que se han añadido a la lista de problemas que pueden afectar a la calidad del vídeo transmitido. Adicionalmente, la adopción de sistemas de encriptación extremo a extremo, por parte de los servicios más importantes de distribución de vídeo online, ha dejado obsoletos los métodos existentes de diagnóstico de la QoE. En esta tesis se identifican los retos actuales en la identificación y diagnóstico de los problemas de transmisión de vídeo online, y se proponen nuevas soluciones para abordar estos problemas. Más concretamente, inicialmente la tesis presenta métodos y herramientas para identificar un conjunto amplio de problemas de QoE y la severidad con los que estos afectan a la experiencia de los usuarios. La siguiente parte de la tesis investiga métodos para localizar partes de la red con un rendimiento bajo que resultan en una disminución de la calidad del servicio ofrecido. En este contexto, se propone una metodología basada en el análisis de datos para detectar áreas de la red móvil que ofrecen un nivel subóptimo de calidad de servicio (en inglés Quality of Service o QoS) y QoE. Además, se desarrolla y se evalúa una solución basada en múltiples puntos de medida que es capaz de diagnosticar los problemas subyacentes que causan la alteración de la experiencia de usuario. La última parte de este trabajo explora adicionalmente la detección de anomalías de rendimiento de la red y presenta un nuevo método para detectar estas situaciones utilizando información contextual. Este enfoque proporciona una mayor precisión en la detección de fallos de la red en presencia de alta variabilidad y puede ayudar a los proveedores a la detección precoz de anomalías antes de que se conviertan en problemas de QoE.La distribució de vídeo online és un ecosistema de serveis i tecnologies, on els proveïdors de continguts es troben en una cursa continua per satisfer les demandes creixents del usuaris de més riquesa de contingut, velocitat de transmissió, funcionalitat i compatibilitat entre diferents plataformes. A la vegada, els operadors de xarxa han d’assegurar que els continguts demandats són entregats a través de la xarxa amb una qualitat satisfactòria segons els acords existents de nivell de servei (en anglès Service Level Agreement o SLA). Tanmateix, el monitoratge i el manteniment d’un nivell satisfactori de la qualitat d’experiència (en anglès Quality of Experience o QoE) del vídeo online ha esdevingut un repte més gran que mai per als operadors. Donada la creixent popularitat del consum de contingut amb dispositius mòbils i a través de xarxes sense fils, han aparegut nous punts de fallada que s’han afegit a la llista de problemes que poden afectar a la qualitat del vídeo transmès. Addicionalment, l’adopció de sistemes d’encriptació extrem a extrem, per part dels serveis més importants de distribució de vídeo online, ha deixat obsolets els mètodes existents de diagnòstic de la QoE. En aquesta tesi s’identifiquen els reptes actuals en la identificació i diagnòstic dels problemes de transmissió de vídeo online, i es proposen noves solucions per abordar aquests problemes. Més concretament, inicialment la tesi presenta mètodes i eines per identificar un conjunt ampli de problemes de QoE i la severitat amb la que aquests afecten a la experiència dels usuaris. La següent part de la tesi investiga mètodes per localitzar parts de la xarxa amb un rendiment baix que resulten en una disminució de la qualitat del servei ofert. En aquest context es proposa una metodologia basada en l’anàlisi de dades per detectar àrees de la xarxa mòbil que ofereixen un nivell subòptim de qualitat de servei (en anglès Quality of Service o QoS) i QoE. A més, es desenvolupa i s’avalua una solució basada en múltiples punts de mesura que és capaç de diagnosticar els problemes subjacents que causen l’alteració de l’experiència d’usuari. L’última part d’aquest treball explora addicionalment la detecció d’anomalies de rendiment de la xarxa i presenta un nou mètode per detectar aquestes situacions utilitzant informació contextual. Aquest enfoc proporciona una major precisió en la detecció de fallades de la xarxa en presencia d’alta variabilitat i pot ajudar als proveïdors a la detecció precoç d’anomalies abans de que es converteixin en problemes de QoE.Postprint (published version