Mobile Video Streaming Applications: A Systematic Review of Test Metrics in Usability Evaluation

In evaluating the usability of mobile video streaming applications, the performance of the applications comes into focus. This is because the performance of mobile streaming applications affects their usability. From this study, video streaming and video quality are identified as the two most evaluated elements in the usability test of mobile video streaming applications. These elements are affected by several related factors that are peculiar to the mobile platforms and domains. These in turn affect the usability of the applications. In mobile platforms, bandwidth is low and network connections are unstable; this is coupled with the limitations caused by the smallness of the screen sizes of the mobile devices. Furthermore, startup delays, jitter, latency and rebuffering are the determining factors for the performance of mobile video streaming. On the other hand, video quality is determined by frame rate, bit rate, and resolution. These factors present themselves due to the mobile context of mobile streaming applications. They combine to influence the performance of the applications as well as their usability. Therefore, in considering the usability of these set of applications, these factors (metrics) are important as they determine the performance of the applications and by and large also affect the usability of the applications. Other factors identified in the study that affect the usability of mobile streaming applications include: functionality, social context and user interface and appearance. On the whole, this paper presents the results of a systematic review of test metrics in the usability evaluation of mobile video streaming applications. The systematic review approach used include: defining the search strategy, selection of primary studies, the extraction of data, and the implementation of a synthesis strategy. Using this methodology, 238 studies were found; however, only 51 relevant studies were eventually selected for the review. The study reveals that time taken for video streaming and the video quality were the two most popular metrics used in the usability test and evaluation of mobile video streaming applications. Besides, most of the studies concentrated on the usability of mobile TV as users switch from traditional TV to mobile TV

Streaming Video QoE Modeling and Prediction: A Long Short-Term Memory Approach

HTTP based adaptive video streaming has become a popular choice of streaming due to the reliable transmission and the flexibility offered to adapt to varying network conditions. However, due to rate adaptation in adaptive streaming, the quality of the videos at the client keeps varying with time depending on the end-to-end network conditions. Further, varying network conditions can lead to the video client running out of playback content resulting in rebuffering events. These factors affect the user satisfaction and cause degradation of the user quality of experience (QoE). It is important to quantify the perceptual QoE of the streaming video users and monitor the same in a continuous manner so that the QoE degradation can be minimized. However, the continuous evaluation of QoE is challenging as it is determined by complex dynamic interactions among the QoE influencing factors. Towards this end, we present LSTM-QoE, a recurrent neural network based QoE prediction model using a Long Short-Term Memory (LSTM) network. The LSTM-QoE is a network of cascaded LSTM blocks to capture the nonlinearities and the complex temporal dependencies involved in the time varying QoE. Based on an evaluation over several publicly available continuous QoE databases, we demonstrate that the LSTM-QoE has the capability to model the QoE dynamics effectively. We compare the proposed model with the state-of-the-art QoE prediction models and show that it provides superior performance across these databases. Further, we discuss the state space perspective for the LSTM-QoE and show the efficacy of the state space modeling approaches for QoE prediction

Mobile video streaming applications: a systematic review of test metrics in usability evaluation

In evaluating the usability of mobile video streaming applications, the performance of the applications comes into focus.This is because the performance of mobile streaming applications affects their usability. From this study, video streaming and video quality are identified as the two most evaluated elements in the usability test of mobile video streaming applications. These elements are affected by several related factors that are peculiar to the mobile platforms and domains. These in turn affect the usability of the applications. In mobile platforms, bandwidth is low and network connections are unstable; this is coupled with the limitations caused by the smallness of the screen sizes of the mobile devices. Furthermore, startup delays, jitter, latency and rebuffering are the determining factors for the performance of mobile video streaming. On the other hand, video quality is determined by frame rate, bit rate, and resolution. These factors present themselves due to the mobile context of mobile streaming applications. They combine to influence the performance of the applications as well as their usability. Therefore, in considering the usability of these set of applications, these factors (metrics) are important as they determine the performance of the applications and by and large also affect the usability of the applications. Other factors identified in the study that affect the usability of mobile streaming applications include: functionality, social context and user interface and appearance. On the whole, this paper presents the results of a systematic review of test metrics in the usability evaluation of mobile video streaming applications. The systematic review approach used include: defining the search strategy, selection of primary studies, the extraction of data, and the implementation of a synthesis strategy. Using this methodology, 238 studies were found; however, only 51 relevant studies were eventually selected for the review. The study reveals that time taken for video streaming and the video quality were the two most popular metrics used in the usability test and evaluation of mobile video streaming applications. Besides, most of the studies concentrated on the usability of mobile TV as users switch from traditional TV to mobile TV

Implementação de uma métrica de qualidade de vídeo em dispositivos móveis que consideram degradações no domínio do tempo e espaço / Video quality metric implementation in mobile devices that considers impairments in the time and spatial domain

O volume de tráfego do serviço de streaming de vídeo tem incrementado consideravelmente nos últimos anos, devido ao sucesso de distribuidores de conteúdo como YouTube e Netflix. Porém, limitações na capacidade e instabilidade de uma rede impactam na qualidade de experiência (QoE) do usuário. Neste trabalho, propõe-se um modelo de avaliação de qualidade do streaming de vídeo, e a sua implementação em dispositivos moveis. O modelo proposto considera as degradações espaciais dos quadros do vídeo e as interrupções temporais. Os resultados experimentais apresentam o impacto dos fatores de degradação na QoE; destacando-se que o modelo proposto alcançou uma alta correlação com testes subjetivos. Adicionalmente, a implementação no dispositivo móvel é de baixo consumo na capacidade de processamento e de energia

Subjective and Objective Quality-of-Experience of Adaptive Video Streaming

With the rapid growth of streaming media applications, there has been a strong demand of Quality-of-Experience (QoE) measurement and QoE-driven video delivery technologies. While the new worldwide standard dynamic adaptive streaming over hypertext transfer protocol (DASH) provides an inter-operable solution to overcome the volatile network conditions, its complex characteristic brings new challenges to the objective video QoE measurement models. How streaming activities such as stalling and bitrate switching events affect QoE is still an open question, and is hardly taken into consideration in the traditionally QoE models. More importantly, with an increasing number of objective QoE models proposed, it is important to evaluate the performance of these algorithms in a comparative setting and analyze the strengths and weaknesses of these methods. In this study, we build two subject-rated streaming video databases. The progressive streaming video database is dedicated to investigate the human responses to the combined effect of video compression, initial buffering, and stalling. The adaptive streaming video database is designed to evaluate the performance of adaptive bitrate streaming algorithms and objective QoE models. We also provide useful insights on the improvement of adaptive bitrate streaming algorithms. Furthermore, we propose a novel QoE prediction approach to account for the instantaneous quality degradation due to perceptual video presentation impairment, the playback stalling events, and the instantaneous interactions between them. Twelve QoE algorithms from four categories including signal fidelity-based, network QoS-based, application QoS-based, and hybrid QoE models are assessed in terms of correlation with human perception on the two streaming video databases. Experimental results show that the proposed model is in close agreement with subjective opinions and significantly outperforms traditional QoE models

Perceptual video quality and quality of experience for adaptive video streaming

We live in a world where images and videos dominate our everyday lives. Every day, an enormous amount of video data is being shared in social media and consumer applications, while video streaming is becoming a new form of digital entertainment. Large-scale video streaming on demand has become possible thanks to numerous engineering achievements in fields such as video compression, high-speed computation and display technologies. Nevertheless, the skyrocketing needs for bandwidth and network resources consumed by video applications challenges modern video content delivery. Since the available bandwidth resources are limited, streaming service providers have to mediate between operation costs, bandwidth efficiency and maximizing user quality of experience. However, these goals are inherently conflicting and require knowledge of how user quality of experience is affected by the network-induced changes in video quality. Being able to understand and predict user quality of experience and perceptually optimize rate allocation, can have significant effects in better network utilization, reduced costs for service providers and improved user satisfaction. The goal of this dissertation is to study and predict user quality of experience in video streaming applications, by exploiting perceptual video quality and human behavioral responses to streaming-related video impairments. To this end, I present the details of three large-scale video subjective studies which target video streaming under multiple viewing conditions, such as display device, session duration, content characteristics and network/buffer conditions. By analyzing how humans react to changes in visual quality and streaming video impairments, I also design numerous video quality and quality of experience prediction models that can be used to evaluate the overall and the continuous-time perceived video quality. Throughout this dissertation, my goal is to perceptually optimize various stages of the video streaming pipeline, such as video encoding and video quality control as well as client-based rate adaptation. Ultimately, I envision that the outcome of this dissertation can be useful for video streaming applications at global scaleElectrical and Computer Engineerin