DASHbed: a testbed framework for large scale empirical evaluation of real-time DASH in wireless scenarios

Recent years have witnessed an explosion of multimedia traffic carried over the Internet. Video-on-demand and live streaming services are the most dominant services. To ensure growth, many streaming providers have invested considerable time and effort to keep pace with ever-increasing users’ demand for better quality and stall abolition. HTTP adaptive streaming (HAS) algorithms are at the core of every major streaming provider service. Recent years have seen sustained development in HAS algorithms. Currently, to evaluate their proposed solutions, researchers need to create a framework and numerous state-of-the-art algorithms. Often, these frameworks lack flexibility and scalability, covering only a limited set of scenarios. To fill this gap, in this paper we propose DASHbed, a highly customizable real-time framework for testing HAS algorithms in a wireless environment. Due to its low memory requirement, DASHbed offers a means of running large-scale experiments with a hundred competing players. Finally, we supplement the proposed framework with a dataset consisting of results for five HAS algorithms tested in various evaluated scenarios. The dataset showcases the abilities of DASHbed and presents the adaptation metrics per segment in the generated content (such as switches, buffer-level, P.1203.1 values, delivery rate, stall duration, etc.), which can be used as a baseline when researchers compare the output of their proposed algorithm against the state-of-the-art algorithms

Packet loss visibility across SD, HD, 3D, and UHD video streams

The trend towards video streaming with increased spatial resolutions and dimensions, SD, HD, 3D, and 4kUHD, even for portable devices has important implications for displayed video quality. There is an interplay between packetization, packet loss visibility, choice of codec, and viewing conditions, which implies that prior studies at lower resolutions may not be as relevant. This paper presents two sets of experiments, the one at a Variable BitRate (VBR) and the other at a Constant BitRate (CBR), which highlight different aspects of the interpretation. The latter experiments also compare and contrast encoding with either an H.264 or an High Efficiency Video Coding (HEVC) codec, with all results recorded as objective Mean Opinion Score (MOS). The video quality assessments will be of interest to those considering: the bitrates and expected quality in error-prone environments; or, in fact, whether to use a reliable transport protocol to prevent all errors, at a cost in jitter and latency, rather than tolerate low levels of packet errors

goDASH - GO accelerated HAS framework for rapid prototyping

In this short paper, we present goDASH, an infrastructure for headless streaming of HTTP adaptive streaming (HAS) video content, implemented in the language golang, an open-source programming language supported by Google. goDASHâ s main functionality is the ability to stream HAS content without decoding actual video (headless player). This results in low memory requirements and the ability to run multiple players in a large-scale-based evaluation setup. goDASH comes complete with numerous state-of-the-art HAS algorithms, and is fully written in the Google golang language, which simplifies the implementation of new adaptation algorithms and functions. goDASH supports two transportation protocols Transmission Control Protocol (TCP) and Quick UDP Internet Connections (QUIC). The QUIC protocol is a relatively new protocol with the promise of performance improvement over the widely used TCP. We believe that goDASH is the first emulation-based HAS player that supports QUIC. The main limitation in using QUIC protocol is the need for a security certificate setup on both ends (client and server) as QUIC demands an encrypted connection. This limitation is eased by providing our own testbed framework, known as goDASHbed. This framework uses a virtual environment to serve video content locally (which allows setting security certificates) through the Mininet virtual emulation tool. As part of Mininet, goDASH can be used in conjunction with other traffic generators

Data-driven visual quality estimation using machine learning

Heutzutage werden viele visuelle Inhalte erstellt und sind zugänglich, was auf Verbesserungen der Technologie wie Smartphones und das Internet zurückzuführen ist. Es ist daher notwendig, die von den Nutzern wahrgenommene Qualität zu bewerten, um das Erlebnis weiter zu verbessern. Allerdings sind nur wenige der aktuellen Qualitätsmodelle speziell für höhere Auflösungen konzipiert, sagen mehr als nur den Mean Opinion Score vorher oder nutzen maschinelles Lernen. Ein Ziel dieser Arbeit ist es, solche maschinellen Modelle für höhere Auflösungen mit verschiedenen Datensätzen zu trainieren und zu evaluieren. Als Erstes wird eine objektive Analyse der Bildqualität bei höheren Auflösungen durchgeführt. Die Bilder wurden mit Video-Encodern komprimiert, hierbei weist AV1 die beste Qualität und Kompression auf. Anschließend werden die Ergebnisse eines Crowd-Sourcing-Tests mit einem Labortest bezüglich Bildqualität verglichen. Weiterhin werden auf Deep Learning basierende Modelle für die Vorhersage von Bild- und Videoqualität beschrieben. Das auf Deep Learning basierende Modell ist aufgrund der benötigten Ressourcen für die Vorhersage der Videoqualität in der Praxis nicht anwendbar. Aus diesem Grund werden pixelbasierte Videoqualitätsmodelle vorgeschlagen und ausgewertet, die aussagekräftige Features verwenden, welche Bild- und Bewegungsaspekte abdecken. Diese Modelle können zur Vorhersage von Mean Opinion Scores für Videos oder sogar für anderer Werte im Zusammenhang mit der Videoqualität verwendet werden, wie z.B. einer Bewertungsverteilung. Die vorgestellte Modellarchitektur kann auf andere Videoprobleme angewandt werden, wie z.B. Videoklassifizierung, Vorhersage der Qualität von Spielevideos, Klassifikation von Spielegenres oder der Klassifikation von Kodierungsparametern. Ein wichtiger Aspekt ist auch die Verarbeitungszeit solcher Modelle. Daher wird ein allgemeiner Ansatz zur Beschleunigung von State-of-the-Art-Videoqualitätsmodellen vorgestellt, der zeigt, dass ein erheblicher Teil der Verarbeitungszeit eingespart werden kann, während eine ähnliche Vorhersagegenauigkeit erhalten bleibt. Die Modelle sind als Open Source veröffentlicht, so dass die entwickelten Frameworks für weitere Forschungsarbeiten genutzt werden können. Außerdem können die vorgestellten Ansätze als Bausteine für neuere Medienformate verwendet werden.Today a lot of visual content is accessible and produced, due to improvements in technology such as smartphones and the internet. This results in a need to assess the quality perceived by users to further improve the experience. However, only a few of the state-of-the-art quality models are specifically designed for higher resolutions, predict more than mean opinion score, or use machine learning. One goal of the thesis is to train and evaluate such machine learning models of higher resolutions with several datasets. At first, an objective evaluation of image quality in case of higher resolutions is performed. The images are compressed using video encoders, and it is shown that AV1 is best considering quality and compression. This evaluation is followed by the analysis of a crowdsourcing test in comparison with a lab test investigating image quality. Afterward, deep learning-based models for image quality prediction and an extension for video quality are proposed. However, the deep learning-based video quality model is not practically usable because of performance constrains. For this reason, pixel-based video quality models using well-motivated features covering image and motion aspects are proposed and evaluated. These models can be used to predict mean opinion scores for videos, or even to predict other video quality-related information, such as a rating distributions. The introduced model architecture can be applied to other video problems, such as video classification, gaming video quality prediction, gaming genre classification or encoding parameter estimation. Furthermore, one important aspect is the processing time of such models. Hence, a generic approach to speed up state-of-the-art video quality models is introduced, which shows that a significant amount of processing time can be saved, while achieving similar prediction accuracy. The models have been made publicly available as open source so that the developed frameworks can be used for further research. Moreover, the presented approaches may be usable as building blocks for newer media formats

Bitstream-based video quality modeling and analysis of HTTP-based adaptive streaming

Die Verbreitung erschwinglicher Videoaufnahmetechnologie und verbesserte Internetbandbreiten ermöglichen das Streaming von hochwertigen Videos (Auflösungen > 1080p, Bildwiederholraten ≥ 60fps) online. HTTP-basiertes adaptives Streaming ist die bevorzugte Methode zum Streamen von Videos, bei der Videoparameter an die verfügbare Bandbreite angepasst wird, was sich auf die Videoqualität auswirkt. Adaptives Streaming reduziert Videowiedergabeunterbrechnungen aufgrund geringer Netzwerkbandbreite, wirken sich jedoch auf die wahrgenommene Qualität aus, weswegen eine systematische Bewertung dieser notwendig ist. Diese Bewertung erfolgt üblicherweise für kurze Abschnitte von wenige Sekunden und während einer Sitzung (bis zu mehreren Minuten). Diese Arbeit untersucht beide Aspekte mithilfe perzeptiver und instrumenteller Methoden. Die perzeptive Bewertung der kurzfristigen Videoqualität umfasst eine Reihe von Labortests, die in frei verfügbaren Datensätzen publiziert wurden. Die Qualität von längeren Sitzungen wurde in Labortests mit menschlichen Betrachtern bewertet, die reale Betrachtungsszenarien simulieren. Die Methodik wurde zusätzlich außerhalb des Labors für die Bewertung der kurzfristigen Videoqualität und der Gesamtqualität untersucht, um alternative Ansätze für die perzeptive Qualitätsbewertung zu erforschen. Die instrumentelle Qualitätsevaluierung wurde anhand von bitstrom- und hybriden pixelbasierten Videoqualitätsmodellen durchgeführt, die im Zuge dieser Arbeit entwickelt wurden. Dazu wurde die Modellreihe AVQBits entwickelt, die auf den Labortestergebnissen basieren. Es wurden vier verschiedene Modellvarianten von AVQBits mit verschiedenen Inputinformationen erstellt: Mode 3, Mode 1, Mode 0 und Hybrid Mode 0. Die Modellvarianten wurden untersucht und schneiden besser oder gleichwertig zu anderen aktuellen Modellen ab. Diese Modelle wurden auch auf 360°- und Gaming-Videos, HFR-Inhalte und Bilder angewendet. Darüber hinaus wird ein Langzeitintegrationsmodell (1 - 5 Minuten) auf der Grundlage des ITU-T-P.1203.3-Modells präsentiert, das die verschiedenen Varianten von AVQBits mit sekündigen Qualitätswerten als Videoqualitätskomponente des vorgeschlagenen Langzeitintegrationsmodells verwendet. Alle AVQBits-Varianten, das Langzeitintegrationsmodul und die perzeptiven Testdaten wurden frei zugänglich gemacht, um weitere Forschung zu ermöglichen.The pervasion of affordable capture technology and increased internet bandwidth allows high-quality videos (resolutions > 1080p, framerates ≥ 60fps) to be streamed online. HTTP-based adaptive streaming is the preferred method for streaming videos, adjusting video quality based on available bandwidth. Although adaptive streaming reduces the occurrences of video playout being stopped (called “stalling”) due to narrow network bandwidth, the automatic adaptation has an impact on the quality perceived by the user, which results in the need to systematically assess the perceived quality. Such an evaluation is usually done on a short-term (few seconds) and overall session basis (up to several minutes). In this thesis, both these aspects are assessed using subjective and instrumental methods. The subjective assessment of short-term video quality consists of a series of lab-based video quality tests that have resulted in publicly available datasets. The overall integral quality was subjectively assessed in lab tests with human viewers mimicking a real-life viewing scenario. In addition to the lab tests, the out-of-the-lab test method was investigated for both short-term video quality and overall session quality assessment to explore the possibility of alternative approaches for subjective quality assessment. The instrumental method of quality evaluation was addressed in terms of bitstream- and hybrid pixel-based video quality models developed as part of this thesis. For this, a family of models, namely AVQBits has been conceived using the results of the lab tests as ground truth. Based on the available input information, four different instances of AVQBits, that is, a Mode 3, a Mode 1, a Mode 0, and a Hybrid Mode 0 model are presented. The model instances have been evaluated and they perform better or on par with other state-of-the-art models. These models have further been applied to 360° and gaming videos, HFR content, and images. Also, a long-term integration (1 - 5 mins) model based on the ITU-T P.1203.3 model is presented. In this work, the different instances of AVQBits with the per-1-sec scores output are employed as the video quality component of the proposed long-term integration model. All AVQBits variants as well as the long-term integration module and the subjective test data are made publicly available for further research

SMASH: a Supervised Machine Learning Approach to Adaptive Video Streaming over HTTP.

The growth of online video-on-demand consumption continues unabated. Existing heuristic-based adaptive bitrate (ABR) selection algorithms are typically designed to optimise video quality within a very narrow context. This may lead to video streaming providers implementing different ABR algorithms/players, based on a network connection, device capabilities, video content, etc., in order to serve the multitude of their usersâ streaming requirements. In this paper, we present SMASH: a Supervised Machine learning approach to Adaptive Streaming over HTTP, which takes a tentative step towards the goal of a one-size-fits-all approach to ABR. We utilise the streaming output from the adaptation logic of nine ABR algorithms across a variety of streaming scenarios (generating nearly one million records) and design a machine learning model, using systematically selected features, to predict the optimal choice of the bitrate of the next video segment to download. Our evaluation results show that not only does SMASH guarantee a high QoE but its performance is consistent across a variety of streaming contexts

DI5GUISE: a highly dynamic framework for real-time simulated 5G evaluation

As the next generation of cellular networks (5G) nears deployment, autonomous smart nodes, such as Internet of Things (IoT) and vehicular devices, are already being developed to utilise the increased throughput and improved resiliency of 5G. Evaluating the real-time scheduling and impact of these new devices and their underlying network protocols within the airinterface of 5G is no small feat. This is typically due to high 5G air-interface hardware costs and unrealistic real-time delivery rates. In this short paper, we expand upon the mmwave (5G) module for NS-3 and present DI5GUISE: a configurable and highly dynamic real-time simulated testbed framework, through which the impact of these nodes can be investigated. We utilise realtime video delivery as an example of high throughput demand applications. The experimental results illustrate that even on low cost commodity hardware, such as Raspberry Pis, over 80Mbps per client in real-time streaming of 4K content can achieve

godash 2.0 - the next evolution of HAS evaluation

In this short demo paper, we introduce godash 2.0 godash is a headless HTTP adaptive streaming (HAS) video streaming platform written in the Google programming language GO. godash has been extensively rewritten for this release so as to provide ease of use, and a host of new features. godash includes options for eight different state of the art adaptive algorithms, five HAS profiles, four video codecs, the ability to stream audio and video segments, two transport protocols, real-time output from five Quality of Experience (QoE) models, as well as a collaborative framework for the evaluation of cooperative HAS streaming. godash also comes complete with its own testbed framework known as godashbed. godashbed uses a virtual environment to serve video content locally (which allows setting security certificates) through the Mininet virtual emulation tool. godashbed has options for large scale evaluation of HAS streaming using 4G/5G bandwidth traces, various modes of background traffic, and a choice of web server, namely: Web Server Gateway Interface (WSGI) and Asynchronous Server Gateway Interface (ASGI). In this manner, godash provides a framework for rapid deployment and testing of new HAS algorithms, QoE models and transport protocols

The benefits of deceit: a malicious client in a 5G cellular network

As we advance towards Smart Cities, autonomous vehicles and the avalanche of IoT devices proposed for the future, we need to give careful consideration to how easily compromised nodes/devices can impact network state. Current proposals for autonomous smart devices typically use cellular networks as the backhaul or final hop. These devices will leverage existing trust-based client-side channel metrics, such as channel quality indicator (CQI), when the base-station determines scheduling decisions. In this short paper, we investigate the scheduling impact of a malicious device when it changes its channel metrics, so as to improve its download rate or even to negate the download rate of others. We utilise real-time 4K ultra-high definition video delivery as an example of high throughput demand application and compare the delivery rates of multiple devices in an open-source 5G simulated NS-3 network. Our results illustrate that when a malicious client deceives the scheduler, the other clients in the network have a noticeable decrease in both viewable quality and underlying delivery rate (25% decrease in the average video quality across the non malicious clients)