Design and optimisation of a (FA)Q-learning-based HTTP adaptive streaming client

In recent years, HTTP (Hypertext Transfer Protocol) adaptive streaming (HAS) has become the de facto standard for adaptive video streaming services. A HAS video consists of multiple segments, encoded at multiple quality levels. State-of-the-art HAS clients employ deterministic heuristics to dynamically adapt the requested quality level based on the perceived network conditions. Current HAS client heuristics are, however, hardwired to fit specific network configurations, making them less flexible to fit a vast range of settings. In this article, a (frequency adjusted) Q-learning HAS client is proposed. In contrast to existing heuristics, the proposed HAS client dynamically learns the optimal behaviour corresponding to the current network environment in order to optimise the quality of experience. Furthermore, the client has been optimised both in terms of global performance and convergence speed. Thorough evaluations show that the proposed client can outperform deterministic algorithms by 11-18% in terms of mean opinion score in a wide range of network configurations

Evaluation of Q-Learning approach for HTTP adaptive streaming

We propose a Q-Learning-based algorithm for an HTTP Adaptive Streaming (HAS) Client that maximizes the perceived quality, taking into account the relation between the estimated bandwidth and the qualities and penalizing the freezes. The results will show that it produces an optimal control as other approaches do, but keeping the adaptivenes

A learning-based algorithm for improved bandwidth-awareness of adaptive streaming clients

HTTP Adaptive Streaming (HAS) is becoming the de-facto standard for Over-The-Top video streaming. A HAS video consists of multiple segments, encoded at multiple quality levels. Allowing the client to select the quality level for every segment, a smoother playback and a higher Quality of Experience (QoE) can be perceived. Although results are promising, current quality selection heuristics are generally hard coded. Fixed parameter values are used to provide an acceptable QoE under all circumstances, resulting in suboptimal solutions. Furthermore, many commercial HAS implementations focus on a video-on-demand scenario, where a large buffer size is used to avoid playout freezes. When the focus is on a live TV scenario however, a low buffer size is typically preferred, as the video play-out delay should be as low as possible. Hard coded implementations using a fixed buffer size are not capable of dealing with both scenarios. In this paper, the concept of reinforcement learning is introduced at client side, allowing to adaptively change the parameter configuration for existing rate adaptation heuristics. Bandwidth characteristics are taken into account in the decision process, thus allowing to improve the client's bandwidth-awareness. Focus in this paper is on actively reducing the average buffer filling, evaluating results for two heuristics: the Microsoft IIS Smooth Streaming heuristic and the QoE-driven Rate Adaptation Heuristic for Adaptive video Streaming by Petrangeli et al. We show that using the proposed learning-based approach, the average buffer filling can be reduced by 8.3% compared to state of the art, while achieving a comparable level of QoE

Federated Deep Reinforcement Learning-based Bitrate Adaptation for Dynamic Adaptive Streaming over HTTP

In video streaming over HTTP, the bitrate adaptation selects the quality of video chunks depending on the current network condition. Some previous works have applied deep reinforcement learning (DRL) algorithms to determine the chunk's bitrate from the observed states to maximize the quality-of-experience (QoE). However, to build an intelligent model that can predict in various environments, such as 3G, 4G, Wifi, \textit{etc.}, the states observed from these environments must be sent to a server for training centrally. In this work, we integrate federated learning (FL) to DRL-based rate adaptation to train a model appropriate for different environments. The clients in the proposed framework train their model locally and only update the weights to the server. The simulations show that our federated DRL-based rate adaptations, called FDRLABR with different DRL algorithms, such as deep Q-learning, advantage actor-critic, and proximal policy optimization, yield better performance than the traditional bitrate adaptation methods in various environments.Comment: 13 pages, 1 colum

Seamless multimedia delivery within a heterogeneous wireless networks environment: are we there yet?

The increasing popularity of live video streaming from mobile devices such as Facebook Live, Instagram Stories, Snapchat, etc. pressurises the network operators to increase the capacity of their networks. However, a simple increase in system capacity will not be enough without considering the provisioning of Quality of Experience (QoE) as the basis for network control, customer loyalty and retention rate and thus increase in network operators revenue. As QoE is gaining strong momentum especially with increasing users’ quality expectations, the focus is now on proposing innovative solutions to enable QoE when delivering video content over heterogeneous wireless networks. In this context, this paper presents an overview of multimedia delivery solutions, identifies the problems and provides a comprehensive classification of related state-of-the-art approaches following three key directions: adaptation, energy efficiency and multipath content delivery. Discussions, challenges and open issues on the seamless multimedia provisioning faced by the current and next generation of wireless networks are also provided

Seamless Multimedia Delivery Within a Heterogeneous Wireless Networks Environment: Are We There Yet?

The increasing popularity of live video streaming from mobile devices, such as Facebook Live, Instagram Stories, Snapchat, etc. pressurizes the network operators to increase the capacity of their networks. However, a simple increase in system capacity will not be enough without considering the provisioning of quality of experience (QoE) as the basis for network control, customer loyalty, and retention rate and thus increase in network operators revenue. As QoE is gaining strong momentum especially with increasing users' quality expectations, the focus is now on proposing innovative solutions to enable QoE when delivering video content over heterogeneous wireless networks. In this context, this paper presents an overview of multimedia delivery solutions, identifies the problems and provides a comprehensive classification of related state-of-the-art approaches following three key directions: 1) adaptation; 2) energy efficiency; and 3) multipath content delivery. Discussions, challenges, and open issues on the seamless multimedia provisioning faced by the current and next generation of wireless networks are also provided

Deep Reinforcement Learning with Importance Weighted A3C for QoE enhancement in Video Delivery Services

Adaptive bitrate (ABR) algorithms are used to adapt the video bitrate based on the network conditions to improve the overall video quality of experience (QoE). Recently, reinforcement learning (RL) and asynchronous advantage actor-critic (A3C) methods have been used to generate adaptive bit rate algorithms and they have been shown to improve the overall QoE as compared to fixed rule ABR algorithms. However, a common issue in the A3C methods is the lag between behaviour policy and target policy. As a result, the behaviour and the target policies are no longer synchronized which results in suboptimal updates. In this work, we present ALISA: An Actor-Learner Architecture with Importance Sampling for efficient learning in ABR algorithms. ALISA incorporates importance sampling weights to give more weightage to relevant experience to address the lag issues with the existing A3C methods. We present the design and implementation of ALISA, and compare its performance to state-of-the-art video rate adaptation algorithms including vanilla A3C implemented in the Pensieve framework and other fixed-rule schedulers like BB, BOLA, and RB. Our results show that ALISA improves average QoE by up to 25%-48% higher average QoE than Pensieve, and even more when compared to fixed-rule schedulers.Comment: Number of pages: 10, Number of figures: 9, Conference name: 24th IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM

Peer to Peer Information Retrieval: An Overview

Peer-to-peer technology is widely used for file sharing. In the past decade a number of prototype peer-to-peer information retrieval systems have been developed. Unfortunately, none of these have seen widespread real- world adoption and thus, in contrast with file sharing, information retrieval is still dominated by centralised solutions. In this paper we provide an overview of the key challenges for peer-to-peer information retrieval and the work done so far. We want to stimulate and inspire further research to overcome these challenges. This will open the door to the development and large-scale deployment of real-world peer-to-peer information retrieval systems that rival existing centralised client-server solutions in terms of scalability, performance, user satisfaction and freedom