Optimized Data Representation for Interactive Multiview Navigation

In contrary to traditional media streaming services where a unique media content is delivered to different users, interactive multiview navigation applications enable users to choose their own viewpoints and freely navigate in a 3-D scene. The interactivity brings new challenges in addition to the classical rate-distortion trade-off, which considers only the compression performance and viewing quality. On the one hand, interactivity necessitates sufficient viewpoints for richer navigation; on the other hand, it requires to provide low bandwidth and delay costs for smooth navigation during view transitions. In this paper, we formally describe the novel trade-offs posed by the navigation interactivity and classical rate-distortion criterion. Based on an original formulation, we look for the optimal design of the data representation by introducing novel rate and distortion models and practical solving algorithms. Experiments show that the proposed data representation method outperforms the baseline solution by providing lower resource consumptions and higher visual quality in all navigation configurations, which certainly confirms the potential of the proposed data representation in practical interactive navigation systems

Optimized Packet Scheduling in Multiview Video Navigation Systems

In multiview video systems, multiple cameras generally acquire the same scene from different perspectives, such that users have the possibility to select their preferred viewpoint. This results in large amounts of highly redundant data, which needs to be properly handled during encoding and transmission over resource-constrained channels. In this work, we study coding and transmission strategies in multicamera systems, where correlated sources send data through a bottleneck channel to a central server, which eventually transmits views to different interactive users. We propose a dynamic correlation-aware packet scheduling optimization under delay, bandwidth, and interactivity constraints. The optimization relies both on a novel rate-distortion model, which captures the importance of each view in the 3D scene reconstruction, and on an objective function that optimizes resources based on a client navigation model. The latter takes into account the distortion experienced by interactive clients as well as the distortion variations that might be observed by clients during multiview navigation. We solve the scheduling problem with a novel trellis-based solution, which permits to formally decompose the multivariate optimization problem thereby significantly reducing the computation complexity. Simulation results show the gain of the proposed algorithm compared to baseline scheduling policies. More in details, we show the gain offered by our dynamic scheduling policy compared to static camera allocation strategies and to schemes with constant coding strategies. Finally, we show that the best scheduling policy consistently adapts to the most likely user navigation path and that it minimizes distortion variations that can be very disturbing for users in traditional navigation systems

Optimal layered representation for adaptive interactive multiview video streaming

We consider an interactive multiview video streaming (IMVS) system where clients select their preferred viewpoint in a given navigation window. To provide high quality IMVS, many high quality views should be transmitted to the clients. However, this is not always possible due to the limited and heterogeneous capabilities of the clients. In this paper, we propose a novel adaptive IMVS solution based on a layered multiview representation where camera views are organized into layered subsets to match the different clients constraints. We formulate an optimization problem for the joint selection of the views subsets and their encoding rates. Then, we propose an optimal and a reduced computational complexity greedy algorithms, both based on dynamic-programming. Simulation results show the good performance of our novel algorithms compared to a baseline algorithm, proving that an effective IMVS adaptive solution should consider the scene content and the client capabilities and their preferences in navigation

Stochastic user behaviour modelling and network simulation for resource management in cooperation with mobile telecommunications and broadcast networks

The latest generations of telecommunications networks have been designed to deliver higher data rates than widely used second generation telecommunications networks, providing flexible communication capabilities that can deliver high quality video images. However, these new generations of telecommunications networks are interference limited, impairing their performance in cases of heavy traffic and high usage. This limits the services offered by a telecommunications network operator to those that the operator is confident their network can meet the demand for. One way to lift this constraint would be for the mobile telecommunications network operator to obtain the cooperation of a broadcast network operator so that during periods when the demand for the service is too high for the telecommunications network to meet, the service can be transferred to the broadcast network. In the United Kingdom the most recent telecommunications networks on the market are third generation UMTS networks while the terrestrial digital broadcast networks are DVB-T networks. This paper proposes a way for UMTS network operators to forecast the traffic associated with high demand services intended to be deployed on the UMTS network and when demand requires to transfer it to a cooperating DVB-T network. The paper aims to justify to UMTS network operators the use of a DVB-T network as a support for a UMTS network by clearly showing how using a DVB-T network to support it can increase the revenue generated by their network

QoE-Aware Resource Allocation For Crowdsourced Live Streaming: A Machine Learning Approach

In the last decade, empowered by the technological advancements of mobile devices and the revolution of wireless mobile network access, the world has witnessed an explosion in crowdsourced live streaming. Ensuring a stable high-quality playback experience is compulsory to maximize the viewers’ Quality of Experience and the content providers’ profits. This can be achieved by advocating a geo-distributed cloud infrastructure to allocate the multimedia resources as close as possible to viewers, in order to minimize the access delay and video stalls. Additionally, because of the instability of network condition and the heterogeneity of the end-users capabilities, transcoding the original video into multiple bitrates is required. Video transcoding is a computationally expensive process, where generally a single cloud instance needs to be reserved to produce one single video bitrate representation. On demand renting of resources or inadequate resources reservation may cause delay of the video playback or serving the viewers with a lower quality. On the other hand, if resources provisioning is much higher than the required, the extra resources will be wasted. In this thesis, we introduce a prediction-driven resource allocation framework, to maximize the QoE of viewers and minimize the resources allocation cost. First, by exploiting the viewers’ locations available in our unique dataset, we implement a machine learning model to predict the viewers’ number near each geo-distributed cloud site. Second, based on the predicted results that showed to be close to the actual values, we formulate an optimization problem to proactively allocate resources at the viewers’ proximity. Additionally, we will present a trade-off between the video access delay and the cost of resource allocation. Considering the complexity and infeasibility of our offline optimization to respond to the volume of viewing requests in real-time, we further extend our work, by introducing a resources forecasting and reservation framework for geo-distributed cloud sites. First, we formulate an offline optimization problem to allocate transcoding resources at the viewers’ proximity, while creating a tradeoff between the network cost and viewers QoE. Second, based on the optimizer resource allocation decisions on historical live videos, we create our time series datasets containing historical records of the optimal resources needed at each geo-distributed cloud site. Finally, we adopt machine learning to build our distributed time series forecasting models to proactively forecast the exact needed transcoding resources ahead of time at each geo-distributed cloud site. The results showed that the predicted number of transcoding resources needed in each cloud site is close to the optimal number of transcoding resources

Interactive multiview video system with low decoding complexity

Research in multimedia is always investigating new ways of improving the immersive experience of the users. One current solution consists in designing systems which offer a high level of interactivity, such as multiview content navigation where the point of view can be changed while watching at a video sequence (e.g., free viewpoint television, gaming, etc.). The coding algorithm designed for the transmission of such media streams must be adapted to these novel decoder needs. However, video plus depth data transmission is usually performed by considering the information flows as two sequences encoded with MVC schemes. Whereas it achieves good compression performance, this coding approach is not appropriate for interactive applications since the decoding of a frame often requires the prior transmission and decoding of several reference frames. Moreover, the techniques recently developed to improve interactivity are generally implemented at the decoder, whose computational complexity requirements are augmented. In this paper, we propose a novel coding scheme for video plus depth sequences that is adapted to user navigation; contrarily to several common approaches, the additional complexity is added on the encoder side so that the decoder stays simple. We further propose to limit the additional bandwidth imposed by interactivity requirements by designing a rate allocation algorithm that builds on a model of the user behavior. A first version of our novel coding architecture is evaluated in terms of rate-distortion performance, where it is shown to offer a high interactivity at a reasonable bandwidth cost