Implementation of a distributed real-time video panorama pipeline for creating high quality virtual views

Today, we are continuously looking for more immersive video systems. Such systems, however, require more content, which can be costly to produce. A full panorama, covering regions of interest, can contain all the information required, but can be difficult to view in its entirety. In this thesis, we discuss a method for creating virtual views from a cylindrical panorama, allowing multiple users to create individual virtual cameras from the same panorama video. We discuss how this method can be used for video delivery, but emphasize on the creation of the initial panorama. The panorama must be created in real-time, and with very high quality. We design and implement a prototype recording pipeline, installed at a soccer stadium, as a part of the Bagadus project. We describe a pipeline capable of producing 4K panorama videos from five HD cameras, in real-time, with possibilities for further upscaling. We explain how the cylindrical panorama can be created, with minimal computational cost and without visible seams. The cameras of our prototype system record video in the incomplete Bayer format, and we also investigate which debayering algorithms are best suited for recording multiple high resolution video streams in real-time

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Supporting Non-Linear and Non-Continuous Media Access in Peer-to-Peer Multimedia Systems

Ph.DDOCTOR OF PHILOSOPH

Cross-layer Optimization for Video Delivery over Wireless Networks

As video streaming is becoming the most popular application of Internet mo- bile, the design and the optimization of video communications over wireless networks is attracting increasingly attention from both academia and indus- try. The main challenges are to enhance the quality of service support, and to dynamically adapt the transmitted video streams to the network condition. The cross-layer methods, i.e., the exchange of information among different layers of the system, is one of the key concepts to be exploited to achieve this goals. In this thesis we propose novel cross-layer optimization frameworks for scalable video coding (SVC) delivery and for HTTP adaptive streaming (HAS) application over the downlink and the uplink of Long Term Evolution (LTE) wireless networks. They jointly address optimized content-aware rate adaptation and radio resource allocation (RRA) with the aim of maximiz- ing the sum of the achievable rates while minimizing the quality difference among multiple videos. For multi-user SVC delivery over downlink wireless systems, where IP/TV is the most representative application, we decompose the optimization problem and we propose the novel iterative local approxi- mation algorithm to derive the optimal solution, by also presenting optimal algorithms to solve the resulting two sub-problems. For multiple SVC de- livery over uplink wireless systems, where healt-care services are the most attractive and challenging application, we propose joint video adaptation and aggregation directly performed at the application layer of the transmit- ting equipment, which exploits the guaranteed bit-rate (GBR) provided by the low-complexity sub-optimal RRA solutions proposed. Finally, we pro- pose a quality-fair adaptive streaming solution to deliver fair video quality to HAS clients in a LTE cell by adaptively selecting the prescribed (GBR) of each user according to the video content in addition to the channel condi- tion. Extensive numerical evaluations show the significant enhancements of the proposed strategies with respect to other state-of-the-art frameworks

Optimization techniques for computationally expensive rendering algorithms

Realistic rendering in computer graphics simulates the interactions of light and surfaces. While many accurate models for surface reflection and lighting, including solid surfaces and participating media have been described; most of them rely on intensive computation. Common practices such as adding constraints and assumptions can increase performance. However, they may compromise the quality of the resulting images or the variety of phenomena that can be accurately represented. In this thesis, we will focus on rendering methods that require high amounts of computational resources. Our intention is to consider several conceptually different approaches capable of reducing these requirements with only limited implications in the quality of the results. The first part of this work will study rendering of time-­¿varying participating media. Examples of this type of matter are smoke, optically thick gases and any material that, unlike the vacuum, scatters and absorbs the light that travels through it. We will focus on a subset of algorithms that approximate realistic illumination using images of real world scenes. Starting from the traditional ray marching algorithm, we will suggest and implement different optimizations that will allow performing the computation at interactive frame rates. This thesis will also analyze two different aspects of the generation of anti-­¿aliased images. One targeted to the rendering of screen-­¿space anti-­¿aliased images and the reduction of the artifacts generated in rasterized lines and edges. We expect to describe an implementation that, working as a post process, it is efficient enough to be added to existing rendering pipelines with reduced performance impact. A third method will take advantage of the limitations of the human visual system (HVS) to reduce the resources required to render temporally antialiased images. While film and digital cameras naturally produce motion blur, rendering pipelines need to explicitly simulate it. This process is known to be one of the most important burdens for every rendering pipeline. Motivated by this, we plan to run a series of psychophysical experiments targeted at identifying groups of motion-­¿blurred images that are perceptually equivalent. A possible outcome is the proposal of criteria that may lead to reductions of the rendering budgets

Diseño centrado en calidad para la difusión Peer-to-Peer de video en vivo

El uso de redes Peer-to-Peer (P2P) es una forma escalable para ofrecer servicios de video sobre Internet. Este documento hace foco en la definición, desarrollo y evaluación de una arquitectura P2P para distribuir video en vivo. El diseño global de la red es guiado por la calidad de experiencia (Quality of Experience - QoE), cuyo principal componente en este caso es la calidad del video percibida por los usuarios finales, en lugar del tradicional diseño basado en la calidad de servicio (Quality of Service - QoE) de la mayoría de los sistemas. Para medir la calidad percibida del video, en tiempo real y automáticamente, extendimos la recientemente propuesta metodología Pseudo-Subjective Quality Assessment (PSQA). Dos grandes líneas de investigación son desarrolladas. Primero, proponemos una técnica de distribución de video desde múltiples fuentes con las características de poder ser optimizada para maximizar la calidad percibida en contextos de muchas fallas y de poseer muy baja señalización (a diferencia de los sistemas existentes). Desarrollamos una metodología, basada en PSQA, que nos permite un control fino sobre la forma en que la señal de video es dividida en partes y la cantidad de redundancia agregada, como una función de la dinámica de los usuarios de la red. De esta forma es posible mejorar la robustez del sistema tanto como sea deseado, contemplando el límite de capacidad en la comunicación. En segundo lugar, presentamos un mecanismo estructurado para controlar la topología de la red. La selección de que usuarios servirán a que otros es importante para la robustez de la red, especialmente cuando los usuarios son heterogéneos en sus capacidades y en sus tiempos de conexión.Nuestro diseño maximiza la calidad global esperada (evaluada usando PSQA), seleccionado una topología que mejora la robustez del sistema. Además estudiamos como extender la red con dos servicios complementarios: el video bajo demanda (Video on Demand - VoD) y el servicio MyTV. El desafío en estos servicios es como realizar búsquedas eficientes sobre la librería de videos, dado al alto dinamismo del contenido. Presentamos una estrategia de "caching" para las búsquedas en estos servicios, que maximiza el número total de respuestas correctas a las consultas, considerando una dinámica particular en los contenidos y restricciones de ancho de banda. Nuestro diseño global considera escenarios reales, donde los casos de prueba y los parámetros de configuración surgen de datos reales de un servicio de referencia en producción. Nuestro prototipo es completamente funcional, de uso gratuito, y basado en tecnologías bien probadas de código abierto

Image Processing Using FPGAs

This book presents a selection of papers representing current research on using field programmable gate arrays (FPGAs) for realising image processing algorithms. These papers are reprints of papers selected for a Special Issue of the Journal of Imaging on image processing using FPGAs. A diverse range of topics is covered, including parallel soft processors, memory management, image filters, segmentation, clustering, image analysis, and image compression. Applications include traffic sign recognition for autonomous driving, cell detection for histopathology, and video compression. Collectively, they represent the current state-of-the-art on image processing using FPGAs

Apport de la Qualité de l’Expérience dans l’optimisation de services multimédia : application à la diffusion de la vidéo et à la VoIP

The emerging and fast growth of multimedia services have created new challenges for network service providers in order to guarantee the best user's Quality of Experience (QoE) in diverse networks with distinctive access technologies. Usually, various methods and techniques are used to predict the user satisfaction level by studying the combined impact of numerous factors. In this thesis, we consider two important multimedia services to evaluate the user perception, which are: video streaming service, and VoIP. This study investigates user's QoE that follows three directions: (1) methodologies for subjective QoE assessment of video services, (2) regulating user's QoE using video a rate adaptive algorithm, and (3) QoE-based power efficient resource allocation methods for Long Term Evaluation-Advanced (LTE-A) for VoIP. Initially, we describe two subjective methods to collect the dataset for assessing the user's QoE. The subjectively collected dataset is used to investigate the influence of different parameters (e.g. QoS, video types, user profile, etc.) on user satisfaction while using the video services. Later, we propose a client-based HTTP rate adaptive video streaming algorithm over TCP protocol to regulate the user's QoE. The proposed method considers three Quality of Service (QoS) parameters that govern the user perception, which are: Bandwidth, Buffer, and dropped Frame rate (BBF). The BBF method dynamically selects the suitable video quality according to network conditions and user's device properties. Lastly, we propose a QoE driven downlink scheduling method, i.e. QoE Power Escient Method (QEPEM) for LTE-A. It esciently allocates the radio resources, and optimizes the use of User Equipment (UE) power utilizing the Discontinuous Reception (DRX) method in LTE-AL'émergence et la croissance rapide des services multimédia dans les réseaux IP ont créé de nouveaux défis pour les fournisseurs de services réseau, qui, au-delà de la Qualité de Service (QoS) issue des paramètres techniques de leur réseau, doivent aussi garantir la meilleure qualité de perception utilisateur (Quality of Experience, QoE) dans des réseaux variés avec différentes technologies d'accès. Habituellement, différentes méthodes et techniques sont utilisées pour prédire le niveau de satisfaction de l'utilisateur, en analysant l'effet combiné de multiples facteurs. Dans cette thèse, nous nous intéressons à la commande du réseau en intégrant à la fois des aspects qualitatifs (perception du niveau de satisfaction de l'usager) et quantitatifs (mesure de paramètres réseau) dans l'objectif de développer des mécanismes capables, à la fois, de s'adapter à la variabilité des mesures collectées et d'améliorer la qualité de perception. Pour ce faire, nous avons étudié le cas de deux services multimédia populaires, qui sont : le streaming vidéo, et la voix sur IP (VoIP). Nous investiguons la QoE utilisateur de ces services selon trois aspects : (1) les méthodologies d'évaluation subjective de la QoE, dans le cadre d'un service vidéo, (2) les techniques d'adaptation de flux vidéo pour garantir un certain niveau de QoE, et (3) les méthodes d'allocation de ressource, tenant compte de la QoE tout en économisant l'énergie, dans le cadre d'un service de VoIP (LTE-A). Nous présentons d'abord deux méthodes pour récolter des jeux de données relatifs à la QoE. Nous utilisons ensuite ces jeux de données (issus des campagnes d'évaluation subjective que nous avons menées) pour comprendre l'influence de différents paramètres (réseau, terminal, profil utilisateur) sur la perception d'un utilisateur d'un service vidéo. Nous proposons ensuite un algorithme de streaming vidéo adaptatif, implémenté dans un client HTTP, et dont le but est d'assurer un certain niveau de QoE et le comparons à l'état de l'art. Notre algorithme tient compte de trois paramètres de QoS (bande passante, taille de mémoires tampons de réception et taux de pertes de paquets) et sélectionne dynamiquement la qualité vidéo appropriée en fonction des conditions du réseau et des propriétés du terminal de l'utilisateur. Enfin, nous proposons QEPEM (QoE Power Efficient Method), un algorithme d'ordonnancement basé sur la QoE, dans le cadre d'un réseau sans fil LTE, en nous intéressant à une allocation dynamique des ressources radio en tenant compte de la consommation énergétiqu