Closing the gap: human factors in cross-device media synchronization

The continuing growth in the mobile phone arena, particularly in terms of device capabilities and ownership is having a transformational impact on media consumption. It is now possible to consider orchestrated multi-stream experiences delivered across many devices, rather than the playback of content from a single device. However, there are significant challenges in realising such a vision, particularly around the management of synchronicity between associated media streams. This is compounded by the heterogeneous nature of user devices, the networks upon which they operate, and the perceptions of users. This paper describes IMSync, an open inter-stream synchronisation framework that is QoE-aware. IMSync adopts efficient monitoring and control mechanisms, alongside a QoE perception model that has been derived from a series of subjective user experiments. Based on an observation of lag, IMSync is able to use this model of impact to determine an appropriate strategy to catch-up with playback whilst minimising the potential detrimental impacts on a users QoE. The impact model adopts a balanced approach: trading off the potential impact on QoE of initiating a re-synchronisation process compared with retaining the current levels of non-synchronicity, in order to maintain high levels of QoE. A series of experiments demonstrate the potential of the framework as a basis for enabling new, immersive media experiences

QoE-aware inter-stream synchronization in open N-screens cloud

The growing popularity and increasing performance of mobile devices is transforming the way in which media can be consumed, from single device playback to orchestrated multi-stream experiences across multiple devices. One of the biggest challenges in realizing such immersive media experience is the dynamic management of synchronicity between associated media streams. This is further complicated by the faceted aspects of user perception and heterogeneity of user devices and networks. This paper introduces a QoE-aware open inter-stream media synchronization framework (IMSync). IMSync employs efficient monitoring and control mechanisms, as well as a bespoke QoE impact model derived from subjective user experiments. Given a current lag, IMSync's aim is to use the impact model to determine a good catch-up strategy that minimizes detrimental impact on QoE. The impact model balances the accumulative impact of re-synchronization processes and the degree of non-synchronicity to ensure the QoE. Experimental results verify the run-time performance of the framework as a foundation for immersive media experience in open N-Screens cloud

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

Inter-destination Multimedia Synchronization: A Contemporary Survey

The advent of social networking applications, media streaming technologies, and synchronous communications has created an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of users can be immersed in a common virtual networked environment in which they can interact and collaborate in real- time within the context of simultaneous media content consumption. In this environment, intra-stream and inter-stream synchronization techniques are used inside the consumers’ playout devices, while synchronization of media streams across multiple separated locations is required. This synchronization is nown as multipoint, group or Inter-Destination Multimedia Synchronization (IDMS) and is needed in many applications such as social TV and synchronous e-learning. This survey paper discusses intraand inter-stream synchronization issues, but it mainly focuses on the most well-known IDMS techniques that can be used in emerging distributed multimedia applications. In addition, it provides some research directions for future work

Creating and broadcasting video-based multi-platform experiences

The majority of TV consumers now watch TV programs in a multi-display environment. Second screens –most often smartphones - are generally used to check information not directly related to the events in the TV content being watched. Broadcasters have tried to orchestrate these different platforms, and there is reason to believe this contributes to user engagement. However, their success has been limited. This might be caused, at least in part, by the different formats of content being used: mobile apps show graphics and text similar to web content, while TV renders a continuous stream of audiovisual content. The arrival of virtual reality displays to the living room further increases the need for consistent experiences across displays. ImmersiaTV is an European H2020 Research and Innovation action that is redesigning the production and delivery process to offer a new form of broadcast multi-platform video. This novel form of multi-platform video is based on the following assumptions: 1. The arrival of head-mounted displays (HMD) to the living room further requires the delivery of content that is consistent across displays, giving freedom to the end-user to engage with content in one or another device. This means that content must be adapted to the specificities of each device, both in terms of content format and interactive input (or lack of interaction, for the case of the TV), but the end-user should be free to adopt one or another device at different moments of the program being broadcasted. 2. This new form of broadcast multi-platform video must seamlessly integrate with and further augment traditional TV and second screen consumer habits. For TV, this means that content can be consumed simply by sitting on the couch and watching, without further input, and that the audiovisual language used follows established conventions. For tablets and smartphones, it means that user input and social media integration must work seamlessly with the specificities of each device. To address these requirements, in the first year of the project we have designed and implemented an end-to-end production, delivery and rendering pipeline for offline content production which specifically addresses these needs. In this short article we outline the design principles, production and delivery methods as well as the tools used and developed for this purpose. Section 2 outlines the design principles adopted, and related academic work. Section 3 introduces the elements constituting the ImmersiaTV end-to-end pipeline and, where there exist, commercial alternatives. Section 4 introduce our main conclusions and further planned work within ImmersiaTV to address some of the limitations detected. The content examples are adapted from the first pilot of ImmersiaTV

IDMS solution for hybrid broadcast broadband delivery within the context of HbbTV standard

"© 2019 IEEE. Personal use of this material is permitted. Permissíon from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertisíng or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works."[EN] Inter-destination media synchronization (IDMS) is a key requirement to enable successful networked shared media experiences between remote users. This paper presents an adaptive, accurate and standard-compliant IDMS solution for hybrid broadcast and broadband delivery. Apart from providing multi- and cross-technology support, the presented IDMS solution is able to accomplish synchronization when different formats/versions of the same, or even related, contents are being played out in a shared session. It is also able to independently manage the playout processes of different groups of users. The IDMS solution has been integrated within an end-to-end platform, which is compatible with the hybrid broadcast broadband TV standard. It has been applied to digital video broadcasting-terrestrial technology and tested for a Social TV scenario, by also including an ad-hoc chat tool as an interaction channel. The results of the conducted (objective and subjective) evaluations prove the statisfactory behavior and performance of the IDMS solution and platform as well as in terms of the perceived quality of experience.This work was supported by Generalitat Valenciana, Investigacion competitiva proyectos, through the Research and Development Program "Grants for research groups to be consolidated, AICO/2017," under Grant AICO/2017/059.Marfil-Reguero, D.; Boronat, F.; Montagud, M.; Sapena Piera, A. (2019). IDMS solution for hybrid broadcast broadband delivery within the context of HbbTV standard. IEEE Transactions on Broadcasting. 65(4):645-663. https://doi.org/10.1109/TBC.2018.2878285S64566365

A Survey on Multimedia-Based Cross-Layer Optimization in Visual Sensor Networks

Visual sensor networks (VSNs) comprised of battery-operated electronic devices endowed with low-resolution cameras have expanded the applicability of a series of monitoring applications. Those types of sensors are interconnected by ad hoc error-prone wireless links, imposing stringent restrictions on available bandwidth, end-to-end delay and packet error rates. In such context, multimedia coding is required for data compression and error-resilience, also ensuring energy preservation over the path(s) toward the sink and improving the end-to-end perceptual quality of the received media. Cross-layer optimization may enhance the expected efficiency of VSNs applications, disrupting the conventional information flow of the protocol layers. When the inner characteristics of the multimedia coding techniques are exploited by cross-layer protocols and architectures, higher efficiency may be obtained in visual sensor networks. This paper surveys recent research on multimedia-based cross-layer optimization, presenting the proposed strategies and mechanisms for transmission rate adjustment, congestion control, multipath selection, energy preservation and error recovery. We note that many multimedia-based cross-layer optimization solutions have been proposed in recent years, each one bringing a wealth of contributions to visual sensor networks

Plataforma Web 2.0 para la Sincronización Distribuida de Contenidos Multimedia e Interacción Social

Shared media experiences between geographically distributed users are gaining momentum. Relevant examples are Social TV, synchronous e-learning and multi-player online games. This paper presents a first release of Wersync, an adaptive web-based platform that provides distributed media synchronization and social interaction (via shared navigation control commands and text chat channels) across remote users. By using Wersync, users can create or join on-going sessions for concurrently consuming the same media content with other remote users in a synchronized manner. Additionally, Wersync provides two social presence mechanisms to encourage the participation of external users in on-going sessions and two privacy mechanisms. Wersync has been developed by exclusively relying on standard web-based technologies, which ensures cross-network, cross-platform and cross-device support. The evaluation results and a link to a demo video prove the satisfactory performance of Wersync, and its functionalities, respectively