Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization

Traditionally, the media consumption model has been a passive and isolated activity. However, the advent of media streaming technologies, interactive social applications, and synchronous communications, as well as the convergence between these three developments, point to an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of consumers, independently of their location and the nature of their end-devices, can be immersed in a common virtual networked environment in which they can share multimedia services, interact and collaborate in real-time within the context of simultaneous media content consumption. In most of these multimedia services and applications, apart from the well-known intra and inter-stream synchronization techniques that are important inside the consumers playout devices, also the synchronization of the playout processes between several distributed receivers, known as multipoint, group or Inter-destination multimedia synchronization (IDMS), becomes essential. Due to the increasing popularity of social networking, this type of multimedia synchronization has gained in popularity in recent years. Although Social TV is perhaps the most prominent use case in which IDMS is useful, in this paper we present up to 19 use cases for IDMS, each one having its own synchronization requirements. Different approaches used in the (recent) past by researchers to achieve IDMS are described and compared. As further proof of the significance of IDMS nowadays, relevant organizations (such as ETSI TISPAN and IETF AVTCORE Group) efforts on IDMS standardization (in which authors have been and are participating actively), defining architectures and protocols, are summarized.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10, and by TNO, under its Future Internet Use Research & Innovation Program. The authors also want to thank Kevin Gross for providing some of the use cases included in Sect. 1.2.Montagud, M.; Boronat Segui, F.; Stokking, H.; Van Brandenburg, R. (2012). Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization. Multimedia Systems. 18(6):459-482. https://doi.org/10.1007/s00530-012-0278-9S459482186Kernchen, R., Meissner, S., Moessner, K., Cesar, P., Vaishnavi, I., Boussard, M., Hesselman, C.: Intelligent multimedia presentation in ubiquitous multidevice scenarios. IEEE Multimedia 17(2), 52–63 (2010)Vaishnavi, I., Cesar, P., Bulterman, D., Friedrich, O., Gunkel, S., Geerts, D.: From IPTV to synchronous shared experiences challenges in design: distributed media synchronization. Signal Process Image Commun 26(7), 370–377 (2011)Geerts, D., Vaishnavi, I., Mekuria, R., Van Deventer, O., Cesar, P.: Are we in sync?: synchronization requirements for watching on-line video together, CHI ‘11, New York, USA (2011)Boronat, F., Lloret, J., García, M.: Multimedia group and inter-stream synchronization techniques: a comparative study. Inf. Syst. 34(1), 108–131 (2009)Chen, M.: A low-latency lip-synchronized videoconferencing system. In: SIGCHI Conference on Human Factors in Computing Systems, CHI’03, ACM, pp. 464–471, New York (2003)Ishibashi, Y., Tasaka, S., Ogawa, H.: Media synchronization quality of reactive control schemes. IEICE Trans. Commun. E86-B(10), 3103–3113 (2003)Ademoye, O.A., Ghinea, G.: Synchronization of olfaction-enhanced multimedia. IEEE Trans. Multimedia 11(3), 561–565 (2009)Cesar, P., Bulterman, D.C.A., Jansen, J., Geerts, D., Knoche, H., Seager, W.: Fragment, tag, enrich, and send: enhancing social sharing of video. ACM Trans. Multimedia Comput. Commun. Appl. 5(3), Article 19, 27 pages (2009)Van Deventer, M.O., Stokking, H., Niamut, O.A., Walraven, F.A., Klos, V.B.: Advanced Interactive Television Service Require Synchronization, IWSSIP 2008. Bratislava, June (2008)Premchaiswadi, W., Tungkasthan, A., Jongsawat, N.: Enhancing learning systems by using virtual interactive classrooms and web-based collaborative work. In: Proceedings of the IEEE Education Engineering Conference (EDUCON 2010), pp. 1531–1537. Madrid, Spain (2010)Diot, C., Gautier, L.: A distributed architecture for multiplayer interactive applications on the internet. IEEE Netw 13(4), 6–15 (1999)Mauve, M., Vogel, J., Hilt, V., Effelsberg, W.: Local-lag and timewarp: providing consistency for replicated continuous applications. IEEE Trans. Multimedia 6(1), 45–57 (2004)Hosoya, K., Ishibashi, Y., Sugawara, S., Psannis, K.E.: Group synchronization control considering difference of conversation roles. In: IEEE 13th International Symposium on Consumer Electronics, ISCE ‘09, pp. 948–952 (2009)Roccetti, M., Ferretti, S., Palazzi, C.: The brave new world of multiplayer online games: synchronization issues with smart solution. In: 11th IEEE Symposium on Object Oriented Real-Time Distributed Computing (ISORC), pp. 587–592 (2008)Ott, D.E., Mayer-Patel, K.: An open architecture for transport-level protocol coordination in distributed multimedia applications. ACM Trans. Multimedia Comput. Commun. Appl. 3(3), 17 (2007)Boronat, F., Montagud, M., Guerri, J.C.: Multimedia group synchronization approach for one-way cluster-to-cluster applications. In: IEEE 34th Conference on Local Computer Networks, LCN 2009, pp. 177–184, Zürich (2009)Boronat, F., Montagud, M., Vidal, V.: Smooth control of adaptive media playout to acquire IDMS in cluster-based applications. In: IEEE LCN 2011, pp. 617–625, Bonn (2011)Huang, Z., Wu, W., Nahrstedt, K., Rivas, R., Arefin, A.: SyncCast: synchronized dissemination in multi-site interactive 3D tele-immersion. In: Proceedings of MMSys, USA (2011)Kim, S.-J., Kuester, F., Kim, K.: A global timestamp-based approach for enhanced data consistency and fairness in collaborative virtual environments. ACM/Springer Multimedia Syst. J. 10(3), 220–229 (2005)Schooler, E.: Distributed music: a foray into networked performance. In: International Network Music Festival, Santa Monica, CA (1993)Miyashita, Y., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis K.E.: QoE assessment of group synchronization in networked chorus with voice and video. In: Proceedings of IEEE TENCON’11, pp. 393–397 (2011)Hesselman, C., Abbadessa, D., Van Der Beek, W., et al.: Sharing enriched multimedia experiences across heterogeneous network infrastructures. IEEE Commun. Mag. 48(6), 54–65 (2010)Montpetit, M., Klym, N., Mirlacher, T.: The future of IPTV—Connected, mobile, personal and social. Multimedia Tools Appl J 53(3), 519–532 (2011)Cesar, P., Bulterman, D.C.A., Jansen, J.: Leveraging the user impact: an architecture for secondary screens usage in an interactive television environment. ACM/Springer Multimedia Syst. 15(3), 127–142 (2009)Lukosch, S.: Transparent latecomer support for synchronous groupware. In: Proceedings of 9th International Workshop on Groupware (CRIWG), Grenoble, France, pp. 26–41 (2003)Steinmetz, R.: Human perception of jitter and media synchronization. IEEE J. Sel. Areas Commun. 14(1), 61–72 (1996)Stokking, H., Van Deventer, M.O., Niamut, O.A., Walraven, F.A., Mekuria, R.N.: IPTV inter-destination synchronization: a network-based approach, ICIN’2010, Berlin (2010)Mekuria, R.N.: Inter-destination media synchronization for TV broadcasts, Master Thesis, Faculty of Electrical Engineering, Mathematics and Computer Science, Department of Network architecture and Services, Delft University of Technology (2011)Pitt Ian, CS2511: Usability engineering lecture notes, localisation of sound sources. http://web.archive.org/web/20100410235208/http:/www.cs.ucc.ie/~ianp/CS2511/HAP.htmlNielsen, J.: Response times: the three important limits. http://www.useit.com/papers/responsetime.html (1994)ITU-T Rec G. 1010: End-User Multimedia QoS Categories. International Telecommunication Union, Geneva (2001)Biersack, E., Geyer, W.: Synchronized delivery and playout of distributed stored multimedia streams. ACM/Springer Multimedia Syst 7(1), 70–90 (1999)Xie, Y., Liu, C., Lee, M.J., Saadawi, T.N.: Adaptive multimedia synchronization in a teleconference system. ACM/Springer Multimedia Syst. 7(4), 326–337 (1999)Laoutaris, N., Stavrakakis, I.: Intrastream synchronization for continuous media streams: a survey of playout schedulers. IEEE Netw. Mag. 16(3), 30–40 (2002)Ishibashi, Y., Tsuji, A., Tasaka, S.: A group synchronization mechanism for stored media in multicast communications. In: Proceedings of the INFOCOM ‘97, Washington (1997)Ishibashi, Y., Tasaka, S.: A group synchronization mechanism for live media in multicast communications. IEEE GLOBECOM’97, pp. 746–752 (1997)Boronat, F., Guerri, J.C., Lloret, J.: An RTP/RTCP based approach for multimedia group and inter-stream synchronization. Multimedia Tools Appl. J. 40(2), 285–319 (2008)Ishibashi, I., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium Communications, Kaohsiung, Taiwan, pp. 317–323 (1999)Lu, Y., Fallica, B., Kuipers, F.A., Kooij, R.E., Van Mieghem, P.: Assessing the quality of experience of SopCast. Int. J. Internet Protoc. Technol 4(1), 11–19 (2009)Shamma, D.A., Bastea-Forte, M., Joubert, N., Liu, Y.: Enhancing online personal connections through synchronized sharing of online video, ACM CHI’08 Extended Abstracts, Florence (2008)Ishibashi, Y., Tasaka, S.: A distributed control scheme for causality and media synchronization in networked multimedia games. In: Proceedings of 11th International Conference on Computer Communications and Networks, pp. 144–149, Miami, USA (2002)Ishibashi, Y., Tomaru, K., Tasaka, S., Inazumi, K.: Group synchronization in networked virtual environments. In: Proceedings of the 38th IEEE International Conference on Communications, pp. 885–890, Alaska, USA (2003)Tasaka, S., Ishibashi, Y., Hayashi, M.: Inter–destination synchronization quality in an integrated wired and wireless network with handover. IEEE GLOBECOM 2, 1560–1565 (2002)Kurokawa, Y., Ishibashi, Y., Asano, T.: Group synchronization control in a remote haptic drawing system. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 572–575, Beijing, China (2007)Hashimoto, T., Ishibashi, Y.: Group Synchronization Control over Haptic Media in a Networked Real-Time Game with Collaborative Work, Netgames’06, Singapore (2006)Nunome, T., Tasaka, S.: Inter-destination synchronization quality in a multicast mobile ad hoc network. In: Proceedings of IEEE 16th International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1366–1370, Berlin, Germany (2005)Brandenburg, R., van Stokking, H., Van Deventer, M.O., Boronat, F., Montagud, M., Gross, K.: RTCP for inter-destination media synchronization, draft-brandenburg-avtcore-rtcp-for-idms-03.txt. In: IETF Audio/Video Transport Core Maintenance Working Group, Internet Draft, March 9 (2012)ETSI TS 181 016 V3.3.1 (2009-07) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); Service Layer Requirements to integrate NGN Services and IPTVETSI TS 182 027 V3.5.1 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IPTV Architecture; IPTV functions supported by the IMS subsystemETSI TS 183 063 V3.5.2 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IMS-based IPTV stage 3 specificationBrandenburg van, R., et al.: RTCP XR Block Type for inter-destination media synchronization, draft-brandenburg-avt-rtcp-for-idms-00.txt. In: IETF Audio/Video Transport Working Group, Internet Draft, Sept 24, 2010Williams, A., et al.: RTP Clock Source Signalling, draft-williams-avtcore-clksrc-00. In: IETF Audio/Video Transport Working Group, Internet Draft, February 28, 201

Video Streaming over Vehicular Ad Hoc Networks: A Comparative Study and Future Perspectives

Vehicular  Ad Hoc Network  (VANET) is emerged as an important research area that provides ubiquitous short-range connectivity among moving vehicles.  This network enables efficient traffic safety and infotainment applications. One of the promising applications is video transmission in vehicle-to-vehicle or vehicle-to-infrastructure environments.  But, video streaming over vehicular environment is a daunting task due to high movement of vehicles. This paper presents a survey on state-of-arts of video streaming over VANET. Furthermore, taxonomy of vehicular video transmission is highlighted in this paper with special focus on significant applications and their requirements with challenges, video content sharing, multi-source video streaming and video broadcast services. The comparative study of the paper compares the video streaming schemes based on type of error resilient technique, objective of study, summary of their study, the utilized simulator and the type of video sharing.  Lastly, we discussed the open issues and research directions related to video communication over VANET

Models and Methods for Network Selection and Balancing in Heterogeneous Scenarios

The outbreak of 5G technologies for wireless communications can be considered a response to the need for widespread coverage, in terms of connectivity and bandwidth, to guarantee broadband services, such as streaming or on-demand programs offered by the main television networks or new generation services based on augmented and virtual reality (AR / VR). The purpose of the study conducted for this thesis aims to solve two of the main problems that will occur with the outbreak of 5G, that is, the search for the best possible connectivity, in order to offer users the resources necessary to take advantage of the new generation services, and multicast as required by the eMBMS. The aim of the thesis is the search for innovative algorithms that will allow to obtain the best connectivity to offer users the resources necessary to use the 5G services in a heterogeneous scenario. Study UF that allows you to improve the search for the best candidate network and to achieve a balance that allows you to avoid congestion of the chosen networks. To achieve these two important focuses, I conducted a study on the main mathematical methods that made it possible to select the network based on QoS parameters based on the type of traffic made by users. A further goal was to improve the computational computation performance they present. Furthermore, I carried out a study in order to obtain an innovative algorithm that would allow the management of multicast. The algorithm that has been implemented responds to the needs present in the eMBMS, in realistic scenarios

Design, Development and Assessment of Control Schemes for IDMS in a Standardized RTCP-based Solution

[EN] Currently, several media sharing applications that allow social interactions between distributed users are gaining momentum. In these networked scenarios, synchronized playout between the involved participants must be provided to enable truly interactive and coherent shared media experiences. This research topic is known as Inter-Destination Media Synchronization (IDMS). This paper presents the design and development of an advanced IDMS solution, which is based on extending the capabilities of RTP/RTCP standard protocols. Particularly, novel RTCP extensions, in combination with several control algorithms and adjustment techniques, have been specified to enable an adaptive, highly accurate and standard compliant IDMS solution. Moreover, as different control or architectural schemes for IDMS exist, and each one is best suited for specific use cases, the IDMS solution has been extended to be able to adopt each one of them. Simulation results prove the satisfactory responsiveness of our IDMS solution in a small scale scenario, as well as its consistent behavior, when using each one of the deployed architectural schemes.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-01-10. TNO's work has been partially funded by European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. ICT-2011-8-318343 (STEER Project). CWI's work has been partially funded by the European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. ICT-2011-7-287723 (REVERIE Project).Montagud Aguar, M.; Boronat Segui, F.; Stokking, H.; Cesar, P. (2014). Design, Development and Assessment of Control Schemes for IDMS in a Standardized RTCP-based Solution. Computer Networks. 70:240-259. https://doi.org/10.1016/j.comnet.2014.06.004S2402597

Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications

[EN] Newer social multimedia applications, such as Social TV or networked multi-player games, enable independent groups (or clusters) of users to interact among themselves and share services within the context of simultaneous media content consumption. In such scenarios, concurrently synchronized playout points must be ensured so as not to degrade the user experience on such interaction. We refer to this process as Inter-Destination Multimedia Synchronization (IDMS). This paper presents the design, implementation and evaluation of an evolved version of an RTCP-based IDMS approach, including an Adaptive Media Playout (AMP) scheme that aims to dynamically and smoothly adjust the playout timing of each one of the geographically distributed consumers in a specific cluster if an allowable asynchrony threshold between their playout states is exceeded. For that purpose, we previously had also to develop a full implementation of RTP/RTCP protocols for NS-2, in which we included the IDMS approach as an optional functionality. Simulation results prove the feasibility of such IDMS and AMP proposals, by adopting several dynamic master reference selection policies, to maintain an overall synchronization status (within allowable limits) in each cluster of participants, while minimizing the occurrence of long-term playout discontinuities (such as skips/pauses) which are subjectively more annoying and less tolerable to users than small variations in the media playout rate.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10. Authors also would like to thank the anonymous reviewers that helped to significantly improve the quality of the paper with their constructive comments.Montagud, M.; Boronat, F. (2012). Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications. Computer Networks. 56(12):2912-2933. https://doi.org/10.1016/j.comnet.2012.05.00329122933561

Managing Network Delay for Browser Multiplayer Games

Latency is one of the key performance elements affecting the quality of experience (QoE) in computer games. Latency in the context of games can be defined as the time between the user input and the result on the screen. In order for the QoE to be satisfactory the game needs to be able to react fast enough to player input. In networked multiplayer games, latency is composed of network delay and local delays. Some major sources of network delay are queuing delay and head-of-line (HOL) blocking delay. Network delay in the Internet can be even in the order of seconds. In this thesis we discuss what feasible networking solutions exist for browser multiplayer games. We conduct a literature study to analyze the Differentiated Services architecture, some salient Active Queue Management (AQM) algorithms (RED, PIE, CoDel and FQ-CoDel), the Explicit Congestion Notification (ECN) concept and network protocols for web browser (WebSocket, QUIC and WebRTC). RED, PIE and CoDel as single-queue implementations would be sub-optimal for providing low latency to game traffic. FQ-CoDel is a multi-queue AQM and provides flow separation that is able to prevent queue-building bulk transfers from notably hampering latency-sensitive flows. WebRTC Data-Channel seems promising for games since it can be used for sending arbitrary application data and it can avoid HOL blocking. None of the network protocols, however, provide completely satisfactory support for the transport needs of multiplayer games: WebRTC is not designed for client-server connections, QUIC is not designed for traffic patterns typical for multiplayer games and WebSocket would require parallel connections to mitigate the effects of HOL blocking

Referee-based architectures for massively multiplayer online games

Network computer games are played amongst players on different hosts across the Internet. Massively Multiplayer Online Games (MMOG) are network games in which thousands of players participate simultaneously in each instance of the virtual world. Current commercial MMOG use a Client/Server (C/S) architecture in which the server simulates and validates the game, and notifies players about the current game state. While C/S is very popular, it has several limitations: (i) C/S has poor scalability as the server is a bandwidth and processing bottleneck; (ii) all updates must be routed through the server, reducing responsiveness; (iii) players with lower client-to-server delay than their opponents have an unfair advantage as they can respond to game events faster; and (iv) the server is a single point of failure.The Mirrored Server (MS) architecture uses multiple mirrored servers connected via a private network. MS achieves better scalability, responsiveness, fairness, and reliability than C/S; however, as updates are still routed through the mirrored servers the problems are not eliminated. P2P network game architectures allow players to exchange updates directly, maximising scalability, responsiveness, and fairness, while removing the single point of failure. However, P2P games are vulnerable to cheating. Several P2P architectures have been proposed to detect and/or prevent game cheating. Nevertheless, they only address a subset of cheating methods. Further, these solutions require costly distributed validation algorithms that increase game delay and bandwidth, and prevent players with high latency from participating.In this thesis we propose a new cheat classification that reflects the levels in which the cheats occur: game, application, protocol, or infrastructure. We also propose three network game architectures: the Referee Anti-Cheat Scheme (RACS), the Mirrored Referee Anti-Cheat Scheme (MRACS), and the Distributed Referee Anti-Cheat Scheme (DRACS); which maximise game scalability, responsiveness, and fairness, while maintaining cheat detection/prevention equal to that in C/S. Each proposed architecture utilises one or more trusted referees to validate the game simulation - similar to the server in C/S - while allowing players to exchange updates directly - similar to peers in P2P.RACS is a hybrid C/S and P2P architecture that improves C/S by using a referee in the server. RACS allows honest players to exchange updates directly between each other, with a copy sent to the referee for validation. By allowing P2P communication RACS has better responsiveness and fairness than C/S. Further, as the referee is not required to forward updates it has better bandwidth and processing scalability. The RACS protocol could be applied to any existing C/S game. Compared to P2P protocols RACS has lower delay, and allows players with high delay to participate. Like in many P2P architectures, RACS divides time into rounds. We have proposed two efficient solutions to find the optimal round length such that the total system delay is minimised.MRACS combines the RACS and MS architectures. A referee is used at each mirror to validate player updates, while allowing players to exchange updates directly. By using multiple mirrored referees the bandwidth required by each referee, and the player-to mirror delays, are reduced; improving the scalability, responsiveness and fairness of RACS, while removing its single point of failure. Direct communication MRACS improves MS in terms of its responsiveness, fairness, and scalability. To maximise responsiveness, we have defined and solved the Client-to-Mirror Assignment (CMA) problem to assign clients to mirrors such that the total delay is minimised, and no mirror is overloaded. We have proposed two sets of efficient solutions: the optimal J-SA/L-SA and the faster heuristic J-Greedy/L-Greedy to solve CMA.DRACS uses referees distributed to player hosts to minimise the publisher / developer infrastructure, and maximise responsiveness and/or fairness. To prevent colluding players cheating DRACS requires every update to be validated by multiple unaffiliated referees, providing cheat detection / prevention equal to that in C/S. We have formally defined the Referee Selection Problem (RSP) to select a set of referees from the untrusted peers such that responsiveness and/or fairness are maximised, while ensuring the probability of the majority of referees colluding is below a pre-defined threshold. We have proposed two efficient algorithms, SRS-1 and SRS-2, to solve the problem.We have evaluated the performances of RACS, MRACS, and DRACS analytically and using simulations. We have shown analytically that RACS, MRACS and DRACS have cheat detection/prevention equivalent to that in C/S. Our analysis shows that RACS has better scalability and responsiveness than C/S; and that MRACS has better scalability and responsiveness than C/S, RACS, and MS. As there is currently no publicly available traces from MMOG we have constructed artificial and realistic inputs. We have used these inputs on all simulations in this thesis to show the benefits of our proposed architectures and algorithms