Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications

[EN] Newer social multimedia applications, such as Social TV or networked multi-player games, enable independent groups (or clusters) of users to interact among themselves and share services within the context of simultaneous media content consumption. In such scenarios, concurrently synchronized playout points must be ensured so as not to degrade the user experience on such interaction. We refer to this process as Inter-Destination Multimedia Synchronization (IDMS). This paper presents the design, implementation and evaluation of an evolved version of an RTCP-based IDMS approach, including an Adaptive Media Playout (AMP) scheme that aims to dynamically and smoothly adjust the playout timing of each one of the geographically distributed consumers in a specific cluster if an allowable asynchrony threshold between their playout states is exceeded. For that purpose, we previously had also to develop a full implementation of RTP/RTCP protocols for NS-2, in which we included the IDMS approach as an optional functionality. Simulation results prove the feasibility of such IDMS and AMP proposals, by adopting several dynamic master reference selection policies, to maintain an overall synchronization status (within allowable limits) in each cluster of participants, while minimizing the occurrence of long-term playout discontinuities (such as skips/pauses) which are subjectively more annoying and less tolerable to users than small variations in the media playout rate.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10. Authors also would like to thank the anonymous reviewers that helped to significantly improve the quality of the paper with their constructive comments.Montagud, M.; Boronat, F. (2012). Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications. Computer Networks. 56(12):2912-2933. https://doi.org/10.1016/j.comnet.2012.05.00329122933561

On the use of adaptive media playout for inter-destination synchronization

[EN] Inter-Destination Multimedia Synchronization (IDMS) is essential in most of the emerging social multimedia applications. In this paper we present a novel Adaptive Media Playout (AMP) scheme that aims to acquire an overall synchronization status between distributed receivers by means of smoothly adjusting their playout timing. Simulation results show that the proposed solution minimizes long-term playout discontinuities (skips and/or pauses) which are subjectively more annoying to users than small variations in the media playout rate.This work was supported by the Universitat Politecnica de Valencia (UPV), under its R&D Support Program in the PAID-01-10 Project, and by Generalitat Valenciana, under its R&D Support Program in the GV 2010/009 Project.Montagud, M.; Boronat, F. (2011). On the use of adaptive media playout for inter-destination synchronization. IEEE Communications Letters. 15(9):863-865. https://doi.org/10.1109/LCOMM.2011.061611.110072S86386515

Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization

Traditionally, the media consumption model has been a passive and isolated activity. However, the advent of media streaming technologies, interactive social applications, and synchronous communications, as well as the convergence between these three developments, point to an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of consumers, independently of their location and the nature of their end-devices, can be immersed in a common virtual networked environment in which they can share multimedia services, interact and collaborate in real-time within the context of simultaneous media content consumption. In most of these multimedia services and applications, apart from the well-known intra and inter-stream synchronization techniques that are important inside the consumers playout devices, also the synchronization of the playout processes between several distributed receivers, known as multipoint, group or Inter-destination multimedia synchronization (IDMS), becomes essential. Due to the increasing popularity of social networking, this type of multimedia synchronization has gained in popularity in recent years. Although Social TV is perhaps the most prominent use case in which IDMS is useful, in this paper we present up to 19 use cases for IDMS, each one having its own synchronization requirements. Different approaches used in the (recent) past by researchers to achieve IDMS are described and compared. As further proof of the significance of IDMS nowadays, relevant organizations (such as ETSI TISPAN and IETF AVTCORE Group) efforts on IDMS standardization (in which authors have been and are participating actively), defining architectures and protocols, are summarized.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10, and by TNO, under its Future Internet Use Research & Innovation Program. The authors also want to thank Kevin Gross for providing some of the use cases included in Sect. 1.2.Montagud, M.; Boronat Segui, F.; Stokking, H.; Van Brandenburg, R. (2012). Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization. Multimedia Systems. 18(6):459-482. https://doi.org/10.1007/s00530-012-0278-9S459482186Kernchen, R., Meissner, S., Moessner, K., Cesar, P., Vaishnavi, I., Boussard, M., Hesselman, C.: Intelligent multimedia presentation in ubiquitous multidevice scenarios. IEEE Multimedia 17(2), 52–63 (2010)Vaishnavi, I., Cesar, P., Bulterman, D., Friedrich, O., Gunkel, S., Geerts, D.: From IPTV to synchronous shared experiences challenges in design: distributed media synchronization. Signal Process Image Commun 26(7), 370–377 (2011)Geerts, D., Vaishnavi, I., Mekuria, R., Van Deventer, O., Cesar, P.: Are we in sync?: synchronization requirements for watching on-line video together, CHI ‘11, New York, USA (2011)Boronat, F., Lloret, J., García, M.: Multimedia group and inter-stream synchronization techniques: a comparative study. Inf. Syst. 34(1), 108–131 (2009)Chen, M.: A low-latency lip-synchronized videoconferencing system. In: SIGCHI Conference on Human Factors in Computing Systems, CHI’03, ACM, pp. 464–471, New York (2003)Ishibashi, Y., Tasaka, S., Ogawa, H.: Media synchronization quality of reactive control schemes. IEICE Trans. Commun. E86-B(10), 3103–3113 (2003)Ademoye, O.A., Ghinea, G.: Synchronization of olfaction-enhanced multimedia. IEEE Trans. Multimedia 11(3), 561–565 (2009)Cesar, P., Bulterman, D.C.A., Jansen, J., Geerts, D., Knoche, H., Seager, W.: Fragment, tag, enrich, and send: enhancing social sharing of video. ACM Trans. Multimedia Comput. Commun. Appl. 5(3), Article 19, 27 pages (2009)Van Deventer, M.O., Stokking, H., Niamut, O.A., Walraven, F.A., Klos, V.B.: Advanced Interactive Television Service Require Synchronization, IWSSIP 2008. Bratislava, June (2008)Premchaiswadi, W., Tungkasthan, A., Jongsawat, N.: Enhancing learning systems by using virtual interactive classrooms and web-based collaborative work. In: Proceedings of the IEEE Education Engineering Conference (EDUCON 2010), pp. 1531–1537. Madrid, Spain (2010)Diot, C., Gautier, L.: A distributed architecture for multiplayer interactive applications on the internet. IEEE Netw 13(4), 6–15 (1999)Mauve, M., Vogel, J., Hilt, V., Effelsberg, W.: Local-lag and timewarp: providing consistency for replicated continuous applications. IEEE Trans. Multimedia 6(1), 45–57 (2004)Hosoya, K., Ishibashi, Y., Sugawara, S., Psannis, K.E.: Group synchronization control considering difference of conversation roles. In: IEEE 13th International Symposium on Consumer Electronics, ISCE ‘09, pp. 948–952 (2009)Roccetti, M., Ferretti, S., Palazzi, C.: The brave new world of multiplayer online games: synchronization issues with smart solution. In: 11th IEEE Symposium on Object Oriented Real-Time Distributed Computing (ISORC), pp. 587–592 (2008)Ott, D.E., Mayer-Patel, K.: An open architecture for transport-level protocol coordination in distributed multimedia applications. ACM Trans. Multimedia Comput. Commun. Appl. 3(3), 17 (2007)Boronat, F., Montagud, M., Guerri, J.C.: Multimedia group synchronization approach for one-way cluster-to-cluster applications. In: IEEE 34th Conference on Local Computer Networks, LCN 2009, pp. 177–184, Zürich (2009)Boronat, F., Montagud, M., Vidal, V.: Smooth control of adaptive media playout to acquire IDMS in cluster-based applications. In: IEEE LCN 2011, pp. 617–625, Bonn (2011)Huang, Z., Wu, W., Nahrstedt, K., Rivas, R., Arefin, A.: SyncCast: synchronized dissemination in multi-site interactive 3D tele-immersion. In: Proceedings of MMSys, USA (2011)Kim, S.-J., Kuester, F., Kim, K.: A global timestamp-based approach for enhanced data consistency and fairness in collaborative virtual environments. ACM/Springer Multimedia Syst. J. 10(3), 220–229 (2005)Schooler, E.: Distributed music: a foray into networked performance. In: International Network Music Festival, Santa Monica, CA (1993)Miyashita, Y., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis K.E.: QoE assessment of group synchronization in networked chorus with voice and video. In: Proceedings of IEEE TENCON’11, pp. 393–397 (2011)Hesselman, C., Abbadessa, D., Van Der Beek, W., et al.: Sharing enriched multimedia experiences across heterogeneous network infrastructures. IEEE Commun. Mag. 48(6), 54–65 (2010)Montpetit, M., Klym, N., Mirlacher, T.: The future of IPTV—Connected, mobile, personal and social. Multimedia Tools Appl J 53(3), 519–532 (2011)Cesar, P., Bulterman, D.C.A., Jansen, J.: Leveraging the user impact: an architecture for secondary screens usage in an interactive television environment. ACM/Springer Multimedia Syst. 15(3), 127–142 (2009)Lukosch, S.: Transparent latecomer support for synchronous groupware. In: Proceedings of 9th International Workshop on Groupware (CRIWG), Grenoble, France, pp. 26–41 (2003)Steinmetz, R.: Human perception of jitter and media synchronization. IEEE J. Sel. Areas Commun. 14(1), 61–72 (1996)Stokking, H., Van Deventer, M.O., Niamut, O.A., Walraven, F.A., Mekuria, R.N.: IPTV inter-destination synchronization: a network-based approach, ICIN’2010, Berlin (2010)Mekuria, R.N.: Inter-destination media synchronization for TV broadcasts, Master Thesis, Faculty of Electrical Engineering, Mathematics and Computer Science, Department of Network architecture and Services, Delft University of Technology (2011)Pitt Ian, CS2511: Usability engineering lecture notes, localisation of sound sources. http://web.archive.org/web/20100410235208/http:/www.cs.ucc.ie/~ianp/CS2511/HAP.htmlNielsen, J.: Response times: the three important limits. http://www.useit.com/papers/responsetime.html (1994)ITU-T Rec G. 1010: End-User Multimedia QoS Categories. International Telecommunication Union, Geneva (2001)Biersack, E., Geyer, W.: Synchronized delivery and playout of distributed stored multimedia streams. ACM/Springer Multimedia Syst 7(1), 70–90 (1999)Xie, Y., Liu, C., Lee, M.J., Saadawi, T.N.: Adaptive multimedia synchronization in a teleconference system. ACM/Springer Multimedia Syst. 7(4), 326–337 (1999)Laoutaris, N., Stavrakakis, I.: Intrastream synchronization for continuous media streams: a survey of playout schedulers. IEEE Netw. Mag. 16(3), 30–40 (2002)Ishibashi, Y., Tsuji, A., Tasaka, S.: A group synchronization mechanism for stored media in multicast communications. In: Proceedings of the INFOCOM ‘97, Washington (1997)Ishibashi, Y., Tasaka, S.: A group synchronization mechanism for live media in multicast communications. IEEE GLOBECOM’97, pp. 746–752 (1997)Boronat, F., Guerri, J.C., Lloret, J.: An RTP/RTCP based approach for multimedia group and inter-stream synchronization. Multimedia Tools Appl. J. 40(2), 285–319 (2008)Ishibashi, I., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium Communications, Kaohsiung, Taiwan, pp. 317–323 (1999)Lu, Y., Fallica, B., Kuipers, F.A., Kooij, R.E., Van Mieghem, P.: Assessing the quality of experience of SopCast. Int. J. Internet Protoc. Technol 4(1), 11–19 (2009)Shamma, D.A., Bastea-Forte, M., Joubert, N., Liu, Y.: Enhancing online personal connections through synchronized sharing of online video, ACM CHI’08 Extended Abstracts, Florence (2008)Ishibashi, Y., Tasaka, S.: A distributed control scheme for causality and media synchronization in networked multimedia games. In: Proceedings of 11th International Conference on Computer Communications and Networks, pp. 144–149, Miami, USA (2002)Ishibashi, Y., Tomaru, K., Tasaka, S., Inazumi, K.: Group synchronization in networked virtual environments. In: Proceedings of the 38th IEEE International Conference on Communications, pp. 885–890, Alaska, USA (2003)Tasaka, S., Ishibashi, Y., Hayashi, M.: Inter–destination synchronization quality in an integrated wired and wireless network with handover. IEEE GLOBECOM 2, 1560–1565 (2002)Kurokawa, Y., Ishibashi, Y., Asano, T.: Group synchronization control in a remote haptic drawing system. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 572–575, Beijing, China (2007)Hashimoto, T., Ishibashi, Y.: Group Synchronization Control over Haptic Media in a Networked Real-Time Game with Collaborative Work, Netgames’06, Singapore (2006)Nunome, T., Tasaka, S.: Inter-destination synchronization quality in a multicast mobile ad hoc network. In: Proceedings of IEEE 16th International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1366–1370, Berlin, Germany (2005)Brandenburg, R., van Stokking, H., Van Deventer, M.O., Boronat, F., Montagud, M., Gross, K.: RTCP for inter-destination media synchronization, draft-brandenburg-avtcore-rtcp-for-idms-03.txt. In: IETF Audio/Video Transport Core Maintenance Working Group, Internet Draft, March 9 (2012)ETSI TS 181 016 V3.3.1 (2009-07) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); Service Layer Requirements to integrate NGN Services and IPTVETSI TS 182 027 V3.5.1 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IPTV Architecture; IPTV functions supported by the IMS subsystemETSI TS 183 063 V3.5.2 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IMS-based IPTV stage 3 specificationBrandenburg van, R., et al.: RTCP XR Block Type for inter-destination media synchronization, draft-brandenburg-avt-rtcp-for-idms-00.txt. In: IETF Audio/Video Transport Working Group, Internet Draft, Sept 24, 2010Williams, A., et al.: RTP Clock Source Signalling, draft-williams-avtcore-clksrc-00. In: IETF Audio/Video Transport Working Group, Internet Draft, February 28, 201

Design, Development and Assessment of Control Schemes for IDMS in a Standardized RTCP-based Solution

[EN] Currently, several media sharing applications that allow social interactions between distributed users are gaining momentum. In these networked scenarios, synchronized playout between the involved participants must be provided to enable truly interactive and coherent shared media experiences. This research topic is known as Inter-Destination Media Synchronization (IDMS). This paper presents the design and development of an advanced IDMS solution, which is based on extending the capabilities of RTP/RTCP standard protocols. Particularly, novel RTCP extensions, in combination with several control algorithms and adjustment techniques, have been specified to enable an adaptive, highly accurate and standard compliant IDMS solution. Moreover, as different control or architectural schemes for IDMS exist, and each one is best suited for specific use cases, the IDMS solution has been extended to be able to adopt each one of them. Simulation results prove the satisfactory responsiveness of our IDMS solution in a small scale scenario, as well as its consistent behavior, when using each one of the deployed architectural schemes.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-01-10. TNO's work has been partially funded by European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. ICT-2011-8-318343 (STEER Project). CWI's work has been partially funded by the European Community's Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No. ICT-2011-7-287723 (REVERIE Project).Montagud Aguar, M.; Boronat Segui, F.; Stokking, H.; Cesar, P. (2014). Design, Development and Assessment of Control Schemes for IDMS in a Standardized RTCP-based Solution. Computer Networks. 70:240-259. https://doi.org/10.1016/j.comnet.2014.06.004S2402597

Security and Privacy Issues in Wireless Mesh Networks: A Survey

This book chapter identifies various security threats in wireless mesh network (WMN). Keeping in mind the critical requirement of security and user privacy in WMNs, this chapter provides a comprehensive overview of various possible attacks on different layers of the communication protocol stack for WMNs and their corresponding defense mechanisms. First, it identifies the security vulnerabilities in the physical, link, network, transport, application layers. Furthermore, various possible attacks on the key management protocols, user authentication and access control protocols, and user privacy preservation protocols are presented. After enumerating various possible attacks, the chapter provides a detailed discussion on various existing security mechanisms and protocols to defend against and wherever possible prevent the possible attacks. Comparative analyses are also presented on the security schemes with regards to the cryptographic schemes used, key management strategies deployed, use of any trusted third party, computation and communication overhead involved etc. The chapter then presents a brief discussion on various trust management approaches for WMNs since trust and reputation-based schemes are increasingly becoming popular for enforcing security in wireless networks. A number of open problems in security and privacy issues for WMNs are subsequently discussed before the chapter is finally concluded.Comment: 62 pages, 12 figures, 6 tables. This chapter is an extension of the author's previous submission in arXiv submission: arXiv:1102.1226. There are some text overlaps with the previous submissio

Inter-destination Multimedia Synchronization: A Contemporary Survey

The advent of social networking applications, media streaming technologies, and synchronous communications has created an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of users can be immersed in a common virtual networked environment in which they can interact and collaborate in real- time within the context of simultaneous media content consumption. In this environment, intra-stream and inter-stream synchronization techniques are used inside the consumers’ playout devices, while synchronization of media streams across multiple separated locations is required. This synchronization is nown as multipoint, group or Inter-Destination Multimedia Synchronization (IDMS) and is needed in many applications such as social TV and synchronous e-learning. This survey paper discusses intraand inter-stream synchronization issues, but it mainly focuses on the most well-known IDMS techniques that can be used in emerging distributed multimedia applications. In addition, it provides some research directions for future work

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

MAGDA: A Mobile Agent based Grid Architecture

Mobile agents mean both a technology and a programming paradigm. They allow for a flexible approach which can alleviate a number of issues present in distributed and Grid-based systems, by means of features such as migration, cloning, messaging and other provided mechanisms. In this paper we describe an architecture (MAGDA – Mobile Agent based Grid Architecture) we have designed and we are currently developing to support programming and execution of mobile agent based application upon Grid systems

Run-time management for future MPSoC platforms

In recent years, we are witnessing the dawning of the Multi-Processor Systemon- Chip (MPSoC) era. In essence, this era is triggered by the need to handle more complex applications, while reducing overall cost of embedded (handheld) devices. This cost will mainly be determined by the cost of the hardware platform and the cost of designing applications for that platform. The cost of a hardware platform will partly depend on its production volume. In turn, this means that ??exible, (easily) programmable multi-purpose platforms will exhibit a lower cost. A multi-purpose platform not only requires ??exibility, but should also combine a high performance with a low power consumption. To this end, MPSoC devices integrate computer architectural properties of various computing domains. Just like large-scale parallel and distributed systems, they contain multiple heterogeneous processing elements interconnected by a scalable, network-like structure. This helps in achieving scalable high performance. As in most mobile or portable embedded systems, there is a need for low-power operation and real-time behavior. The cost of designing applications is equally important. Indeed, the actual value of future MPSoC devices is not contained within the embedded multiprocessor IC, but in their capability to provide the user of the device with an amount of services or experiences. So from an application viewpoint, MPSoCs are designed to ef??ciently process multimedia content in applications like video players, video conferencing, 3D gaming, augmented reality, etc. Such applications typically require a lot of processing power and a signi??cant amount of memory. To keep up with ever evolving user needs and with new application standards appearing at a fast pace, MPSoC platforms need to be be easily programmable. Application scalability, i.e. the ability to use just enough platform resources according to the user requirements and with respect to the device capabilities is also an important factor. Hence scalability, ??exibility, real-time behavior, a high performance, a low power consumption and, ??nally, programmability are key components in realizing the success of MPSoC platforms. The run-time manager is logically located between the application layer en the platform layer. It has a crucial role in realizing these MPSoC requirements. As it abstracts the platform hardware, it improves platform programmability. By deciding on resource assignment at run-time and based on the performance requirements of the user, the needs of the application and the capabilities of the platform, it contributes to ??exibility, scalability and to low power operation. As it has an arbiter function between different applications, it enables real-time behavior. This thesis details the key components of such an MPSoC run-time manager and provides a proof-of-concept implementation. These key components include application quality management algorithms linked to MPSoC resource management mechanisms and policies, adapted to the provided MPSoC platform services. First, we describe the role, the responsibilities and the boundary conditions of an MPSoC run-time manager in a generic way. This includes a de??nition of the multiprocessor run-time management design space, a description of the run-time manager design trade-offs and a brief discussion on how these trade-offs affect the key MPSoC requirements. This design space de??nition and the trade-offs are illustrated based on ongoing research and on existing commercial and academic multiprocessor run-time management solutions. Consequently, we introduce a fast and ef??cient resource allocation heuristic that considers FPGA fabric properties such as fragmentation. In addition, this thesis introduces a novel task assignment algorithm for handling soft IP cores denoted as hierarchical con??guration. Hierarchical con??guration managed by the run-time manager enables easier application design and increases the run-time spatial mapping freedom. In turn, this improves the performance of the resource assignment algorithm. Furthermore, we introduce run-time task migration components. We detail a new run-time task migration policy closely coupled to the run-time resource assignment algorithm. In addition to detailing a design-environment supported mechanism that enables moving tasks between an ISP and ??ne-grained recon??gurable hardware, we also propose two novel task migration mechanisms tailored to the Network-on-Chip environment. Finally, we propose a novel mechanism for task migration initiation, based on reusing debug registers in modern embedded microprocessors. We propose a reactive on-chip communication management mechanism. We show that by exploiting an injection rate control mechanism it is possible to provide a communication management system capable of providing a soft (reactive) QoS in a NoC. We introduce a novel, platform independent run-time algorithm to perform quality management, i.e. to select an application quality operating point at run-time based on the user requirements and the available platform resources, as reported by the resource manager. This contribution also proposes a novel way to manage the interaction between the quality manager and the resource manager. In order to have a the realistic, reproducible and ??exible run-time manager testbench with respect to applications with multiple quality levels and implementation tradev offs, we have created an input data generation tool denoted Pareto Surfaces For Free (PSFF). The the PSFF tool is, to the best of our knowledge, the ??rst tool that generates multiple realistic application operating points either based on pro??ling information of a real-life application or based on a designer-controlled random generator. Finally, we provide a proof-of-concept demonstrator that combines these concepts and shows how these mechanisms and policies can operate for real-life situations. In addition, we show that the proposed solutions can be integrated into existing platform operating systems