Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications

[EN] Newer social multimedia applications, such as Social TV or networked multi-player games, enable independent groups (or clusters) of users to interact among themselves and share services within the context of simultaneous media content consumption. In such scenarios, concurrently synchronized playout points must be ensured so as not to degrade the user experience on such interaction. We refer to this process as Inter-Destination Multimedia Synchronization (IDMS). This paper presents the design, implementation and evaluation of an evolved version of an RTCP-based IDMS approach, including an Adaptive Media Playout (AMP) scheme that aims to dynamically and smoothly adjust the playout timing of each one of the geographically distributed consumers in a specific cluster if an allowable asynchrony threshold between their playout states is exceeded. For that purpose, we previously had also to develop a full implementation of RTP/RTCP protocols for NS-2, in which we included the IDMS approach as an optional functionality. Simulation results prove the feasibility of such IDMS and AMP proposals, by adopting several dynamic master reference selection policies, to maintain an overall synchronization status (within allowable limits) in each cluster of participants, while minimizing the occurrence of long-term playout discontinuities (such as skips/pauses) which are subjectively more annoying and less tolerable to users than small variations in the media playout rate.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10. Authors also would like to thank the anonymous reviewers that helped to significantly improve the quality of the paper with their constructive comments.Montagud, M.; Boronat, F. (2012). Enhanced adaptive RTCP-based inter-destination multimedia synchronization approach for distributed applications. Computer Networks. 56(12):2912-2933. https://doi.org/10.1016/j.comnet.2012.05.00329122933561

Guest editorial

http://www.sciencedirect.com/science/article/B6TYP-46RDF5H-2/1/4d892d8249d5d0271e6b0a8dc9bd23a

Packet prioritizing and delivering for multimedia streaming

Ph.DDOCTOR OF PHILOSOPH

Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization

Traditionally, the media consumption model has been a passive and isolated activity. However, the advent of media streaming technologies, interactive social applications, and synchronous communications, as well as the convergence between these three developments, point to an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of consumers, independently of their location and the nature of their end-devices, can be immersed in a common virtual networked environment in which they can share multimedia services, interact and collaborate in real-time within the context of simultaneous media content consumption. In most of these multimedia services and applications, apart from the well-known intra and inter-stream synchronization techniques that are important inside the consumers playout devices, also the synchronization of the playout processes between several distributed receivers, known as multipoint, group or Inter-destination multimedia synchronization (IDMS), becomes essential. Due to the increasing popularity of social networking, this type of multimedia synchronization has gained in popularity in recent years. Although Social TV is perhaps the most prominent use case in which IDMS is useful, in this paper we present up to 19 use cases for IDMS, each one having its own synchronization requirements. Different approaches used in the (recent) past by researchers to achieve IDMS are described and compared. As further proof of the significance of IDMS nowadays, relevant organizations (such as ETSI TISPAN and IETF AVTCORE Group) efforts on IDMS standardization (in which authors have been and are participating actively), defining architectures and protocols, are summarized.This work has been financed, partially, by Universitat Politecnica de Valencia (UPV), under its R&D Support Program in PAID-05-11-002-331 Project and in PAID-01-10, and by TNO, under its Future Internet Use Research & Innovation Program. The authors also want to thank Kevin Gross for providing some of the use cases included in Sect. 1.2.Montagud, M.; Boronat Segui, F.; Stokking, H.; Van Brandenburg, R. (2012). Inter-Destination Multimedia Synchronization; Schemes, Use Cases and Standardization. Multimedia Systems. 18(6):459-482. https://doi.org/10.1007/s00530-012-0278-9S459482186Kernchen, R., Meissner, S., Moessner, K., Cesar, P., Vaishnavi, I., Boussard, M., Hesselman, C.: Intelligent multimedia presentation in ubiquitous multidevice scenarios. IEEE Multimedia 17(2), 52–63 (2010)Vaishnavi, I., Cesar, P., Bulterman, D., Friedrich, O., Gunkel, S., Geerts, D.: From IPTV to synchronous shared experiences challenges in design: distributed media synchronization. Signal Process Image Commun 26(7), 370–377 (2011)Geerts, D., Vaishnavi, I., Mekuria, R., Van Deventer, O., Cesar, P.: Are we in sync?: synchronization requirements for watching on-line video together, CHI ‘11, New York, USA (2011)Boronat, F., Lloret, J., García, M.: Multimedia group and inter-stream synchronization techniques: a comparative study. Inf. Syst. 34(1), 108–131 (2009)Chen, M.: A low-latency lip-synchronized videoconferencing system. In: SIGCHI Conference on Human Factors in Computing Systems, CHI’03, ACM, pp. 464–471, New York (2003)Ishibashi, Y., Tasaka, S., Ogawa, H.: Media synchronization quality of reactive control schemes. IEICE Trans. Commun. E86-B(10), 3103–3113 (2003)Ademoye, O.A., Ghinea, G.: Synchronization of olfaction-enhanced multimedia. IEEE Trans. Multimedia 11(3), 561–565 (2009)Cesar, P., Bulterman, D.C.A., Jansen, J., Geerts, D., Knoche, H., Seager, W.: Fragment, tag, enrich, and send: enhancing social sharing of video. ACM Trans. Multimedia Comput. Commun. Appl. 5(3), Article 19, 27 pages (2009)Van Deventer, M.O., Stokking, H., Niamut, O.A., Walraven, F.A., Klos, V.B.: Advanced Interactive Television Service Require Synchronization, IWSSIP 2008. Bratislava, June (2008)Premchaiswadi, W., Tungkasthan, A., Jongsawat, N.: Enhancing learning systems by using virtual interactive classrooms and web-based collaborative work. In: Proceedings of the IEEE Education Engineering Conference (EDUCON 2010), pp. 1531–1537. Madrid, Spain (2010)Diot, C., Gautier, L.: A distributed architecture for multiplayer interactive applications on the internet. IEEE Netw 13(4), 6–15 (1999)Mauve, M., Vogel, J., Hilt, V., Effelsberg, W.: Local-lag and timewarp: providing consistency for replicated continuous applications. IEEE Trans. Multimedia 6(1), 45–57 (2004)Hosoya, K., Ishibashi, Y., Sugawara, S., Psannis, K.E.: Group synchronization control considering difference of conversation roles. In: IEEE 13th International Symposium on Consumer Electronics, ISCE ‘09, pp. 948–952 (2009)Roccetti, M., Ferretti, S., Palazzi, C.: The brave new world of multiplayer online games: synchronization issues with smart solution. In: 11th IEEE Symposium on Object Oriented Real-Time Distributed Computing (ISORC), pp. 587–592 (2008)Ott, D.E., Mayer-Patel, K.: An open architecture for transport-level protocol coordination in distributed multimedia applications. ACM Trans. Multimedia Comput. Commun. Appl. 3(3), 17 (2007)Boronat, F., Montagud, M., Guerri, J.C.: Multimedia group synchronization approach for one-way cluster-to-cluster applications. In: IEEE 34th Conference on Local Computer Networks, LCN 2009, pp. 177–184, Zürich (2009)Boronat, F., Montagud, M., Vidal, V.: Smooth control of adaptive media playout to acquire IDMS in cluster-based applications. In: IEEE LCN 2011, pp. 617–625, Bonn (2011)Huang, Z., Wu, W., Nahrstedt, K., Rivas, R., Arefin, A.: SyncCast: synchronized dissemination in multi-site interactive 3D tele-immersion. In: Proceedings of MMSys, USA (2011)Kim, S.-J., Kuester, F., Kim, K.: A global timestamp-based approach for enhanced data consistency and fairness in collaborative virtual environments. ACM/Springer Multimedia Syst. J. 10(3), 220–229 (2005)Schooler, E.: Distributed music: a foray into networked performance. In: International Network Music Festival, Santa Monica, CA (1993)Miyashita, Y., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis K.E.: QoE assessment of group synchronization in networked chorus with voice and video. In: Proceedings of IEEE TENCON’11, pp. 393–397 (2011)Hesselman, C., Abbadessa, D., Van Der Beek, W., et al.: Sharing enriched multimedia experiences across heterogeneous network infrastructures. IEEE Commun. Mag. 48(6), 54–65 (2010)Montpetit, M., Klym, N., Mirlacher, T.: The future of IPTV—Connected, mobile, personal and social. Multimedia Tools Appl J 53(3), 519–532 (2011)Cesar, P., Bulterman, D.C.A., Jansen, J.: Leveraging the user impact: an architecture for secondary screens usage in an interactive television environment. ACM/Springer Multimedia Syst. 15(3), 127–142 (2009)Lukosch, S.: Transparent latecomer support for synchronous groupware. In: Proceedings of 9th International Workshop on Groupware (CRIWG), Grenoble, France, pp. 26–41 (2003)Steinmetz, R.: Human perception of jitter and media synchronization. IEEE J. Sel. Areas Commun. 14(1), 61–72 (1996)Stokking, H., Van Deventer, M.O., Niamut, O.A., Walraven, F.A., Mekuria, R.N.: IPTV inter-destination synchronization: a network-based approach, ICIN’2010, Berlin (2010)Mekuria, R.N.: Inter-destination media synchronization for TV broadcasts, Master Thesis, Faculty of Electrical Engineering, Mathematics and Computer Science, Department of Network architecture and Services, Delft University of Technology (2011)Pitt Ian, CS2511: Usability engineering lecture notes, localisation of sound sources. http://web.archive.org/web/20100410235208/http:/www.cs.ucc.ie/~ianp/CS2511/HAP.htmlNielsen, J.: Response times: the three important limits. http://www.useit.com/papers/responsetime.html (1994)ITU-T Rec G. 1010: End-User Multimedia QoS Categories. International Telecommunication Union, Geneva (2001)Biersack, E., Geyer, W.: Synchronized delivery and playout of distributed stored multimedia streams. ACM/Springer Multimedia Syst 7(1), 70–90 (1999)Xie, Y., Liu, C., Lee, M.J., Saadawi, T.N.: Adaptive multimedia synchronization in a teleconference system. ACM/Springer Multimedia Syst. 7(4), 326–337 (1999)Laoutaris, N., Stavrakakis, I.: Intrastream synchronization for continuous media streams: a survey of playout schedulers. IEEE Netw. Mag. 16(3), 30–40 (2002)Ishibashi, Y., Tsuji, A., Tasaka, S.: A group synchronization mechanism for stored media in multicast communications. In: Proceedings of the INFOCOM ‘97, Washington (1997)Ishibashi, Y., Tasaka, S.: A group synchronization mechanism for live media in multicast communications. IEEE GLOBECOM’97, pp. 746–752 (1997)Boronat, F., Guerri, J.C., Lloret, J.: An RTP/RTCP based approach for multimedia group and inter-stream synchronization. Multimedia Tools Appl. J. 40(2), 285–319 (2008)Ishibashi, I., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium Communications, Kaohsiung, Taiwan, pp. 317–323 (1999)Lu, Y., Fallica, B., Kuipers, F.A., Kooij, R.E., Van Mieghem, P.: Assessing the quality of experience of SopCast. Int. J. Internet Protoc. Technol 4(1), 11–19 (2009)Shamma, D.A., Bastea-Forte, M., Joubert, N., Liu, Y.: Enhancing online personal connections through synchronized sharing of online video, ACM CHI’08 Extended Abstracts, Florence (2008)Ishibashi, Y., Tasaka, S.: A distributed control scheme for causality and media synchronization in networked multimedia games. In: Proceedings of 11th International Conference on Computer Communications and Networks, pp. 144–149, Miami, USA (2002)Ishibashi, Y., Tomaru, K., Tasaka, S., Inazumi, K.: Group synchronization in networked virtual environments. In: Proceedings of the 38th IEEE International Conference on Communications, pp. 885–890, Alaska, USA (2003)Tasaka, S., Ishibashi, Y., Hayashi, M.: Inter–destination synchronization quality in an integrated wired and wireless network with handover. IEEE GLOBECOM 2, 1560–1565 (2002)Kurokawa, Y., Ishibashi, Y., Asano, T.: Group synchronization control in a remote haptic drawing system. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 572–575, Beijing, China (2007)Hashimoto, T., Ishibashi, Y.: Group Synchronization Control over Haptic Media in a Networked Real-Time Game with Collaborative Work, Netgames’06, Singapore (2006)Nunome, T., Tasaka, S.: Inter-destination synchronization quality in a multicast mobile ad hoc network. In: Proceedings of IEEE 16th International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1366–1370, Berlin, Germany (2005)Brandenburg, R., van Stokking, H., Van Deventer, M.O., Boronat, F., Montagud, M., Gross, K.: RTCP for inter-destination media synchronization, draft-brandenburg-avtcore-rtcp-for-idms-03.txt. In: IETF Audio/Video Transport Core Maintenance Working Group, Internet Draft, March 9 (2012)ETSI TS 181 016 V3.3.1 (2009-07) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); Service Layer Requirements to integrate NGN Services and IPTVETSI TS 182 027 V3.5.1 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IPTV Architecture; IPTV functions supported by the IMS subsystemETSI TS 183 063 V3.5.2 (2011-03) Telecommunications and Internet converged Services and Protocols for Advanced Networking (TISPAN); IMS-based IPTV stage 3 specificationBrandenburg van, R., et al.: RTCP XR Block Type for inter-destination media synchronization, draft-brandenburg-avt-rtcp-for-idms-00.txt. In: IETF Audio/Video Transport Working Group, Internet Draft, Sept 24, 2010Williams, A., et al.: RTP Clock Source Signalling, draft-williams-avtcore-clksrc-00. In: IETF Audio/Video Transport Working Group, Internet Draft, February 28, 201

Inter-destination Multimedia Synchronization: A Contemporary Survey

The advent of social networking applications, media streaming technologies, and synchronous communications has created an evolution towards dynamic shared media experiences. In this new model, geographically distributed groups of users can be immersed in a common virtual networked environment in which they can interact and collaborate in real- time within the context of simultaneous media content consumption. In this environment, intra-stream and inter-stream synchronization techniques are used inside the consumers’ playout devices, while synchronization of media streams across multiple separated locations is required. This synchronization is nown as multipoint, group or Inter-Destination Multimedia Synchronization (IDMS) and is needed in many applications such as social TV and synchronous e-learning. This survey paper discusses intraand inter-stream synchronization issues, but it mainly focuses on the most well-known IDMS techniques that can be used in emerging distributed multimedia applications. In addition, it provides some research directions for future work

Content-Aware Multimedia Communications

The demands for fast, economic and reliable dissemination of multimedia information are steadily growing within our society. While people and economy increasingly rely on communication technologies, engineers still struggle with their growing complexity. Complexity in multimedia communication originates from several sources. The most prominent is the unreliability of packet networks like the Internet. Recent advances in scheduling and error control mechanisms for streaming protocols have shown that the quality and robustness of multimedia delivery can be improved significantly when protocols are aware of the content they deliver. However, the proposed mechanisms require close cooperation between transport systems and application layers which increases the overall system complexity. Current approaches also require expensive metrics and focus on special encoding formats only. A general and efficient model is missing so far. This thesis presents efficient and format-independent solutions to support cross-layer coordination in system architectures. In particular, the first contribution of this work is a generic dependency model that enables transport layers to access content-specific properties of media streams, such as dependencies between data units and their importance. The second contribution is the design of a programming model for streaming communication and its implementation as a middleware architecture. The programming model hides the complexity of protocol stacks behind simple programming abstractions, but exposes cross-layer control and monitoring options to application programmers. For example, our interfaces allow programmers to choose appropriate failure semantics at design time while they can refine error protection and visibility of low-level errors at run-time. Based on some examples we show how our middleware simplifies the integration of stream-based communication into large-scale application architectures. An important result of this work is that despite cross-layer cooperation, neither application nor transport protocol designers experience an increase in complexity. Application programmers can even reuse existing streaming protocols which effectively increases system robustness.Der Bedarf unsere Gesellschaft nach kostengünstiger und zuverlässiger Kommunikation wächst stetig. Während wir uns selbst immer mehr von modernen Kommunikationstechnologien abhängig machen, müssen die Ingenieure dieser Technologien sowohl den Bedarf nach schneller Einführung neuer Produkte befriedigen als auch die wachsende Komplexität der Systeme beherrschen. Gerade die Übertragung multimedialer Inhalte wie Video und Audiodaten ist nicht trivial. Einer der prominentesten Gründe dafür ist die Unzuverlässigkeit heutiger Netzwerke, wie z.B.~dem Internet. Paketverluste und schwankende Laufzeiten können die Darstellungsqualität massiv beeinträchtigen. Wie jüngste Entwicklungen im Bereich der Streaming-Protokolle zeigen, sind jedoch Qualität und Robustheit der Übertragung effizient kontrollierbar, wenn Streamingprotokolle Informationen über den Inhalt der transportierten Daten ausnutzen. Existierende Ansätze, die den Inhalt von Multimediadatenströmen beschreiben, sind allerdings meist auf einzelne Kompressionsverfahren spezialisiert und verwenden berechnungsintensive Metriken. Das reduziert ihren praktischen Nutzen deutlich. Außerdem erfordert der Informationsaustausch eine enge Kooperation zwischen Applikationen und Transportschichten. Da allerdings die Schnittstellen aktueller Systemarchitekturen nicht darauf vorbereitet sind, müssen entweder die Schnittstellen erweitert oder alternative Architekturkonzepte geschaffen werden. Die Gefahr beider Varianten ist jedoch, dass sich die Komplexität eines Systems dadurch weiter erhöhen kann. Das zentrale Ziel dieser Dissertation ist es deshalb, schichtenübergreifende Koordination bei gleichzeitiger Reduzierung der Komplexität zu erreichen. Hier leistet die Arbeit zwei Beträge zum aktuellen Stand der Forschung. Erstens definiert sie ein universelles Modell zur Beschreibung von Inhaltsattributen, wie Wichtigkeiten und Abhängigkeitsbeziehungen innerhalb eines Datenstroms. Transportschichten können dieses Wissen zur effizienten Fehlerkontrolle verwenden. Zweitens beschreibt die Arbeit das Noja Programmiermodell für multimediale Middleware. Noja definiert Abstraktionen zur Übertragung und Kontrolle multimedialer Ströme, die die Koordination von Streamingprotokollen mit Applikationen ermöglichen. Zum Beispiel können Programmierer geeignete Fehlersemantiken und Kommunikationstopologien auswählen und den konkreten Fehlerschutz dann zur Laufzeit verfeinern und kontrolliere

Resource allocation and adaptive scheduling for scalable video streaming

The obvious recent advances in areas such as video compression and network architectures allow for the deployment of novel video distribution applications. These have the potential to provide ubiquitous media access to end users. In recent years, applications based on audio and video streaming have turned out to be immensely popular and the Internet has become the most widely used vector for media content distribution, due to its high availability and connectivity. However, the nature of the Internet infrastructure is not adapted to the specific characteristics of multimedia traffic, which presents a certain tolerance to losses, but strict delay and high bandwidth requirements. In this thesis, our goal is to improve the efficiency of media delivery over the existing network architecture. In order to do so we consider the delivery of scalable video in three main delivery scenarios, namely one-to-one client server architectures, one-to-many broadcasting architectures, and many-to-one distributed streaming architectures. First, we propose a distributed media-friendly rate allocation algorithm for the delivery of both finely and coarsely scalable video streams. Unlike existing solutions, our algorithm explicitly takes the characteristics of media streams into consideration. As a result, it provides rate allocations that better fit the heterogeneous characteristics of media streams. We outline an implementation that is robust to random feedback delays and that permits a scalable deployment of the algorithm. The rate allocation that is computed by our algorithm achieves network stability and high bandwidth utilization. It moreover allows to maximize the average received quality for all streams that are delivered in the network. While considering the transmission of coarsely layered streams, we derive conditions on the encoding rates of the video layers. These conditions depend on the allowed end-to-end delay and on the rate allocation algorithm that controls the sending rates. They allow us to take full advantage of the allocated transmission rates. Second, we investigate the problem of jointly addressing the needs of multiple receivers that consume different versions of a layered media stream in a broadcasting scenario. We provide optimal scheduling algorithms that jointly optimize the playback delay and the buffer occupancy at all of these receivers when the used channel is known. Furthermore we analyze low complexity heuristics based optimization techniques, which provide close to optimal results when only limited channel knowledge is available. Finally, we explore the possibility to exploit the inherent network diversity that is provided by the Internet infrastructure. In particular, we consider media delivery schemes where multiple senders are available for the transmission of a scalable video stream to a single client. Such an architecture is referred to as a distributed streaming architecture. It has the benefit of aggregating multiple unreliable channels into a single more robust channel with high availability. Through the use of Fountain codes, we are able to transform the distributed streaming problem into a rate allocation problem of lower complexity. The solution to this problem is shown to depend not only on the average packet loss rate, but also on the average length of packet loss bursts that are observed on each of the available channels. The coding scheme that we suggest enables our system to adapt the streamed content to the network characteristics, as well as to the needs of the receiving client

Detection and Mitigation of Impairments for Real-Time Multimedia Applications

Measures of Quality of Service (QoS) for multimedia services should focus on phenomena that are observable to the end-user. Metrics such as delay and loss may have little direct meaning to the end-user because knowledge of specific coding and/or adaptive techniques is required to translate delay and loss to the user-perceived performance. Impairment events, as defined in this dissertation, are observable by the end-users independent of coding, adaptive playout or packet loss concealment techniques employed by their multimedia applications. Methods for detecting real-time multimedia (RTM) impairment events from end-to-end measurements are developed here and evaluated using 26 days of PlanetLab measurements collected over nine different Internet paths. Furthermore, methods for detecting impairment-causing network events like route changes and congestion are also developed. The advanced detection techniques developed in this work can be used by applications to detect and match response to network events. The heuristics-based techniques for detecting congestion and route changes were evaluated using PlanetLab measurements. It was found that Congestion events occurred for 6-8 hours during the days on weekdays on two paths. The heuristics-based route change detection algorithm detected 71\% of the visible layer 2 route changes and did not detect the events that occurred too close together in time or the events for which the minimum RTT change was small. A practical model-based route change detector named the parameter unaware detector (PUD) is also developed in this deissertation because it was expected that model-based detectors would perform better than the heuristics-based detector. Also, the optimal detector named the parameter aware detector (PAD) is developed and is useful because it provides the upper bound on the performance of any detector. The analysis for predicting the performance of PAD is another important contribution of this work. Simulation results prove that the model-based PUD algorithm has acceptable performance over a larger region of the parameter space than the heuristics-based algorithm and this difference in performance increases with an increase in the window size. Also, it is shown that both practical algorithms have a smaller acceptable performance region compared to the optimal algorithm. The model-based algorithms proposed in this dissertation are based on the assumption that RTTs have a Gamma density function. This Gamma distribution assumption may not hold when there are wireless links in the path. A study of CDMA 1xEVDO networks was initiated to understand the delay characteristics of these networks. During this study, it was found that the widely deployed proportional-fair (PF) scheduler can be corrupted accidentally or deliberately to cause RTM impairments. This is demonstrated using measurements conducted over both in-lab and deployed CDMA 1xEVDO networks. A new variant to PF that solves the impairment vulnerability of the PF algorithm is proposed and evaluated using ns-2 simulations. It is shown that this new scheduler solution together with a new adaptive-alpha initialization stratergy reduces the starvation problem of the PF algorithm