DESIGN, DEVELOPMENT AND EVALUATION OF AN ADAPTIVE AND STANDARDIZED RTP/RTCP-BASED IDMS SOLUTION

Nowadays, we are witnessing a transition from physical togetherness towards networked togetherness around media content. Novel forms of shared media experiences are gaining momentum, allowing geographically distributed users to concurrently consume the same media content while socially interacting (e.g., via text, audio or video chat). Relevant use cases are, for example, Social TV, networked games and multi-party conferencing. However, realizing enjoyable shared media services faces many challenges. In particular, a key technological enabler is the concurrent synchronization of the media playout across multiple locations, which is known as Inter-Destination Multimedia Synchronization (IDMS). This PhD thesis presents an inter-operable, adaptive and accurate IDMS solution, based on extending the capabilities of RTP/RTCP standard protocols (RFC 3550). Concretely, two new RTCP messages for IDMS have been defined to carry out the necessary information to achieve IDMS. Such RTCP extensions have been standardized within the IETF, in RFC 7272. In addition, novel standard-compliant Early Event-Driven (EED) RTCP feedback reporting mechanisms have been also designed to enhance the performance in terms of interactivity, flexibility, dynamism and accuracy when performing IDMS. The designed IDMS solution makes use of globally synchronized clocks (e.g., using NTP) and can adopt different (centralized and distributed) architectural schemes to exchange the RTCP messages for IDMS. This allows efficiently providing IDMS in a variety of networked scenarios and applications, with different requirements (e.g., interactivity, scalability, robustness…) and available resources (e.g., bandwidth, latency, multicast support…). Likewise, various monitoring and control algorithms, such as dynamic strategies for selecting the reference timing to synchronize with, and fault tolerance mechanisms, have been added. Moreover, the proposed IDMS solution includes a novel Adaptive Media Playout (AMP) technique, which aims to smoothly adjust the media playout rate, within perceptually tolerable ranges, every time an asynchrony threshold is exceeded. Prototypes of the IDMS solution have been implemented in both a simulation and in real media framework. The evaluation tests prove the consistent behavior and the satisfactory performance of each one of the designed components (e.g.,protocols, architectural schemes, master selection policies, adjustment techniques…). Likewise, comparison results between the different developed alternatives for such components are also provided. In general, the obtained results demonstrate the ability of this RTP/RTCP-based IDMS solution to concurrently and independently maintain an overall synchronization status (within allowable limits) in different logical groups of users, while avoiding annoying playout discontinuities and hardly increasing the computation and traffic load.Montagut Climent, MA. (2015). DESIGN, DEVELOPMENT AND EVALUATION OF AN ADAPTIVE AND STANDARDIZED RTP/RTCP-BASED IDMS SOLUTION [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/48549TESISPremios Extraordinarios de tesis doctorale

Synchronization of streamed audio between multiple playback devices over an unmanaged IP network

When designing and implementing a prototype supporting inter-destination media synchronization – synchronized playback between multiple devices receiving the same stream – there are a lot of aspects that need to be considered, especially when working with unmanaged networks. Not only is a proper streaming protocol essential, but also a way to obtain and maintain the synchronization of the clocks of the devices. The thesis had a few constraints, namely that the server producing the stream should be written for the .NET-platform and that the clients receiving it should be using the media framework GStreamer. This framework provides methods for both achieving synchronization as well as resynchronization. As the provided resynchro- nization methods introduced distortions in the audio, an alternative method was implemented. This method focused on minimizing the distortions, thus maintain- ing a smooth playback. After the prototype had been implemented, it was tested to see how well it performed under the influence of packet loss and delay. The accuracy of the synchronization was also tested under optimal conditions using two different time synchronization protocols. What could be concluded from this was that a good synchronization could be maintained on unloaded networks using the proposed method, but when introducing delay the prototype struggled more. This was mainly due to the usage of the Network Time Protocol (NTP), which is known to perform badly on networks with asymmetric paths.When working with synchronized playback it is not enough just obtain- ing it – it also needs to be maintained. Implementing a prototype thus involves many parts ranging from choosing a proper streaming protocol, to handling glitch free resynchronization of audio. Synchronization between multiple speakers has a wide area of application, ranging from home entertainment solutions to big malls where announcements should appear synchronized over the entire perimeter. In order to achieve this, two main parts are involved: the streaming of the audio, and the actual synchronization. The streaming itself poses problems mostly since the prototype should not only work on dedicated networks, but rather on all kinds, such as the Internet. As the information over these networks are transmitted in packets, and the path from source to destination crosses many sub networks, the packets may be delayed or even lost. This may create an audible distortion in the playback. The next part is the synchronization. This is most easily achieved by putting a time on each packet stating when in the future it should be played out. If then all receivers play it back at the specified time, synchronization is achieved. This however requires that all the receivers share the idea of when a specific time is – the clocks at all the receivers must be synchronized. By using existing software and hardware solutions, such as the Network Time Protocol (NTP) or the Precision Time Protocol (PTP), this can be accomplished. The accuracy of the synchronization is therefore partly dependent on how well these solutions work. Another valid aspect is how accurate the synchronization must be for the sound to be perceived as synchronized by humans. This is usually in the range of a few tens of milliseconds to five milliseconds depending on the sound. When a global time has been distributed to all receivers, matters get more complicated as there is more than one clock to consider at each receiver. Apart from the previously mentioned clock, now called the ’system clock’, there is also an audio clock, which is a hardware clock positioned on the sound card. This audio clock decides the rate at which media is played out. Altering the system clock to synchronize it to a common time is one thing, but altering the audio clock while media is being played will inevitably mean a jump in the playback, and thus a distortion. Although an initial synchronization can be achieved, the two clocks will over time tick in slightly different pace, thus drifting away from each other. This creates a need for the audio clock to continuously correct itself to follow the system clock. In the media framework GStreamer, used for handling the media at the re- ceivers, two alternatives to solve the correction problem were available. Quick evaluations of these two methods however showed that either audible glitches or ’oscillations’ occurred in the sound, when the clocks were corrected. A new method, which basically combines the two existing, was therefore implemented. With this method the audio clock is continuously corrected, but in a smaller and less aggressive way. Listening tests revealed much smaller, often not audible, distortions, while the synchronization performance was at par with the existing methods. More thorough testing showed that the synchronization over networks with light traffic was in the microsecond-range, thus far below the threshold of what will appear as synchronized. During worse conditions – simulated hostile environments – the synchronization quickly reached unacceptable levels though. This was due to the previously mentioned NTP, and not the implemented method on the other hand

MediaSync: Handbook on Multimedia Synchronization

This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences