Audiovisual temporal integration in reverberant environments

AbstractWith teleconferencing becoming more accessible as a communication platform, researchers are working to understand the consequences of the interaction between human perception and this unfamiliar environment. Given the enclosed space of a teleconference room, along with the physical separation between the user, microphone and speakers, the transmitted audio often becomes mixed with the reverberating auditory components from the room. As a result, the audio can be perceived as smeared in time, and this can affect the user experience and perceived quality. Moreover, other challenges remain to be solved. For instance, during encoding, compression and transmission, the audio and video streams are typically treated separately. Consequently, the signals are rarely perfectly aligned and synchronous. In effect, timing affects both reverberation and audiovisual synchrony, and the two challenges may well be inter-dependent. This study explores the temporal integration of audiovisual continuous speech and speech syllables, along with a non-speech event, across a range of asynchrony levels for different reverberation conditions. Non-reverberant stimuli are compared to stimuli with added reverberation recordings. Findings reveal that reverberation does not affect the temporal integration of continuous speech. However, reverberation influences the temporal integration of the isolated speech syllables and the action-oriented event, with perceived subjective synchrony skewed towards audio lead asynchrony and away from the more common audio lag direction. Furthermore, less time is spent on simultaneity judgements for the longer sequences when the temporal offsets get longer and when reverberation is introduced, suggesting that both asynchrony and reverberation add to the demands of the task

Applicability of group communication for increased scalability in MMOGs

Massive multiplayer online games (MMOGs) are today the driving factor for the development of distributed interactive applications, and they are increasing in size and complex-ity. Even a small MMOG supports thousands of players, the biggest support hundreds of thousands of concurrent players. Since they are typically built as strict client-server systems, they suffer from the inherent scalability problem of the architecture. Computing power and bandwidth limita-tions close to the server limit the possible number of players. Also, the latency of communication between players through the server will be higher than using direct communication. In the paper, we address these issues and investigate im-provement options. A typical MMOG consists of a virtual world with a con-cept of time and space that is similar to the real world. In it, players are represented by avatars. Only subsets of these avatars interact with each other at any given time. This allows us to divide them into groups, and communication among group members becomes a multi-party communica-tion problem. Thus, to reduce resource consumption, we compare the performance of several algorithms for group communication with the current central server approach. We use overlay multicast as the means of providing group communication, and research algorithms for creating short-est path trees, spanning trees, delay-bounded spanning trees and, more specific, applying Steiner tree heuristics. Our experimental results indicate that different approaches are useful to reduce resource consumption while achieving a good perceived quality under varying conditions, such as frequent changes in group membership and the demand for low latency. 1

Toward Free and Open Source Film Projection for Digital Cinema

International audienceCinema industry has chosen Digital Cinema Package (DCP) as encoding format for the distribution of digital films. DCP uses JPEG2000 for video compression. An efficient implementation of coding and decoding for this format is complex, however. Currently deployed equipment is expensive and has high maintenance costs, preventing art-house cinema theaters from acquiring it. Therefore, we conduct this research activity in cooperation with Utopia cinemas, a group of art-house cinemas, whose main requirement (besides functional ones) is to provide Free and Open Source Software (FOSS). This paper presents a solution that achieves real-time JPEG2000 decoding and DCP presentation based on widespread open source multimedia tools, namely VLC and libavcodec library. We present the improvements that were made in VLC to support the DCP packaging format, as well as details on JPEG2000 decoding inside libavcodec (optimization and lossy decoding). We also evaluate the performance of the decoding chai

Delay Sensitivity Classification of Cloud Gaming Content

Cloud Gaming is an emerging service that catches growing interest in the research community as well as industry. While the paradigm shift from a game execution on clients to streaming games from the cloud offers a variety of benefits, the new services also require a highly reliable and low latency network to achieve a satisfying Quality of Experience (QoE) for its users. Using a cloud gaming service with high latency would harm the interaction of the user with the game, leading to a decrease in playing performance and thus frustration of players. However, the negative effect of delay on gaming QoE depends strongly on the game content. At a certain level of delay, a slow-paced card game is typically not as delay sensitive as a shooting game. For optimal resource allocation and quality estimation, it is highly important for cloud providers, game developers, and network planners to consider the impact of the game content. This paper contributes to a better understanding of the delay impact on QoE for cloud gaming applications by identifying game characteristics influencing the delay perception of users. In addition, an expert evaluation methodology to quantify these characteristics, as well as a delay sensitivity classification based on a decision tree is presented. The ratings of 14 experts for the quantification indicated an excellent level of agreement which demonstrates the reliability of the proposed method. Additionally, the decision tree reached an accuracy of 86.6 % on determining the delay sensitivity classes which were derived from a large dataset of subjective input quality ratings during a series of experiments.Comment: Accepted In International Workshop on Immersive Mixed and Virtual Environment Systems 2020. ACM, Istanbul, Turke

The Nornir run-time system for parallel programs using Kahn process networks on multi-core machines – A flexible alternative to MapReduce

Even though shared-memory concurrency is a paradigm frequently used for developing parallel applications on small- and middle-sized machines, experience has shown that it is hard to use. This is largely caused by synchronization primitives which are low-level, inherently non-deterministic, and, consequently, non-intuitive to use. In this paper, we present the Nornir run-time system. Nornir is comparable to well-known frameworks such as MapReduce and Dryad that are recognized for their efficiency and simplicity. Unlike these frameworks, Nornir also supports process structures containing branches and cycles. Nornir is based on the formalism of Kahn process networks, which is a shared-nothing, message-passing model of concurrency. We deem this model a simple and deterministic alternative to shared-memory concurrency. Experiments with real and synthetic benchmarks on up to 8 CPUs show that performance in most cases scales almost linearly with the number of CPUs, when not limited by data dependencies. We also show that the modeling flexibility allows Nornir to outperform its MapReduce counterparts using well-known benchmarks. This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited

Lecture de DCP pour le cinéma numérique avec le lecteur multimédia VLC et libav/ffmpeg

National audienceLes salles de cinéma sont passées à l’ère numérique. Pour la distribution des copies numériques de films, la Digital Cinema Initiative (DCI) a choisi d’encoder les films au format Digital Cinema Package (DCP). Notre activité est menée en collaboration avec les cinémas Utopia, qui forment un réseau de salles indépendantes. La principale exigence d’Utopia est de fournir des logiciels libres et gratuits pour le cinéma numérique. Le format DCP utilise la compression JPEG2000 pour la vidéo, en raison de son taux de compression élevé pour les grandes images. Réaliser une implémentation efficace de codage et de décodage de ce format est complexe. Néanmoins, nous proposons une implémentation améliorée du décodage pour la projection dans les salles obscures. Les équipements actuellement déployés dans les salles sont chers à l’achat et ont un coût de maintenance élevé, ce qui empêche les petites salles indépendantes de s’équiper. Notre but est de proposer une solution logicielle adaptée à la projection des DCP. Cet article présente une solution qui réalise le décodage en temps réel et la projection, en s’appuyant sur des outils multimédia libres et standards comme le lecteur VLC et les bibliothèques libav/ffmpeg. Nous présentons les améliorations implémentées dans VLC pour supporter les DCP, qui incluent la lecture des fichiers et la synchronisation entre audio et vidéo. Nous détaillons ensuite la réalisation du décodeur JPEG2000 dans libav/ffmpeg. Pour finir, nous évaluons les performances de lecture atteintes

Expert driven semi-supervised elucidation tool for medical endoscopic videos

In this paper, we present a novel application for elucidating all kind of videos that require expert knowledge, e.g., sport videos, medical videos etc., focusing on endoscopic surgery and video capsule endoscopy. In the medical domain, the knowledge of experts for tagging and interpretation of videos is of high value. As a result of the stressful working environment of medical doctors, they often simply do not have time for extensive annotations. We therefore present a semi-supervised method to gather the annotations in a very easy and time saving way for the experts and we show how this information can be used later on

Be your own cameraman: real-time support for zooming and panning into stored and live panoramic video

International audienceHigh-resolution panoramic video with a wide eld-of-view is popular in many contexts. However, in many examples, like surveillance and sports, it is often desirable to zoom and pan into the generated video. A challenge in this respect is real-time support, but in this demo, we present an end-to- end real-time panorama system with interactive zoom and panning. Our system installed at Alfheim stadium, a Nor- wegian premier league soccer team, generates a cylindrical panorama from ve 2K cameras live where the perspective is corrected in real-time when presented to the client. This gives a better and more natural zoom compared to existing systems using perspective panoramas and zoom operations using plain crop. Our experimental results indicate that vir- tual views can be generated far below the frame-rate thresh- old, i.e., on a GPU, the processing requirement per frame is about 10 milliseconds. The proposed demo lets participants interactively zoom and pan into stored panorama videos generated at Alfheim stadium and from a live 2-camera array on-site

Reducing Internet Latency : A Survey of Techniques and their Merit

Bob Briscoe, Anna Brunstrom, Andreas Petlund, David Hayes, David Ros, Ing-Jyh Tsang, Stein Gjessing, Gorry Fairhurst, Carsten Griwodz, Michael WelzlPeer reviewedPreprin