Quality of experience in telemeetings and videoconferencing: a comprehensive survey

Telemeetings such as audiovisual conferences or virtual meetings play an increasingly important role in our professional and private lives. For that reason, system developers and service providers will strive for an optimal experience for the user, while at the same time optimizing technical and financial resources. This leads to the discipline of Quality of Experience (QoE), an active field originating from the telecommunication and multimedia engineering domains, that strives for understanding, measuring, and designing the quality experience with multimedia technology. This paper provides the reader with an entry point to the large and still growing field of QoE of telemeetings, by taking a holistic perspective, considering both technical and non-technical aspects, and by focusing on current and near-future services. Addressing both researchers and practitioners, the paper first provides a comprehensive survey of factors and processes that contribute to the QoE of telemeetings, followed by an overview of relevant state-of-the-art methods for QoE assessment. To embed this knowledge into recent technology developments, the paper continues with an overview of current trends, focusing on the field of eXtended Reality (XR) applications for communication purposes. Given the complexity of telemeeting QoE and the current trends, new challenges for a QoE assessment of telemeetings are identified. To overcome these challenges, the paper presents a novel Profile Template for characterizing telemeetings from the holistic perspective endorsed in this paper

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc. Assessing the quality of ODVs is significant for service-providers to improve the user's Quality of Experience (QoE). However, most existing quality assessment studies for ODVs only focus on the visual distortions of videos, while ignoring that the overall QoE also depends on the accompanying audio signals. In this paper, we first establish a large-scale audio-visual quality assessment dataset for omnidirectional videos, which includes 375 distorted omnidirectional audio-visual (A/V) sequences generated from 15 high-quality pristine omnidirectional A/V contents, and the corresponding perceptual audio-visual quality scores. Then, we design three baseline methods for full-reference omnidirectional audio-visual quality assessment (OAVQA), which combine existing state-of-the-art single-mode audio and video QA models via multimodal fusion strategies. We validate the effectiveness of the A/V multimodal fusion method for OAVQA on our dataset, which provides a new benchmark for omnidirectional QoE evaluation. Our dataset is available at https://github.com/iamazxl/OAVQA.Comment: 12 pages, 5 figures, to be published in CICAI202

Quality of experience for 3-d immersive media streaming

Recent advances in media capture and processing technologies have enabled new forms of true 3-D media content that increase the degree of user immersion. The demand for more engaging forms of entertainment means that content distributors and broadcasters need to fine-tune their delivery mechanisms over the Internet as well as develop new models for quantifying and predicting user experience of these new forms of content. In the work described in this paper, we undertake one of the first studies into the quality of experience (QoE) of real-time 3-D media content streamed to virtual reality (VR) headsets for entertainment purposes, in the context of game spectating. Our focus is on tele-immersive media that embed real users within virtual environments of interactive games. A key feature of engaging and realistic experiences in full 3-D media environments, is allowing users unrestricted viewpoints. However, this comes at the cost of increased network bandwidth and the need of limiting network effects in order to transmit a realistic, real-time representation of the participants. The visual quality of 3-D media is affected by geometry and texture parameters while the temporal aspects of smooth movement and synchronization are affected by lag introduced by network transmission effects. In this paper, we investigate varying network conditions for a set of tele-immersive media sessions produced in a range of visual quality levels. Further, we investigate user navigation issues that inhibit free viewpoint VR spectating of live 3-D media. After reporting on a study with multiple users we analyze the results and assess the overall QoE with respect to a range of visual quality and latency parameters. We propose a neural network QoE prediction model for 3-D media, constructed from a combination of visual and network parameters

Creating the Internet of Augmented Things: An Open-Source Framework to Make IoT Devices and Augmented and Mixed Reality Systems Talk to Each Other

[Abstract] Augmented Reality (AR) and Mixed Reality (MR) devices have evolved significantly in the last years, providing immersive AR/MR experiences that allow users to interact with virtual elements placed on the real-world. However, to make AR/MR devices reach their full potential, it is necessary to go further and let them collaborate with the physical elements around them, including the objects that belong to the Internet of Things (IoT). Unfortunately, AR/MR and IoT devices usually make use of heterogeneous technologies that complicate their intercommunication. Moreover, the implementation of the intercommunication mechanisms requires involving specialized developers with have experience on the necessary technologies. To tackle such problems, this article proposes the use of a framework that makes it easy to integrate AR/MR and IoT devices, allowing them to communicate dynamically and in real time. The presented AR/MR-IoT framework makes use of standard and open-source protocols and tools like MQTT, HTTPS or Node-RED. After detailing the inner workings of the framework, it is illustrated its potential through a practical use case: a smart power socket that can be monitored and controlled through Microsoft HoloLens AR/MR glasses. The performance of such a practical use case is evaluated and it is demonstrated that the proposed framework, under normal operation conditions, enables to respond in less than 100 ms to interaction and data update requests.Xunta de Galicia; IN853B-2018/0

A survey on Quality of Experience of Virtual and Augmented Reality environments

Οι εφαρμογές εικονικής και επαυξημένης πραγματικότητας αποτελούν μια νέα και πολλά υποσχόμενη τεχνολογία, με εφαρμογές στην ιατρική, την εκπαίδευση, τα βιντεοπαιχνίδια, το ηλεκτρονικό εμπόριο και πολλές άλλες. Αυτή η τεχνολογία θέτει μια νέα πρόκληση για τους σχεδιαστές εφαρμογών, καθώς και για τους παρόχους υπηρεσιών δικτύου, επειδή είναι εντατική ως προς τους απαιτούμενους πόρους ώστε να είναι σε θέση να παρέχει συναρπαστική εμπειρία στους χρήστες της. Αυτή η πρόκληση γίνεται ακόμη πιο δύσκολη στα δίκτυα κινητής τηλεφωνίας, λόγω παραγόντων που είναι δύσκολο να μοντελοποιηθούν και να προβλεφθούν, όπως η κινητικότητα, η στρατηγική μεταβίβασης και η κατανομή πόρων. Αυτή η διπλωματική φιλοδοξεί να παράσχει μια ανασκόπηση των τεχνικών και μεθόδων εκτίμησης της ποιότητας της εμπειρίας (QoE) και των μεθόδων που έχουν αναπτυχθεί γύρω από αυτές τις εφαρμογές. Στην πρώτη ενότητα, εξετάζουμε τις στρατηγικές παροχής QoE για εφαρμογές εικονικής πραγματικότητας. Αυτή η ενότητα εξετάζει αρκετές περιπτώσεις εφαρμογών εικονικής πραγματικότητας, όπως ένα λογισμικό προσομοίωσης βαρέων μηχανημάτων, μια εκπαιδευτική εφαρμογή και άλλες ψηφιακές εφαρμογές εμβύθισης. Το εύρος και η ποικιλία των εφαρμογών και των μεθόδων εκτίμησης της ποιότητας της εμπειρίας οδηγούν σε αντικρουόμενα συμπεράσματα σχετικά με τις μεθόδους αξιολόγησης QoE. Στην επόμενη ενότητα αναφερόμαστε σε εφαρμογές επαυξημένης πραγματικότητας, και πάλι με μια αναφορά σε μια μεγάλη ποικιλία εφαρμογών, όπως ένας βοηθός επαυξημένης πραγματικότητας, βιντεοπαιχνίδια επαυξημένης πραγματικότητας και άλλες ψηφιακές εφαρμογές. Τα συμπεράσματα σε αυτήν την ενότητα είναι πιο ισχυρά και τα συναισθήματα των ανθρώπων μπορούν να σχηματίσουν πιο ουσιαστικά πορίσματα. Στην τελευταία ενότητα διερευνούμε την QoE σε εφαρμογές εικονικής και επαυξημένης πραγματικότητας για κινητές συσκευές και κινητά δίκτυα. Σε αυτό το μέρος ασχολούμαστε με πιο τεχνικές πτυχές όπως η διαχείριση της κινητικότητας, οι στρατηγικές μεταβίβασης και οι αλγόριθμοι κατανομής πόρων και ο αντίκτυπος που έχουν αυτοί στην εμπειρία των χρηστών.Virtual and augmented reality applications constitute a new and promising technology, with applications in medicine, education, gaming, e-commerce and many more. This technology poses a new challenge to application designers, as well as network service providers, because it is resource intensive in order to be able to provide the immersive experience to its users. This task becomes even more challenging in mobile networks, due to factors that are difficult to be modeled and predicted, such as mobility, handoff strategy and resource allocation. This thesis aspires to provide a review of Quality of Experience (QoE) estimation and provision techniques and methods that have been developed around these applications. In the first section, we review QoE provisioning strategies for virtual reality applications. This section examines some corner cases of augmented reality applications, such as a heavy machinery simulation software, an educational application, and many digital immersive applications. The scope and diversity of applications and implementation methods lead to some conflicting conclusions in relation to QoE evaluation methods. In the next section we refer to augmented reality applications, again with a reference to a wide variety of applications such as an augmented reality task assistant, augmented reality video games and digital immersive applications. The conclusions in this section are more robust and peoples’ feelings can form more meaningful aggregations. In the last section we investigate the QoE in virtual and augmented reality applications when these applications are implemented in mobile devices. Τhis part is concerned with more technical aspects such as mobility management, handoff strategies and resource allocation algorithms and their impact on users’ experience

Beyond reality - extending a presentation trainer with an immersive VR module

The development of multimodal sensor-based applications designed to support learners with the improvement of their skills is expensive since most of these applications are tailor-made and built from scratch. In this paper, we show how the Presentation Trainer (PT), a multimodal sensor-based application designed to support the development of public speaking skills, can be modularly extended with a Virtual Reality real-time feedback module (VR module), which makes usage of the PT more immersive and comprehensive. The described study consists of a formative evaluation and has two main objectives. Firstly, a technical objective is concerned with the feasibility of extending the PT with an immersive VR Module. Secondly, a user experience objective focuses on the level of satisfaction of interacting with the VR extended PT. To study these objectives, we conducted user tests with 20 participants. Results from our test show the feasibility of modularly extending existing multimodal sensor-based applications, and in terms of learning and user experience, results indicate a positive attitude of the participants towards using the application (PT+VR module). (DIPF/Orig.

Exploiting fashion x-commerce through the empowerment of voice in the fashion virtual reality arena. Integrating voice assistant and virtual reality technologies for fashion communication

The ongoing development of eXtended Reality (XR) technologies is supporting a rapid increase of their performances along with a progressive decrease of their costs, making them more and more attractive for a large class of consumers. As a result, their widespread use is expected within the next few years. This may foster new opportunities for e-commerce strategies, giving birth to an XR-based commerce (x-commerce) ecosystem. With respect to web and mobile-based shopping experiences, x-commerce could more easily support brick-and-mortar store-like experiences. One interesting and consolidated one amounts to the interactions among customers and shop assistants inside fashion stores. In this work, we concentrate on such aspects with the design and implementation of an XR-based shopping experience, where vocal dialogues with an Amazon Alexa virtual assistant are supported, to experiment with a more natural and familiar contact with the store environment. To verify the validity of such an approach, we asked a group of fashion experts to try two different XR store experiences: with and without the voice assistant integration. The users are then asked to answer a questionnaire to rate their experiences. The results support the hypothesis that vocal interactions may contribute to increasing the acceptance and comfortable perception of XR-based fashion shopping

Mulsemedia: State of the art, perspectives, and challenges

Mulsemedia-multiple sensorial media-captures a wide variety of research efforts and applications. This article presents a historic perspective on mulsemedia work and reviews current developments in the area. These take place across the traditional multimedia spectrum-from virtual reality applications to computer games-as well as efforts in the arts, gastronomy, and therapy, to mention a few. We also describe standardization efforts, via the MPEG-V standard, and identify future developments and exciting challenges the community needs to overcome