A survey on Quality of Experience of Virtual and Augmented Reality environments

Οι εφαρμογές εικονικής και επαυξημένης πραγματικότητας αποτελούν μια νέα και πολλά υποσχόμενη τεχνολογία, με εφαρμογές στην ιατρική, την εκπαίδευση, τα βιντεοπαιχνίδια, το ηλεκτρονικό εμπόριο και πολλές άλλες. Αυτή η τεχνολογία θέτει μια νέα πρόκληση για τους σχεδιαστές εφαρμογών, καθώς και για τους παρόχους υπηρεσιών δικτύου, επειδή είναι εντατική ως προς τους απαιτούμενους πόρους ώστε να είναι σε θέση να παρέχει συναρπαστική εμπειρία στους χρήστες της. Αυτή η πρόκληση γίνεται ακόμη πιο δύσκολη στα δίκτυα κινητής τηλεφωνίας, λόγω παραγόντων που είναι δύσκολο να μοντελοποιηθούν και να προβλεφθούν, όπως η κινητικότητα, η στρατηγική μεταβίβασης και η κατανομή πόρων. Αυτή η διπλωματική φιλοδοξεί να παράσχει μια ανασκόπηση των τεχνικών και μεθόδων εκτίμησης της ποιότητας της εμπειρίας (QoE) και των μεθόδων που έχουν αναπτυχθεί γύρω από αυτές τις εφαρμογές. Στην πρώτη ενότητα, εξετάζουμε τις στρατηγικές παροχής QoE για εφαρμογές εικονικής πραγματικότητας. Αυτή η ενότητα εξετάζει αρκετές περιπτώσεις εφαρμογών εικονικής πραγματικότητας, όπως ένα λογισμικό προσομοίωσης βαρέων μηχανημάτων, μια εκπαιδευτική εφαρμογή και άλλες ψηφιακές εφαρμογές εμβύθισης. Το εύρος και η ποικιλία των εφαρμογών και των μεθόδων εκτίμησης της ποιότητας της εμπειρίας οδηγούν σε αντικρουόμενα συμπεράσματα σχετικά με τις μεθόδους αξιολόγησης QoE. Στην επόμενη ενότητα αναφερόμαστε σε εφαρμογές επαυξημένης πραγματικότητας, και πάλι με μια αναφορά σε μια μεγάλη ποικιλία εφαρμογών, όπως ένας βοηθός επαυξημένης πραγματικότητας, βιντεοπαιχνίδια επαυξημένης πραγματικότητας και άλλες ψηφιακές εφαρμογές. Τα συμπεράσματα σε αυτήν την ενότητα είναι πιο ισχυρά και τα συναισθήματα των ανθρώπων μπορούν να σχηματίσουν πιο ουσιαστικά πορίσματα. Στην τελευταία ενότητα διερευνούμε την QoE σε εφαρμογές εικονικής και επαυξημένης πραγματικότητας για κινητές συσκευές και κινητά δίκτυα. Σε αυτό το μέρος ασχολούμαστε με πιο τεχνικές πτυχές όπως η διαχείριση της κινητικότητας, οι στρατηγικές μεταβίβασης και οι αλγόριθμοι κατανομής πόρων και ο αντίκτυπος που έχουν αυτοί στην εμπειρία των χρηστών.Virtual and augmented reality applications constitute a new and promising technology, with applications in medicine, education, gaming, e-commerce and many more. This technology poses a new challenge to application designers, as well as network service providers, because it is resource intensive in order to be able to provide the immersive experience to its users. This task becomes even more challenging in mobile networks, due to factors that are difficult to be modeled and predicted, such as mobility, handoff strategy and resource allocation. This thesis aspires to provide a review of Quality of Experience (QoE) estimation and provision techniques and methods that have been developed around these applications. In the first section, we review QoE provisioning strategies for virtual reality applications. This section examines some corner cases of augmented reality applications, such as a heavy machinery simulation software, an educational application, and many digital immersive applications. The scope and diversity of applications and implementation methods lead to some conflicting conclusions in relation to QoE evaluation methods. In the next section we refer to augmented reality applications, again with a reference to a wide variety of applications such as an augmented reality task assistant, augmented reality video games and digital immersive applications. The conclusions in this section are more robust and peoples’ feelings can form more meaningful aggregations. In the last section we investigate the QoE in virtual and augmented reality applications when these applications are implemented in mobile devices. Τhis part is concerned with more technical aspects such as mobility management, handoff strategies and resource allocation algorithms and their impact on users’ experience

Networking Architecture and Key Technologies for Human Digital Twin in Personalized Healthcare: A Comprehensive Survey

Digital twin (DT), refers to a promising technique to digitally and accurately represent actual physical entities. One typical advantage of DT is that it can be used to not only virtually replicate a system's detailed operations but also analyze the current condition, predict future behaviour, and refine the control optimization. Although DT has been widely implemented in various fields, such as smart manufacturing and transportation, its conventional paradigm is limited to embody non-living entities, e.g., robots and vehicles. When adopted in human-centric systems, a novel concept, called human digital twin (HDT) has thus been proposed. Particularly, HDT allows in silico representation of individual human body with the ability to dynamically reflect molecular status, physiological status, emotional and psychological status, as well as lifestyle evolutions. These prompt the expected application of HDT in personalized healthcare (PH), which can facilitate remote monitoring, diagnosis, prescription, surgery and rehabilitation. However, despite the large potential, HDT faces substantial research challenges in different aspects, and becomes an increasingly popular topic recently. In this survey, with a specific focus on the networking architecture and key technologies for HDT in PH applications, we first discuss the differences between HDT and conventional DTs, followed by the universal framework and essential functions of HDT. We then analyze its design requirements and challenges in PH applications. After that, we provide an overview of the networking architecture of HDT, including data acquisition layer, data communication layer, computation layer, data management layer and data analysis and decision making layer. Besides reviewing the key technologies for implementing such networking architecture in detail, we conclude this survey by presenting future research directions of HDT

When Internet of Things meets Metaverse: Convergence of Physical and Cyber Worlds

In recent years, the Internet of Things (IoT) is studied in the context of the Metaverse to provide users immersive cyber-virtual experiences in mixed reality environments. This survey introduces six typical IoT applications in the Metaverse, including collaborative healthcare, education, smart city, entertainment, real estate, and socialization. In the IoT-inspired Metaverse, we also comprehensively survey four pillar technologies that enable augmented reality (AR) and virtual reality (VR), namely, responsible artificial intelligence (AI), high-speed data communications, cost-effective mobile edge computing (MEC), and digital twins. According to the physical-world demands, we outline the current industrial efforts and seven key requirements for building the IoT-inspired Metaverse: immersion, variety, economy, civility, interactivity, authenticity, and independence. In addition, this survey describes the open issues in the IoT-inspired Metaverse, which need to be addressed to eventually achieve the convergence of physical and cyber worlds.info:eu-repo/semantics/publishedVersio

User experience and robustness in social virtual reality applications

Cloud-based applications that rely on emerging technologies such as social virtual reality are increasingly being deployed at high-scale in e.g., remote-learning, public safety, and healthcare. These applications increasingly need mechanisms to maintain robustness and immersive user experience as a joint consideration to minimize disruption in service availability due to cyber attacks/faults. Specifically, effective modeling and real-time adaptation approaches need to be investigated to ensure that the application functionality is resilient and does not induce undesired cybersickness levels. In this thesis, we investigate a novel ‘DevSecOps' paradigm to jointly tune both the robustness and immersive performance factors in social virtual reality application design/operations. We characterize robustness factors considering Security, Privacy and Safety (SPS), and immersive performance factors considering Quality of Application, Quality of Service, and Quality of Experience (3Q). We achieve “harmonized security and performance by design” via modeling the SPS and 3Q factors in cloud-hosted applications using attack-fault trees (AFT) and an accurate quantitative analysis via formal verification techniques i.e., statistical model checking (SMC). We develop a real-time adaptive control capability to manage SPS/3Q issues affecting a critical anomaly event that induces undesired cybersickness. This control capability features a novel dynamic rule-based approach for closed-loop decision making augmented by a knowledge base for the SPS/3Q issues of individual and/or combination events. Correspondingly, we collect threat intelligence on application and network based cyber-attacks that disrupt immersiveness, and develop a multi-label K-NN classifier as well as statistical analysis techniques for critical anomaly event detection. We validate the effectiveness of our solution approach in a real-time cloud testbed featuring vSocial, a social virtual reality based learning environment that supports delivery of Social Competence Intervention (SCI) curriculum for youth. Based on our experiment findings, we show that our solution approach enables: (i) identification of the most vulnerable components that impact user immersive experience to formally conduct risk assessment, (ii) dynamic decision making for controlling SPS/3Q issues inducing undesirable cybersickness levels via quantitative metrics of user feedback and effective anomaly detection, and (iii) rule-based policies following the NIST SP 800-160 principles and cloud-hosting recommendations for a more secure, privacy-preserving, and robust cloud-based application configuration with satisfactory immersive user experience.Includes bibliographical references (pages 133-146)

Feature Space Augmentation: Improving Prediction Accuracy of Classical Problems in Cognitive Science and Computer Vison

The prediction accuracy in many classical problems across multiple domains has seen a rise since computational tools such as multi-layer neural nets and complex machine learning algorithms have become widely accessible to the research community. In this research, we take a step back and examine the feature space in two problems from very different domains. We show that novel augmentation to the feature space yields higher performance. Emotion Recognition in Adults from a Control Group: The objective is to quantify the emotional state of an individual at any time using data collected by wearable sensors. We define emotional state as a mixture of amusement, anger, disgust, fear, sadness, anxiety and neutral and their respective levels at any time. The generated model predicts an individual’s dominant state and generates an emotional spectrum, 1x7 vector indicating levels of each emotional state and anxiety. We present an iterative learning framework that alters the feature space uniquely to an individual’s emotion perception, and predicts the emotional state using the individual specific feature space. Hybrid Feature Space for Image Classification: The objective is to improve the accuracy of existing image recognition by leveraging text features from the images. As humans, we perceive objects using colors, dimensions, geometry and any textual information we can gather. Current image recognition algorithms rely exclusively on the first 3 and do not use the textual information. This study develops and tests an approach that trains a classifier on a hybrid text based feature space that has comparable accuracy to the state of the art CNN’s while being significantly inexpensive computationally. Moreover, when combined with CNN’S the approach yields a statistically significant boost in accuracy. Both models are validated using cross validation and holdout validation, and are evaluated against the state of the art

RCEA-360VR: Real-time, continuous emotion annotation in 360◦ VR videos for collecting precise viewport-dependent ground truth labels

Precise emotion ground truth labels for 360◦ virtual reality (VR) video watching are essential for fne-grained predictions under varying viewing behavior. However, current annotation techniques either rely on post-stimulus discrete self-reports, or real-time, con- tinuous emotion annotations (RCEA) but only for desktop/mobile settings. We present RCEA for 360◦ VR videos (RCEA-360VR), where we evaluate in a controlled study (N=32) the usability of two peripheral visualization techniques: HaloLight and DotSize. We furthermore develop a method that considers head movements when fusing labels. Using physiological, behavioral, and subjective measures, we show that (1) both techniques do not increase users’ workload, sickness, nor break presence (2) our continuous valence and arousal annotations are consistent with discrete within-VR and original stimuli ratings (3) users exhibit high similarity in viewing behavior, where fused ratings perfectly align with intended labels. Our work contributes usable and efective techniques for collecting fne-grained viewport-dependent emotion labels in 360◦ VR

Quality of experience in affective pervasive environments

The confluence of miniaturised powerful devices, widespread communication networks and mass remote storage has caused a fundamental shift in the user interaction design paradigm. The distinction between system and user in pervasive environments is evolving into an increasingly integrated loop of interaction, raising a number of opportunities to provide enhanced and personalised experiences. We propose a platform, based on a smart architecture, to address the identified opportunities in pervasive computing. Smart systems aim at acting upon an environment for improving quality of experience: a subjective measure that has been defined as an emotional reaction to products or services. The inclusion of an emotional dimension allows us to measure individual user responses and deliver personalised services with the potential to influence experiences positively. The platform, Cloud2Bubble, leverages pervasive systems to aggregate user and environment data with the goal of addressing personal preferences and supra-functional requirements. This, combined with its societal implications, results in a set of design principles as a concrete fruition of design contractualism. In particular, this thesis describes: - a review of intelligent ubiquitous environments and relevant technologies, including a definition of user experience as a dynamic affective construct; - a specification of main components for personal data aggregation and service personalisation, without compromising privacy, security or usability; - the implementation of a software platform and a methodological procedure for its instantiation; - an evaluation of the developed platform and its benefits for urban mobility and public transport information systems; - a set of design principles for the design of ubiquitous systems, with an impact on individual experience and collective awareness. Cloud2Bubble contributes towards the development of affective intelligent ubiquitous systems with the potential to enhance user experience in pervasive environments. In addition, the platform aims at minimising the risk of user digital exposure while supporting collective action.Open Acces

Machine Learning for Auditory Hierarchy

Coleman, W. (2021). Machine Learning for Auditory Hierarchy. This dissertation is submitted for the degree of Doctor of Philosophy, Technological University Dublin. Audio content is predominantly delivered in a stereo audio file of a static, pre-formed mix. The content creator makes volume, position and effects decisions, generally for presentation in stereo speakers, but has no control ultimately over how the content will be consumed. This leads to poor listener experience when, for example, a feature film is mixed such that the dialogue is at a low level relative to the sound effects. Consumers can complain that they must turn the volume up to hear the words, but back down again because the effects levels are too loud. Addressing this problem requires a television mix optimised for the stereo speakers used in the vast majority of homes, which is not always available