115 research outputs found

    Is Multimedia Multisensorial? - A Review of Mulsemedia Systems

    Get PDF
    © 2018 Copyright held by the owner/author(s). Mulsemedia - multiple sensorial media - makes possible the inclusion of layered sensory stimulation and interaction through multiple sensory channels. e recent upsurge in technology and wearables provides mulsemedia researchers a vehicle for potentially boundless choice. However, in order to build systems that integrate various senses, there are still some issues that need to be addressed. is review deals with mulsemedia topics remained insu ciently explored by previous work, with a focus on multi-multi (multiple media - multiple senses) perspective, where multiple types of media engage multiple senses. Moreover, it addresses the evolution of previously identi ed challenges in this area and formulates new exploration directions.This article was funded by the European Union’s Horizon 2020 Research and Innovation program under Grant Agreement no. 688503

    MediaSync: Handbook on Multimedia Synchronization

    Get PDF
    This book provides an approachable overview of the most recent advances in the fascinating field of media synchronization (mediasync), gathering contributions from the most representative and influential experts. Understanding the challenges of this field in the current multi-sensory, multi-device, and multi-protocol world is not an easy task. The book revisits the foundations of mediasync, including theoretical frameworks and models, highlights ongoing research efforts, like hybrid broadband broadcast (HBB) delivery and users' perception modeling (i.e., Quality of Experience or QoE), and paves the way for the future (e.g., towards the deployment of multi-sensory and ultra-realistic experiences). Although many advances around mediasync have been devised and deployed, this area of research is getting renewed attention to overcome remaining challenges in the next-generation (heterogeneous and ubiquitous) media ecosystem. Given the significant advances in this research area, its current relevance and the multiple disciplines it involves, the availability of a reference book on mediasync becomes necessary. This book fills the gap in this context. In particular, it addresses key aspects and reviews the most relevant contributions within the mediasync research space, from different perspectives. Mediasync: Handbook on Multimedia Synchronization is the perfect companion for scholars and practitioners that want to acquire strong knowledge about this research area, and also approach the challenges behind ensuring the best mediated experiences, by providing the adequate synchronization between the media elements that constitute these experiences

    Perceived synchronization of mulsemedia services

    Get PDF
    Multimedia synchronization involves a temporal relationship between audio and visual media components. The presentation of "in-sync" data streams is essential to achieve a natural impression, as "out-of-sync" effects are often associated with user quality of experience (QoE) decrease. Recently, multi-sensory media (mulsemedia) has been demonstrated to provide a highly immersive experience for its users. Unlike traditional multimedia, mulsemedia consists of other media types (i.e., haptic, olfaction, taste, etc.) in addition to audio and visual content. Therefore, the goal of achieving high quality mulsemedia transmission is to present no or little synchronization errors between the multiple media components. In order to achieve this ideal synchronization, there is a need for comprehensive knowledge of the synchronization requirements at the user interface. This paper presents the results of a subjective study carried out to explore the temporal boundaries within which haptic and air-flow media objects can be successfully synchronized with video media. Results show that skews between sensorial media and multimedia might still give the effect that the mulsemedia sequence is "in-sync" and provide certain constraints under which synchronization errors might be tolerated. The outcomes of the paper are used to provide recommendations for mulsemedia service providers in order for their services to be associated with acceptable user experience levels, e.g. haptic media could be presented with a delay of up to 1 s behind video content, while air-flow media could be released either 5 s ahead of or 3 s behind video content

    Proposed Fuzzy Real-Time HaPticS Protocol Carrying Haptic Data and Multisensory Streams

    Get PDF
    Sensory and haptic data transfers to critical real-time applications over the Internet require better than best effort transport, strict timely and reliable ordered deliveries. Multi-sensory applications usually include video and audio streams with real-time control and sensory data, which aggravate and compress within real-time flows. Such real-time are vulnerable to synchronization to synchronization problems, if combined with poor Internet links. Apart from the use of differentiated QoS and MPLS services, several haptic transport protocols have been proposed to confront such issues, focusing on minimizing flows rate disruption while maintaining a steady transmission rate at the sender. Nevertheless, these protocols fail to cope with network variations and queuing delays posed by the Internet routers. This paper proposes a new haptic protocol that tries to alleviate such inadequacies using three different metrics: mean frame delay, jitter and frame loss calculated at the receiver end and propagated to the sender. In order to dynamically adjust flow rate in a fuzzy controlled manners, the proposed protocol includes a fuzzy controller to its protocol structure. The proposed FRTPS protocol (Fuzzy Real-Time haPticS protocol), utilizes crisp inputs into a fuzzification process followed by fuzzy control rules in order to calculate a crisp level output service class, denoted as Service Rate Level (SRL). The experimental results of FRTPS over RTP show that FRTPS outperforms RTP in cases of congestion incidents, out of order deliveries and goodput

    Multimodality with Eye tracking and Haptics: A New Horizon for Serious Games?

    Get PDF
    The goal of this review is to illustrate the emerging use of multimodal virtual reality that can benefit learning-based games. The review begins with an introduction to multimodal virtual reality in serious games and we provide a brief discussion of why cognitive processes involved in learning and training are enhanced under immersive virtual environments. We initially outline studies that have used eye tracking and haptic feedback independently in serious games, and then review some innovative applications that have already combined eye tracking and haptic devices in order to provide applicable multimodal frameworks for learning-based games. Finally, some general conclusions are identified and clarified in order to advance current understanding in multimodal serious game production as well as exploring possible areas for new applications

    Mulsemedia in Telecommunication and Networking Education: A Novel Teaching Approach that Improves the Learning Process

    Get PDF
    The advent and increased use of new technologies, such as innovative mulsemedia and multi-modal content distribution mechanisms, have brought new challenges and diverse opportunities for technology enhanced learning (TEL). NEWTON is a Horizon 2020 European project that revolutionizes the educational process through innovative TEL methodologies and tools, integrated in a pan-European STEM-related learning network platform. This article focuses on one of these novel TEL methodologies (i.e., mulsemedia) and presents how NEWTON enables mulsemedia- enhanced teaching and learning of STEM subjects, with a particular focus on telecommunication and networking related modules. The article also discusses the very promising results of NEWTON case studies carried out with engineering students across two different universities in Spain and Ireland, respectively. The case studies focused on analyzing the impact on the learning process of the mulsemedia-enhanced teaching in the context of telecommunication and networking modules. The main conclusion of the article is that mulsemedia-enhanced education significantly increases students' learning experience and improves their knowledge gain

    A survey of haptics in serious gaming

    Get PDF
    Serious gaming often requires high level of realism for training and learning purposes. Haptic technology has been proved to be useful in many applications with an additional perception modality complementary to the audio and the vision. It provides novel user experience to enhance the immersion of virtual reality with a physical control-layer. This survey focuses on the haptic technology and its applications in serious gaming. Several categories of related applications are listed and discussed in details, primarily on haptics acts as cognitive aux and main component in serious games design. We categorize haptic devices into tactile, force feedback and hybrid ones to suit different haptic interfaces, followed by description of common haptic gadgets in gaming. Haptic modeling methods, in particular, available SDKs or libraries either for commercial or academic usage, are summarized. We also analyze the existing research difficulties and technology bottleneck with haptics and foresee the future research directions

    Multimodality in VR: A survey

    Get PDF
    Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive, to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR, and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research, and demonstrate that multimodality plays a fundamental role in VR; enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer

    Mulsemedia Communication Research Challenges for Metaverse in 6G Wireless Systems

    Full text link
    Although humans have five basic senses, sight, hearing, touch, smell, and taste, most multimedia systems in current systems only capture two of them, namely, sight and hearing. With the development of the metaverse and related technologies, there is a growing need for a more immersive media format that leverages all human senses. Multisensory media(Mulsemedia) that can stimulate multiple senses will play a critical role in the near future. This paper provides an overview of the history, background, use cases, existing research, devices, and standards of mulsemedia. Emerging mulsemedia technologies such as Extended Reality (XR) and Holographic-Type Communication (HTC) are introduced. Additionally, the challenges in mulsemedia research from the perspective of wireless communication and networking are discussed. The potential of 6G wireless systems to address these challenges is highlighted, and several research directions that can advance mulsemedia communications are identified
    corecore