Designing passenger experiences for in-car Mixed Reality

In day-to-day life, people spend a considerable amount of their time on the road. People seek to invest travel time for work and well-being through interaction with mobile and multimedia applications on personal devices such as smartphones and tablets. However, for new computing paradigms, such as mobile mixed reality (MR), their usefulness in this everyday transport context, in-car MR remains challenging. When future passengers immerse in three-dimensional virtual environments, they become increasingly disconnected from the cabin space, vehicle motion, and other people around them. This degraded awareness of the real environment endangers the passenger experience on the road, which initially motivates this thesis to question: can immersive technology become useful in the everyday transport context, such as for in-car scenarios? If so, how should we design in-car MR technology to foster passenger access and connectedness to both physical and virtual worlds, ensuring ride safety, comfort, and joy? To this aim, this thesis contributes via three aspects: 1) Understanding passenger use of in-car MR —first, I present a model for in-car MR interaction through user research. As interviews with daily commuters reveal, passengers are concerned with their physical integrity when facing spatial conflicts between borderless virtual environments and the confined cabin space. From this, the model aims to help researchers spatially organize information and how user interfaces vary in the proximity of the user. Additionally, a field experiment reveals contextual feedback about motion sickness when using immersive technology on the road. This helps refine the model and instruct the following experiments. 2) Mixing realities in car rides —second, this thesis explores a series of prototypes and experiments to examine how in-car MR technology can enable passengers to feel present in virtual environments while maintaining awareness of the real environment. The results demonstrate technical solutions for physical integrity and situational awareness by incorporating essential elements of the RE into virtual reality. Empirical evidence provides a set of dimensions into the in-car MR model, guiding the design decisions of mixing realities. 3) Transcending the transport context —third, I extend the model to other everyday contexts beyond transport that share spatial and social constraints, such as the confined and shared living space at home. A literature review consolidates leveraging daily physical objects as haptic feedback for MR interaction across spatial scales. A laboratory experiment discovers how context-aware MR systems that consider physical configurations can support social interaction with copresent others in close shared spaces. These results substantiate the scalability of the in-car MR model to other contexts. Finally, I conclude with a holistic model for mobile MR interaction across everyday contexts, from home to on the road. With my user research, prototypes, empirical evaluation, and model, this thesis paves the way for understanding the future passenger use of immersive technology, addressing today’s technical limitations of MR in mobile interaction, and ultimately fostering mobile users’ ubiquitous access and close connectedness to MR anytime and anywhere in their daily lives.Im modernen Leben verbringen die Menschen einen beträchtlichen Teil ihrer Zeit mit dem täglichen Pendeln. Die Menschen versuchen, die Reisezeit für ihre Arbeit und ihr Wohlbefinden durch die Interaktion mit mobilen und multimedialen Anwendungen auf persönlichen Geräten wie Smartphones und Tablets zu nutzen. Doch für neue Computing-Paradigmen, wie der mobilen Mixed Reality (MR), bleibt ihre Nützlichkeit in diesem alltäglichen Verkehrskontext, der MR im Auto, eine Herausforderung. Wenn künftige Passagiere in dreidimensionale virtuelle Umgebungen eintauchen, werden sie zunehmend von der Kabine, der Fahrzeugbewegung und den Menschen in ihrer Umgebung abgekoppelt. Diese verminderte Wahrnehmung der realen Umgebung gefährdet das Fahrverhalten der Passagiere im Straßenverkehr, was diese Arbeit zunächst zu der Frage motiviert: Können immersive Systeme im alltäglichen Verkehrskontext, z.B. in Fahrzeugszenarien, nützlich werden? Wenn ja, wie sollten wir die MR-Technologie im Auto gestalten, um den Zugang und die Verbindung der Passagiere mit der physischen und der virtuellen Welt zu fördern und dabei Sicherheit, Komfort und Freude an der Fahrt zu gewährleisten? Zu diesem Zweck trägt diese Arbeit zu drei Aspekten bei: 1) Verständnis der Nutzung von MR im Auto durch die Passagiere - Zunächst wird ein Modell für die MR-Interaktion im Auto durch user research vorgestellt. Wie aus Interviews mit täglichen Pendlern hervorgeht, sind die Passagiere um ihre körperliche Unversehrtheit besorgt, wenn sie mit räumlichen Konflikten zwischen grenzenlosen virtuellen Umgebungen und dem begrenzten Kabinenraum konfrontiert werden. Das Modell soll Forschern dabei helfen, Informationen und Benutzerschnittstellen räumlich zu organisieren, die in der Nähe des Benutzers variieren. Darüber hinaus zeigt ein Feldexperiment kontextbezogenes Feedback zur Reisekrankheit bei der Nutzung immersiver Technologien auf der Straße. Dies hilft, das Modell zu verfeinern und die folgenden Experimente zu instruieren. 2) Vermischung von Realitäten bei Autofahrten - Zweitens wird in dieser Arbeit anhand einer Reihe von Prototypen und Experimenten untersucht, wie die MR-Technologie im Auto es den Passagieren ermöglichen kann, sich in virtuellen Umgebungen präsent zu fühlen und gleichzeitig das Bewusstsein für die reale Umgebung zu behalten. Die Ergebnisse zeigen technische Lösungen für räumliche Beschränkungen und Situationsbewusstsein, indem wesentliche Elemente der realen Umgebung in VR integriert werden. Die empirischen Erkenntnisse bringen eine Reihe von Dimensionen in das Modell der MR im Auto ein, die die Designentscheidungen für gemischte Realitäten leiten. 3) Über den Verkehrskontext hinaus - Drittens erweitere ich das Modell auf andere Alltagskontexte jenseits des Verkehrs, in denen räumliche und soziale Zwänge herrschen, wie z.B. in einem begrenzten und gemeinsam genutzten Wohnbereich zu Hause. Eine Literaturrecherche konsolidiert die Nutzung von Alltagsgegenständen als haptisches Feedback für MR-Interaktion über räumliche Skalen hinweg. Ein Laborexperiment zeigt, wie kontextbewusste MR-Systeme, die physische Konfigurationen berücksichtigen, soziale Interaktion mit anderen Personen in engen gemeinsamen Räumen ermöglichen. Diese Ergebnisse belegen die Übertragbarkeit des MR-Modells im Auto auf andere Kontexte. Schließlich schließe ich mit einem ganzheitlichen Modell für mobile MR-Interaktion in alltäglichen Kontexten, von zu Hause bis unterwegs. Mit meiner user research, meinen Prototypen und Evaluierungsexperimenten sowie meinem Modell ebnet diese Dissertation den Weg für das Verständnis der zukünftigen Nutzung immersiver Technologien durch Passagiere, für die Überwindung der heutigen technischen Beschränkungen von MR in der mobilen Interaktion und schließlich für die Förderung des allgegenwärtigen Zugangs und der engen Verbindung der mobilen Nutzer zu MR jederzeit und überall in ihrem täglichen Leben

Internet of Vehicles: Motivation, Layered Architecture, Network Model, Challenges, and Future Aspects

© 2013 IEEE. Internet of Things is smartly changing various existing research areas into new themes, including smart health, smart home, smart industry, and smart transport. Relying on the basis of 'smart transport,' Internet of Vehicles (IoV) is evolving as a new theme of research and development from vehicular ad hoc networks (VANETs). This paper presents a comprehensive framework of IoV with emphasis on layered architecture, protocol stack, network model, challenges, and future aspects. Specifically, following the background on the evolution of VANETs and motivation on IoV an overview of IoV is presented as the heterogeneous vehicular networks. The IoV includes five types of vehicular communications, namely, vehicle-to-vehicle, vehicle-to-roadside, vehicle-to-infrastructure of cellular networks, vehicle-to-personal devices, and vehicle-to-sensors. A five layered architecture of IoV is proposed considering functionalities and representations of each layer. A protocol stack for the layered architecture is structured considering management, operational, and security planes. A network model of IoV is proposed based on the three network elements, including cloud, connection, and client. The benefits of the design and development of IoV are highlighted by performing a qualitative comparison between IoV and VANETs. Finally, the challenges ahead for realizing IoV are discussed and future aspects of IoV are envisioned

Employing Emerging Technologies to Develop and Evaluate In-Vehicle Intelligent Systems for Driver Support: Infotainment AR HUD Case Study

The plurality of current infotainment devices within the in-vehicle space produces an unprecedented volume of incoming data that overwhelm the typical driver, leading to higher collision probability. This work presents an investigation to an alternative option which aims to manage the incoming information while offering an uncluttered and timely manner of presenting and interacting with the incoming data safely. The latter is achieved through the use of an augmented reality (AR) head-up display (HUD) system, which projects the information within the driver’s field of view. An uncluttered gesture recognition interface provides the interaction with the AR visuals. For the assessment of the system’s effectiveness, we developed a full-scale virtual reality driving simulator which immerses the drivers in challenging, collision-prone, scenarios. The scenarios unfold within a digital twin model of the surrounding motorways of the city of Glasgow. The proposed system was evaluated in contrast to a typical head-down display (HDD) interface system by 30 users, showing promising results that are discussed in detail

Employing Emerging Technologies to Develop and Evaluate In-Vehicle Intelligent Systems for Driver Support: Infotainment AR HUD Case Study

The plurality of current infotainment devices within the in-vehicle space produces an unprecedented volume of incoming data that overwhelm the typical driver, leading to higher collision probability. This work presents an investigation to an alternative option which aims to manage the incoming information while offering an uncluttered and timely manner of presenting and interacting with the incoming data safely. The latter is achieved through the use of an augmented reality (AR) head-up display (HUD) system, which projects the information within the driver’s field of view. An uncluttered gesture recognition interface provides the interaction with the AR visuals. For the assessment of the system’s effectiveness, we developed a full-scale virtual reality driving simulator which immerses the drivers in challenging, collision-prone, scenarios. The scenarios unfold within a digital twin model of the surrounding motorways of the city of Glasgow. The proposed system was evaluated in contrast to a typical head-down display (HDD) interface system by 30 users, showing promising results that are discussed in detail

Towards video streaming in IoT environments: vehicular communication perspective

Multimedia oriented Internet of Things (IoT) enables pervasive and real-time communication of video, audio and image data among devices in an immediate surroundings. Today's vehicles have the capability of supporting real time multimedia acquisition. Vehicles with high illuminating infrared cameras and customized sensors can communicate with other on-road devices using dedicated short-range communication (DSRC) and 5G enabled communication technologies. Real time incidence of both urban and highway vehicular traffic environment can be captured and transmitted using vehicle-to-vehicle and vehicle-to-infrastructure communication modes. Video streaming in vehicular IoT (VSV-IoT) environments is in growing stage with several challenges that need to be addressed ranging from limited resources in IoT devices, intermittent connection in vehicular networks, heterogeneous devices, dynamism and scalability in video encoding, bandwidth underutilization in video delivery, and attaining application-precise quality of service in video streaming. In this context, this paper presents a comprehensive review on video streaming in IoT environments focusing on vehicular communication perspective. Specifically, significance of video streaming in vehicular IoT environments is highlighted focusing on integration of vehicular communication with 5G enabled IoT technologies, and smart city oriented application areas for VSV-IoT. A taxonomy is presented for the classification of related literature on video streaming in vehicular network environments. Following the taxonomy, critical review of literature is performed focusing on major functional model, strengths and weaknesses. Metrics for video streaming in vehicular IoT environments are derived and comparatively analyzed in terms of their usage and evaluation capabilities. Open research challenges in VSV-IoT are identified as future directions of research in the area. The survey would benefit both IoT and vehicle industry practitioners and researchers, in terms of augmenting understanding of vehicular video streaming and its IoT related trends and issues

Smartphone-based vehicle telematics: a ten-year anniversary

This is the author accepted manuscript. The final version is available from the publisher via the DOI in this recordJust as it has irrevocably reshaped social life, the fast growth of smartphone ownership is now beginning to revolutionize the driving experience and change how we think about automotive insurance, vehicle safety systems, and traffic research. This paper summarizes the first ten years of research in smartphone-based vehicle telematics, with a focus on user-friendly implementations and the challenges that arise due to the mobility of the smartphone. Notable academic and industrial projects are reviewed, and system aspects related to sensors, energy consumption, and human-machine interfaces are examined. Moreover, we highlight the differences between traditional and smartphone-based automotive navigation, and survey the state of the art in smartphone-based transportation mode classification, vehicular ad hoc networks, cloud computing, driver classification, and road condition monitoring. Future advances are expected to be driven by improvements in sensor technology, evidence of the societal benefits of current implementations, and the establishment of industry standards for sensor fusion and driver assessment

Towards Tactile Internet in Beyond 5G Era: Recent Advances, Current Issues and Future Directions

Tactile Internet (TI) is envisioned to create a paradigm shift from the content-oriented communications to steer/control-based communications by enabling real-time transmission of haptic information (i.e., touch, actuation, motion, vibration, surface texture) over Internet in addition to the conventional audiovisual and data traffics. This emerging TI technology, also considered as the next evolution phase of Internet of Things (IoT), is expected to create numerous opportunities for technology markets in a wide variety of applications ranging from teleoperation systems and Augmented/Virtual Reality (AR/VR) to automotive safety and eHealthcare towards addressing the complex problems of human society. However, the realization of TI over wireless media in the upcoming Fifth Generation (5G) and beyond networks creates various non-conventional communication challenges and stringent requirements in terms of ultra-low latency, ultra-high reliability, high data-rate connectivity, resource allocation, multiple access and quality-latency-rate tradeoff. To this end, this paper aims to provide a holistic view on wireless TI along with a thorough review of the existing state-of-the-art, to identify and analyze the involved technical issues, to highlight potential solutions and to propose future research directions. First, starting with the vision of TI and recent advances and a review of related survey/overview articles, we present a generalized framework for wireless TI in the Beyond 5G Era including a TI architecture, the main technical requirements, the key application areas and potential enabling technologies. Subsequently, we provide a comprehensive review of the existing TI works by broadly categorizing them into three main paradigms; namely, haptic communications, wireless AR/VR, and autonomous, intelligent and cooperative mobility systems. Next, potential enabling technologies across physical/Medium Access Control (MAC) and network layers are identified and discussed in detail. Also, security and privacy issues of TI applications are discussed along with some promising enablers. Finally, we present some open research challenges and recommend promising future research directions

Stepping into safety: a systematic review of extended reality technology applications in enhancing vulnerable road user safety

This is an accepted manuscript of an article published by Emerald in Smart and Sustainable Built Environment on 04/07/2024, available online: https://doi.org/10.1108/SASBE-10-2023-0321 The accepted version of the publication may differ from the final published version.Purpose In alignment with the European Union’s Vision Zero initiative to eliminate road fatalities by 2050, leveraging technological advancements becomes crucial for addressing the challenges of vulnerable road users (VRUs), and for mitigating the impact of human error. Despite increasing scholarly interest in applications of extended reality (XR), a research gap persists, particularly in the role of XR in transportation safety. Therefore, the aim of the study was to fill this gap through a systematic literature review to evaluate comprehensively the potential scope and practical applicability of XR technologies in enhancing the safety of VRUs. Design/methodology/approach A systematic review was undertaken, following PRISMA guidelines meticulously, in which 80 relevant articles from databases, such as Scopus and Science Direct, were identified and analysed. Findings The results of the analysis revealed the potential of XR beyond pedestrians and cyclists, and highlighted a lack of research about the impact of XR with regard to the personal traits or abilities of VRUs. The results of a thorough analysis confirmed the potential of XR as a promising solution for an approach to collaborative co-creation in addressing the safety challenges of VRUs. In addition, the integration of eye-tracking with virtual reality emerged as a promising innovation for enhancing the safety of vulnerable road users. Research limitations/implications Theoretical implications include enhancing the understanding of applications of XR in VRUs’ safety and providing insights into future research possibilities and methodological approaches. Valuable insights into search strategies and inclusion-exclusion criteria can guide future research methodologies. Practical implications Practically, the findings from the study offer insights to assist urban planners and transportation authorities in incorporating XR technologies effectively for VRUs safety. Identifying areas for further development of XR technology could inspire innovation and investment in solutions designed to meet the safety needs of VRUs, such as enhanced visualisation tools and immersive training simulations. Originality/value The findings of previous research underscore the vast potential of XR technologies within the built environment, yet their utilisation remains limited in the urban transport sector. The intricacies of urban traffic scenarios pose significant challenges for VRUs, making participation in mobility studies hazardous. Hence, it is crucial to explore the scope of emerging technologies in addressing VRUs issues as a pre-requisite for establishing comprehensive safety measures.The SOTERIA project has received funding from the European Commissions’ Horizon Europe Research and Innovation Programme under Grant Agreement No 101077433, as well as by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [Grant Nos: 10052969, 10058247, 10059948, and 10064506] to whom all partner organisations and authors express their sincere gratitude.Published onlin

A Testing and Experimenting Environment for Microscopic Traffic Simulation Utilizing Virtual Reality and Augmented Reality

Microscopic traffic simulation (MTS) is the emulation of real-world traffic movements in a virtual environment with various traffic entities. Typically, the movements of the vehicles in MTS follow some predefined algorithms, e.g., car-following models, lane changing models, etc. Moreover, existing MTS models only provide a limited capability of two- and/or three-dimensional displays that often restrict the user’s viewpoint to a flat screen. Their downscaled scenes neither provide a realistic representation of the environment nor allow different users to simultaneously experience or interact with the simulation model from different perspectives. These limitations neither allow the traffic engineers to effectively disseminate their ideas to various stakeholders of different backgrounds nor allow the analysts to have realistic data about the vehicle or pedestrian movements. This dissertation intends to alleviate those issues by creating a framework and a prototype for a testing environment where MTS can have inputs from user-controlled vehicles and pedestrians to improve their traffic entity movement algorithms as well as have an immersive M3 (multi-mode, multi-perspective, multi-user) visualization of the simulation using Virtual Reality (VR) and Augmented Reality (AR) technologies. VR environments are created using highly realistic 3D models and environments. With modern game engines and hardware available on the market, these VR applications can provide a highly realistic and immersive experience for a user. Different experiments performed by real users in this study prove that utilizing VR technology for different traffic related experiments generated much more favorable results than the traditional displays. Moreover, using AR technologies for pedestrian studies is a novel approach that allows a user to walk in the real world and the simulation world at a one-to-one scale. This capability opens a whole new avenue of user experiment possibilities. On top of that, the in-environment communication chat system will allow researchers to perform different Advanced Driver Assistance System (ADAS) studies without ever needing to leave the simulation environment. Last but not least, the distributed nature of the framework enables users to participate from different geographic locations with their choice of display device (desktop, smartphone, VR, or AR). The prototype developed for this dissertation is readily available on a test webpage, and a user can easily download the prototype application without needing to install anything. The user also can run the remote MTS server and then connect their client application to the server

Toward a Bio-Inspired System Architecting Framework: Simulation of the Integration of Autonomous Bus Fleets & Alternative Fuel Infrastructures in Closed Sociotechnical Environments

Cities are set to become highly interconnected and coordinated environments composed of emerging technologies meant to alleviate or resolve some of the daunting issues of the 21st century such as rapid urbanization, resource scarcity, and excessive population demand in urban centers. These cybernetically-enabled built environments are expected to solve these complex problems through the use of technologies that incorporate sensors and other data collection means to fuse and understand large sums of data/information generated from other technologies and its human population. Many of these technologies will be pivotal assets in supporting and managing capabilities in various city sectors ranging from energy to healthcare. However, among these sectors, a significant amount of attention within the recent decade has been in the transportation sector due to the flood of new technological growth and cultivation, which is currently seeing extensive research, development, and even implementation of emerging technologies such as autonomous vehicles (AVs), the Internet of Things (IoT), alternative xxxvi fueling sources, clean propulsion technologies, cloud/edge computing, and many other technologies. Within the current body of knowledge, it is fairly well known how many of these emerging technologies will perform in isolation as stand-alone entities, but little is known about their performance when integrated into a transportation system with other emerging technologies and humans within the system organization. This merging of new age technologies and humans can make analyzing next generation transportation systems extremely complex to understand. Additionally, with new and alternative forms of technologies expected to come in the near-future, one can say that the quantity of technologies, especially in the smart city context, will consist of a continuously expanding array of technologies whose capabilities will increase with technological advancements, which can change the performance of a given system architecture. Therefore, the objective of this research is to understand the system architecture implications of integrating different alternative fueling infrastructures with autonomous bus (AB) fleets in the transportation system within a closed sociotechnical environment. By being able to understand the system architecture implications of alternative fueling infrastructures and AB fleets, this could provide performance-based input into a more sophisticated approach or framework which is proposed as a future work of this research