Merging the Real and the Virtual: An Exploration of Interaction Methods to Blend Realities

We investigate, build, and design interaction methods to merge the real with the virtual. An initial investigation looks at spatial augmented reality (SAR) and its effects on pointing with a real mobile phone. A study reveals a set of trade-offs between the raycast, viewport, and direct pointing techniques. To further investigate the manipulation of virtual content within a SAR environment, we design an interaction technique that utilizes the distance that a user holds mobile phone away from their body. Our technique enables pushing virtual content from a mobile phone to an external SAR environment, interact with that content, rotate-scale-translate it, and pull the content back into the mobile phone. This is all done in a way that ensures seamless transitions between the real environment of the mobile phone and the virtual SAR environment. To investigate the issues that occur when the physical environment is hidden by a fully immersive virtual reality (VR) HMD, we design and investigate a system that merges a realtime 3D reconstruction of the real world with a virtual environment. This allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical reality without losing their sense of immersion or presence inside a virtual world. A study with VR users demonstrates the affordances provided by the system and how it can be used to enhance current VR experiences. We then move to AR, to investigate the limitations of optical see-through HMDs and the problem of communicating the internal state of the virtual world with unaugmented users. To address these issues and enable new ways to visualize, manipulate, and share virtual content, we propose a system that combines a wearable SAR projector. Demonstrations showcase ways to utilize the projected and head-mounted displays together, such as expanding field of view, distributing content across depth surfaces, and enabling bystander collaboration. We then turn to videogames to investigate how spectatorship of these virtual environments can be enhanced through expanded video rendering techniques. We extract and combine additional data to form a cumulative 3D representation of the live game environment for spectators, which enables each spectator to individually control a personal view into the stream while in VR. A study shows that users prefer spectating in VR when compared with a comparable desktop rendering

Towards highly informative learning analytics

Among various trending topics that can be investigated in the field of educational technology, there is a clear and high demand for using artificial intelligence (AI) and educational data to improve the whole learning and teaching cycle. This spans from collecting and estimating the prior knowledge of learners for a certain subject to the actual learning process and its assessment. AI in education cuts across almost all educational technology disciplines and is key to many other technological innovations for educational institutions. The use of data to inform decision-making in education and training is not new, but the scope and scale of its potential impact on teaching and learning have silently increased by orders of magnitude over the last few years. The release of ChatGPT was another driver to finally make everyone aware of the potential effects of AI technology in the digital education system of today. We are now at a stage where data can be automatically harvested at previously unimagined levels of granularity and variety. Analysis of these data with AI has the potential to provide evidence-based insights into learners’ abilities and patterns of behaviour that, in turn, can provide crucial action points to guide curriculum and course design, personalised assistance, generate assessments, and the development of new educational offerings. AI in education has many connected research communities like Artificial Intelligence in Education (AIED), Educational Data Mining (EDM), or Learning Analytics (LA). LA is the term that is used for research, studies, and applications that try to understand and support the behaviour of learners based on large sets of collected data

Multimodal Content Delivery for Geo-services

This thesis describes a body of work carried out over several research projects in the area of multimodal interaction for location-based services. Research in this area has progressed from using simulated mobile environments to demonstrate the visual modality, to the ubiquitous delivery of rich media using multimodal interfaces (geo- services). To effectively deliver these services, research focused on innovative solutions to real-world problems in a number of disciplines including geo-location, mobile spatial interaction, location-based services, rich media interfaces and auditory user interfaces. My original contributions to knowledge are made in the areas of multimodal interaction underpinned by advances in geo-location technology and supported by the proliferation of mobile device technology into modern life. Accurate positioning is a known problem for location-based services, contributions in the area of mobile positioning demonstrate a hybrid positioning technology for mobile devices that uses terrestrial beacons to trilaterate position. Information overload is an active concern for location-based applications that struggle to manage large amounts of data, contributions in the area of egocentric visibility that filter data based on field-of-view demonstrate novel forms of multimodal input. One of the more pertinent characteristics of these applications is the delivery or output modality employed (auditory, visual or tactile). Further contributions in the area of multimodal content delivery are made, where multiple modalities are used to deliver information using graphical user interfaces, tactile interfaces and more notably auditory user interfaces. It is demonstrated how a combination of these interfaces can be used to synergistically deliver context sensitive rich media to users - in a responsive way - based on usage scenarios that consider the affordance of the device, the geographical position and bearing of the device and also the location of the device

Ambiguous worlds: understanding the design of first-person walker games

The ‘walker’ is a burgeoning form of videogame with a growing body of literature primarily discussing how the genre challenges the accepted norms of games. These discussions widely use the derogatory term ‘walking simulator’, which implies its non-game status. What is also clear from these discussions is how these games draw on, but also push back against game design conventions. Walking is the primary means of interaction in walker games, rather than prioritising ‘skill-based’ mechanics. For example, the mechanics of gameplay in walker games are typically minimal, slow and non-violent. The unique design focus of walker games exists within a contested and complicated area of game design literature, yet many players find the exploration and experiences of these game environments to be compelling. This research asks: What gameplay experiences do walkers elicit, and how might designers understand these experiences? What are the game design attributes that engage players to explore 3D walker environments? How can these design attributes be used to design first-person walker games and 3D games more broadly? Drawing upon game design and design research literature, I explore these questions with specific focus on player interaction and level design in walker game world exploration. My research approach consists of three major investigative stages. I conduct a formal analysis of four existing walker games: Dear Esther, Proteus, Gone Home and The Stanley Parable. My analysis reveals four key themes for investigating walker design. These four themes are then applied to the development of my design project, WORLD4, a multi-view exploration game where players explore a 3D layered abstract world. I develop a methodological approach based upon indie gameplay testing to conduct an analysis of WORLD4’s design through a two-stage qualitative player study. Based on these three stages of analysis, I conclude that a particular kind of experience of curiosity emerges, driving player exploration in walker games. My findings indicate that the experience of curiosity is fostered by incorporating ambiguity into the game design, which modifies game world exploration into a more investigative and interpretive activity. I support this conclusion through three design themes to understand the player experience of WORLD4 and six design strategies for fostering ambiguity in the design of exploratory game environments. More generally, I contribute a perspective on game design that emphasises ambiguity in order to create heightened and compelling exploration experiences

Breaking Virtual Barriers : Investigating Virtual Reality for Enhanced Educational Engagement

Virtual reality (VR) is an innovative technology that has regained popularity in recent years. In the field of education, VR has been introduced as a tool to enhance learning experiences. This thesis presents an exploration of how VR is used from the context of educators and learners. The research employed a mixed-methods approach, including surveying and interviewing educators, and conducting empirical studies to examine engagement, usability, and user behaviour within VR. The results revealed educators are interested in using VR for a wide range of scenarios, including thought exercises, virtual field trips, and simulations. However, they face several barriers to incorporating VR into their practice, such as cost, lack of training, and technical challenges. A subsequent study found that virtual reality can no longer be assumed to be more engaging than desktop equivalents. This empirical study showed that engagement levels were similar in both VR and non-VR environments, suggesting that the novelty effect of VR may be less pronounced than previously assumed. A study against a VR mind mapping artifact, VERITAS, demonstrated that complex interactions are possible on low-cost VR devices, making VR accessible to educators and students. The analysis of user behaviour within this VR artifact showed that quantifiable strategies emerge, contributing to the understanding of how to design for collaborative VR experiences. This thesis provides insights into how the end-users in the education space perceive and use VR. The findings suggest that while educators are interested in using VR, they face barriers to adoption. The research highlights the need to design VR experiences, with understanding of existing pedagogy, that are engaging with careful thought applied to complex interactions, particularly for collaborative experiences. This research contributes to the understanding of the potential of VR in education and provides recommendations for educators and designers to enhance learning experiences using VR

A design space for social object labels in museums

Taking a problematic user experience with ubiquitous annotation as its point of departure, this thesis defines and explores the design space for Social Object Labels (SOLs), small interactive displays aiming to support users' in-situ engagement with digital annotations of physical objects and places by providing up-to-date information before, during and after interaction. While the concept of ubiquitous annotation has potential applications in a wide range of domains, the research focuses in particular on SOLs in a museum context, where they can support the institution's educational goals by engaging visitors in the interpretation of exhibits and providing a platform for public discourse to complement official interpretations provided on traditional object labels. The thesis defines and structures the design space for SOLs, investigates how they can support social interpretation in museums and develops empirically validated design recommendations. Reflecting the developmental character of the research, it employs Design Research as a methodological framework, which involves the iterative development and evaluation of design artefacts together with users and other stakeholders. The research identifies the particular characteristics of SOLs and structures their design space into ten high-level aspects, synthesised from taxonomies and heuristics for similar display concepts and complemented with aspects emerging from the iterative design and evaluation of prototypes. It presents findings from a survey exploring visitors' mental models, preferences and expectations of commenting in museums and translates them into requirements for SOLs. It reports on scenario-based design activities, expert interviews with museum professionals, formative user studies and co-design sessions, and two empirical evaluations of SOL prototypes in a gallery environment. Pulling together findings from these research activities it then formulates design recommendations for SOLs and supports them with related evidence and implementation examples. The main contributions are (i) to delineate and structure the design space for SOLs, which helps to ground SOLs in the literature and understand them as a distinct display concept with its own characteristics; (ii) to explore, for the first time, a visitor perspective on commenting in museums, which can inform research, development and policies on user-generated content in museums and the wider cultural heritage sector; (iii) to develop empirically validated design recommendations, which can inform future research and development into SOLs and related display concept. The thesis concludes by summarising findings in relation to its stated research questions, restating its contributions from ubiquitous computing, domain and methodology perspectives, and discussing open issues and future work

Graphics Technology in Space Applications (GTSA 1989)

This document represents the proceedings of the Graphics Technology in Space Applications, which was held at NASA Lyndon B. Johnson Space Center on April 12 to 14, 1989 in Houston, Texas. The papers included in these proceedings were published in general as received from the authors with minimum modifications and editing. Information contained in the individual papers is not to be construed as being officially endorsed by NASA

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Design Transactions

Design Transactions presents the outcome of new research to emerge from ‘Innochain’, a consortium of six leading European architectural and engineering-focused institutions and their industry partners. The book presents new advances in digital design tooling that challenge established building cultures and systems. It offers new sustainable and materially smart design solutions with a strong focus on changing the way the industry thinks, designs, and builds our physical environment. Divided into sections exploring communication, simulation and materialisation, Design Transactions explores digital and physical prototyping and testing that challenges the traditional linear construction methods of incremental refinement. This novel research investigates ‘the digital chain’ between phases as an opportunity for extended interdisciplinary design collaboration. The highly illustrated book features work from 15 early-stage researchers alongside chapters from world-leading industry collaborators and academics