An Overview of Capturing Live Experience with Virtual and Augmented Reality

In this paper, we review the use of virtual and augmented reality technologies for capture and externalization of tacit knowledge from complex activity in knowledge-intensive professions. We focus on technologies for converting experience hidden in activity with the aim to boost industry competitiveness, innovation, and facilitate learning on the job. As such types of knowledge and experience are difficult to capture and represent in traditional media, we explore emerging technology along two lines of investigation. First, we look at applications of virtual reality to then, second, focus on using sensors, augmented reality, and wearable technologies. We discuss existing and future applications of experience capturing with virtual and augmented reality technologies. This review provides a comprehensive overview for those interested in recording virtual, real, and augmented reality technologies. This review provides a comprehensive overview for those interested in recording virtual, real, and augmented activities, methods for delivering the recorded data, and extracting knowledge

Transitioning from Transmedia to Transreality Storyboarding to Improve the Co-Creation of the Experience Space

Transmedia storytelling is a digital based marketing approach in present day consumer markets. Typically applied to spanning or segueing stories or experiences across media such as film, books, comics and video-games to reach broader target audiences, often triggering a narrative, into which customers can participate and co-create the narrative. Common aims at customer engagement have been through shared stories on present day social media. However, for the creative-consumer, sharing on social media falls short of fully immersive storytelling ecology. Creatives (traditional designers and consumers) would benefit through tools and processes for incrementally expanding dimensions, mediums, fidelity, and shared interactions and senses across multiple media and interactive realities. This paper presents use cases of Transreality Storyboarding Framework (TSF), a design framework that affords creation of experience spaces for consumer-product engagement. Further, we propose a TSF app, to allow non-expert designers/everyday-consumers to contribute to storytelling, participation and production of product experience spaces

Games, Simulations, Immersive Environments, and Emerging Technologies

International audienceThis entry presents an overview of advanced technologies to support teaching and learning. The use of innovative interactive systems for education has never been higher. Far from being just a trend, the objective is to use the current technology to cover educational needs and create relevant pedagogical situations. The arguments in their favor are generally their positive effects on learners’ motivation and the necessity to provide learning methods adapted to our growing digital culture. The new learning technologies and emerging trends are first reviewed hereunder. We thus define and discuss learning games, gamification, simulation, immersive environments and other emerging technologies. Then, the current limits and remaining scientific challenges are highlighted

Phrasing Bimanual Interaction for Visual Design

Architects and other visual thinkers create external representations of their ideas to support early-stage design. They compose visual imagery with sketching to form abstract diagrams as representations. When working with digital media, they apply various visual operations to transform representations, often engaging in complex sequences. This research investigates how to build interactive capabilities to support designers in putting together, that is phrasing, sequences of operations using both hands. In particular, we examine how phrasing interactions with pen and multi-touch input can support modal switching among different visual operations that in many commercial design tools require using menus and tool palettes—techniques originally designed for the mouse, not pen and touch. We develop an interactive bimanual pen+touch diagramming environment and study its use in landscape architecture design studio education. We observe interesting forms of interaction that emerge, and how our bimanual interaction techniques support visual design processes. Based on the needs of architects, we develop LayerFish, a new bimanual technique for layering overlapping content. We conduct a controlled experiment to evaluate its efficacy. We explore the use of wearables to identify which user, and distinguish what hand, is touching to support phrasing together direct-touch interactions on large displays. From design and development of the environment and both field and controlled studies, we derive a set methods, based upon human bimanual specialization theory, for phrasing modal operations through bimanual interactions without menus or tool palettes

Designing an augmented reality exhibition: Leonardo's Impossible Machines

This paper discusses the origins, development and results of the animated and augmented reality aspects of the exhibition ‘Leonardo’s Impossible Machines’ that was developed at Ravensbourne University London and Birkbeck, University of London, with support from the Museo Galileo. The exhibition included novel reconstructions and visualisations of Leonardo’s perpetual motion machines from the Codex Forster, and the process is explained here, along with the challenges of mounting a combined physical and AR show

Biosensing and Actuation—Platforms Coupling Body Input-Output Modalities for Affective Technologies

Research in the use of ubiquitous technologies, tracking systems and wearables within mental health domains is on the rise. In recent years, affective technologies have gained traction and garnered the interest of interdisciplinary fields as the research on such technologies matured. However, while the role of movement and bodily experience to affective experience is well-established, how to best address movement and engagement beyond measuring cues and signals in technology-driven interactions has been unclear. In a joint industry-academia effort, we aim to remodel how affective technologies can help address body and emotional self-awareness. We present an overview of biosignals that have become standard in low-cost physiological monitoring and show how these can be matched with methods and engagements used by interaction designers skilled in designing for bodily engagement and aesthetic experiences. Taking both strands of work together offers unprecedented design opportunities that inspire further research. Through first-person soma design, an approach that draws upon the designer’s felt experience and puts the sentient body at the forefront, we outline a comprehensive work for the creation of novel interactions in the form of couplings that combine biosensing and body feedback modalities of relevance to affective health. These couplings lie within the creation of design toolkits that have the potential to render rich embodied interactions to the designer/user. As a result we introduce the concept of “orchestration”. By orchestration, we refer to the design of the overall interaction: coupling sensors to actuation of relevance to the affective experience; initiating and closing the interaction; habituating; helping improve on the users’ body awareness and engagement with emotional experiences; soothing, calming, or energising, depending on the affective health condition and the intentions of the designer. Through the creation of a range of prototypes and couplings we elicited requirements on broader orchestration mechanisms. First-person soma design lets researchers look afresh at biosignals that, when experienced through the body, are called to reshape affective technologies with novel ways to interpret biodata, feel it, understand it and reflect upon our bodies

THE FUTURE OF DIGITAL WORK - USE CASES FOR AUGMENTED REALITY GLASSES

Microsoft’s HoloLens enables true augmented reality (AR) by placing virtual objects within the real world. This paper aims at presenting trades (based on ISIC) that can benefit from AR as well as possible use cases. Firstly, the authors conducted a systematic literature search to identi-fy relevant papers. Six databases (including EBSCOhost, ScienceDirect and SpringerLink) were scanned for the term “HoloLens”. Out of 680 results, two researchers identified 150 articles as thematically relevant. Secondly, these papers were analysed utilising qualitative content analy-sis. Findings reveal 26 trades where AR glasses are in use for practice or research purposes. The most frequent are human health, education and research. In addition, we provide a cata-logue of 7 main use cases, such as Process Guidance or Data Access and Visualisation as well as 27 sub use cases addressing corresponding functionalities in more detail. The results of this paper are trades and application scenarios for AR glasses. Thus, this article contributes to re-search in the field of service systems design, especially AR glasses-based service systems, and provide evidence for the future of digital work