Spatial Interaction for Immersive Mixed-Reality Visualizations

Growing amounts of data, both in personal and professional settings, have caused an increased interest in data visualization and visual analytics. Especially for inherently three-dimensional data, immersive technologies such as virtual and augmented reality and advanced, natural interaction techniques have been shown to facilitate data analysis. Furthermore, in such use cases, the physical environment often plays an important role, both by directly influencing the data and by serving as context for the analysis. Therefore, there has been a trend to bring data visualization into new, immersive environments and to make use of the physical surroundings, leading to a surge in mixed-reality visualization research. One of the resulting challenges, however, is the design of user interaction for these often complex systems. In my thesis, I address this challenge by investigating interaction for immersive mixed-reality visualizations regarding three core research questions: 1) What are promising types of immersive mixed-reality visualizations, and how can advanced interaction concepts be applied to them? 2) How does spatial interaction benefit these visualizations and how should such interactions be designed? 3) How can spatial interaction in these immersive environments be analyzed and evaluated? To address the first question, I examine how various visualizations such as 3D node-link diagrams and volume visualizations can be adapted for immersive mixed-reality settings and how they stand to benefit from advanced interaction concepts. For the second question, I study how spatial interaction in particular can help to explore data in mixed reality. There, I look into spatial device interaction in comparison to touch input, the use of additional mobile devices as input controllers, and the potential of transparent interaction panels. Finally, to address the third question, I present my research on how user interaction in immersive mixed-reality environments can be analyzed directly in the original, real-world locations, and how this can provide new insights. Overall, with my research, I contribute interaction and visualization concepts, software prototypes, and findings from several user studies on how spatial interaction techniques can support the exploration of immersive mixed-reality visualizations.Zunehmende Datenmengen, sowohl im privaten als auch im beruflichen Umfeld, führen zu einem zunehmenden Interesse an Datenvisualisierung und visueller Analyse. Insbesondere bei inhärent dreidimensionalen Daten haben sich immersive Technologien wie Virtual und Augmented Reality sowie moderne, natürliche Interaktionstechniken als hilfreich für die Datenanalyse erwiesen. Darüber hinaus spielt in solchen Anwendungsfällen die physische Umgebung oft eine wichtige Rolle, da sie sowohl die Daten direkt beeinflusst als auch als Kontext für die Analyse dient. Daher gibt es einen Trend, die Datenvisualisierung in neue, immersive Umgebungen zu bringen und die physische Umgebung zu nutzen, was zu einem Anstieg der Forschung im Bereich Mixed-Reality-Visualisierung geführt hat. Eine der daraus resultierenden Herausforderungen ist jedoch die Gestaltung der Benutzerinteraktion für diese oft komplexen Systeme. In meiner Dissertation beschäftige ich mich mit dieser Herausforderung, indem ich die Interaktion für immersive Mixed-Reality-Visualisierungen im Hinblick auf drei zentrale Forschungsfragen untersuche: 1) Was sind vielversprechende Arten von immersiven Mixed-Reality-Visualisierungen, und wie können fortschrittliche Interaktionskonzepte auf sie angewendet werden? 2) Wie profitieren diese Visualisierungen von räumlicher Interaktion und wie sollten solche Interaktionen gestaltet werden? 3) Wie kann räumliche Interaktion in diesen immersiven Umgebungen analysiert und ausgewertet werden? Um die erste Frage zu beantworten, untersuche ich, wie verschiedene Visualisierungen wie 3D-Node-Link-Diagramme oder Volumenvisualisierungen für immersive Mixed-Reality-Umgebungen angepasst werden können und wie sie von fortgeschrittenen Interaktionskonzepten profitieren. Für die zweite Frage untersuche ich, wie insbesondere die räumliche Interaktion bei der Exploration von Daten in Mixed Reality helfen kann. Dabei betrachte ich die Interaktion mit räumlichen Geräten im Vergleich zur Touch-Eingabe, die Verwendung zusätzlicher mobiler Geräte als Controller und das Potenzial transparenter Interaktionspanels. Um die dritte Frage zu beantworten, stelle ich schließlich meine Forschung darüber vor, wie Benutzerinteraktion in immersiver Mixed-Reality direkt in der realen Umgebung analysiert werden kann und wie dies neue Erkenntnisse liefern kann. Insgesamt trage ich mit meiner Forschung durch Interaktions- und Visualisierungskonzepte, Software-Prototypen und Ergebnisse aus mehreren Nutzerstudien zu der Frage bei, wie räumliche Interaktionstechniken die Erkundung von immersiven Mixed-Reality-Visualisierungen unterstützen können

Ambient Gestures

We present Ambient Gestures, a novel gesture-based system designed to support ubiquitous ‘in the environment’ interactions with everyday computing technology. Hand gestures and audio feedback allow users to control computer applications without reliance on a graphical user interface, and without having to switch from the context of a non-computer task to the context of the computer. The Ambient Gestures system is composed of a vision recognition software application, a set of gestures to be processed by a scripting application and a navigation and selection application that is controlled by the gestures. This system allows us to explore gestures as the primary means of interaction within a multimodal, multimedia environment. In this paper we describe the Ambient Gestures system, define the gestures and the interactions that can be achieved in this environment and present a formative study of the system. We conclude with a discussion of our findings and future applications of Ambient Gestures in ubiquitous computing

Defining Interaction within Immersive Virtual Environments

PhDThis thesis is concerned with the design of Virtual Environments (YEs) - in particular with the tools and techniques used to describe interesting and useful environments. This concern is not only with respect to the appearance of objects in the VE but also with their behaviours and their reactions to actions of the participants. The main research hypothesis is that there are several advantages to constructing these interactions and behaviours whilst remaining immersed within the VE which they describe. These advantages include the fact that editing is done interactively with immediate effect and without having to resort to the usual edit-compile-test cycle. This means that the participant doesn't have to leave the VE and lose their sense of presence within it, and editing tasks can take advantage of the enhanced spatial cognition and naturalistic interaction metaphors a VE provides. To this end a data flow dialogue architecture with an immersive virtual environment presentation system was designed and built. The data flow consists of streams of data that originate at sensors that register the body state of the participant, flowing through filters that modify the streams and affect the yE. The requirements for such a system and the filters it should contain are derived from two pieces of work on interaction metaphors, one based on a desktop system using a novel input device and the second a navigation technique for an immersive system. The analysis of these metaphors highlighted particular tasks that such a virtual environment dialogue architecture (VEDA) system might be used to solve, and illustrate the scope of interactions that should be accommodated. Initial evaluation of the VEDA system is provided by moderately sized demonstration environments and tools constructed by the author. Further evaluation is provided by an in-depth study where three novice VE designers were invited to construct VEs with the VEDA system. This highlighted the flexibility that the VEDA approach provides and the utility of the immersive presentation over traditional techniques in that it allows the participant to use more natural and expressive techniques in the construction process. In other words the evaluation shows how the immersive facilities of VEs can be exploited in the process of constructing further VEs

An empirical comparison of interaction styles for map interfaces in immersive virtual environments

Geographical Information Systems (GIS) can be visualized using immersive technologies like Virtual Reality (VR). Before using this kind of technologies it is required to explore which interactions are affordable, efficient and satisfactory from the users' point of view. The purpose of this work is to provide insight on how to design efficient and natural interaction on GIS VR interfaces. This study presents a within-subjects comparative study that assesses the usability and performance of two popular interaction strategies: body-based interaction and device based interaction. In body-based interaction, participants use their hands and head orientation to control the VR map. In the second case, users interact with the Oculus Touch controller. Thirty two users participated in an experiment whose results suggest that interacting with the controller improves performance of the selection task, in terms of time spent and error rate. Also, the results show a preference of users for the controller in terms of perceived usability.This work is supported by the project PACE funded by the Spanish Ministry of Economy, Industry and Competitiveness (TIN2016-77690-R

GIFT: Gesture-Based Interaction by Fingers Tracking, an Interaction Technique for Virtual Environment

Three Dimensional (3D) interaction is the plausible human interaction inside a Virtual Environment (VE). The rise of the Virtual Reality (VR) applications in various domains demands for a feasible 3D interface. Ensuring immersivity in a virtual space, this paper presents an interaction technique where manipulation is performed by the perceptive gestures of the two dominant fingers; thumb and index. The two fingertip-thimbles made of paper are used to trace states and positions of the fingers by an ordinary camera. Based on the positions of the fingers, the basic interaction tasks; selection, scaling, rotation, translation and navigation are performed by intuitive gestures of the fingers. Without keeping a gestural database, the features-free detection of the fingers guarantees speedier interactions. Moreover, the system is user-independent and depends neither on the size nor on the color of the users’ hand. With a case-study project; Interactions by the Gestures of Fingers (IGF) the technique is implemented for evaluation. The IGF application traces gestures of the fingers using the libraries of OpenCV at the back-end. At the front-end, the objects of the VE are rendered accordingly using the Open Graphics Library; OpenGL. The system is assessed in a moderate lighting condition by a group of 15 users. Furthermore, usability of the technique is investigated in games. Outcomes of the evaluations revealed that the approach is suitable for VR applications both in terms of cost and accuracy

Immersive Analytics of Large Dynamic Networks via Overview and Detail Navigation

Analysis of large dynamic networks is a thriving research field, typically relying on 2D graph representations. The advent of affordable head mounted displays however, sparked new interest in the potential of 3D visualization for immersive network analytics. Nevertheless, most solutions do not scale well with the number of nodes and edges and rely on conventional fly- or walk-through navigation. In this paper, we present a novel approach for the exploration of large dynamic graphs in virtual reality that interweaves two navigation metaphors: overview exploration and immersive detail analysis. We thereby use the potential of state-of-the-art VR headsets, coupled with a web-based 3D rendering engine that supports heterogeneous input modalities to enable ad-hoc immersive network analytics. We validate our approach through a performance evaluation and a case study with experts analyzing a co-morbidity network

Evaluation of Smartphone-based interaction techniques in a CAVE in the context of immersive digital project review

Immersive digital project reviews consist in using virtual reality (VR) as a tool for discussion between various stakeholders of a project. In the automotive industry, the digital car prototype model is the common thread that binds them. It is used during immersive digital project reviews between designers, engineers, ergonomists, etc. The digital mockup is also used to assess future car architecture, habitability or perceived quality requirements with the aim to reduce using physical mockups for optimized cost, delay and quality efficiency. Among the difficulties identified by the users, handling the mockup is a major one. Inspired by current uses of nomad devices (multi-touch gestures, IPhone UI look’n’feel and AR applications), we designed a navigation technique taking advantage of these popular input devices: Space scrolling allows moving around the mockup. In this paper, we present the results of a study we conducted on the usability and acceptability of the proposed smartphone-based interaction metaphor compared to traditional technique and we provide indications of the most efficient choices for different use-cases accordingly. It was carried out in a traditional 4-sided CAVE and its purpose is to assess a chosen set of interaction techniques to be implemented in Renault’s new 5-sides 4K x 4K wall high performance CAVE. The proposed new metaphor using nomad devices is well accepted by novice VR users and future implementation should allow an efficient industrial use. Their use is an easy and user friendly alternative of the existing traditional control devices such as a joystick

Lessons from digital puppetry - Updating a design framework for a perceptual user interface

While digital puppeteering is largely used just to augment full body motion capture in digital production, its technology and traditional concepts could inform a more naturalized multi-modal human computer interaction than is currently used with the new perceptual systems such as Kinect. Emerging immersive social media networks with their fully live virtual or augmented environments and largely inexperienced users would benefit the most from this strategy. This paper intends to define digital puppeteering as it is currently understood, and summarize its broad shortcomings based on expert evaluation. Based on this evaluation it will suggest updates and experiments using current perceptual technology and concepts in cognitive processing for existing human computer interaction taxonomy. This updated framework may be more intuitive and suitable in developing extensions to an emerging perceptual user interface for the general public