Toward General Purpose 3D User Interfaces: Extending Windowing Systems to Three Dimensions

Recent growth in the commercial availability of consumer grade 3D user interface devices like the Microsoft Kinect and the Oculus Rift, coupled with the broad availability of high performance 3D graphics hardware, has put high quality 3D user interfaces firmly within the reach of consumer markets for the first time ever. However, these devices require custom integration with every application which wishes to use them, seriously limiting application support, and there is no established mechanism for multiple applications to use the same 3D interface hardware simultaneously. This thesis proposes that these problems can be solved in the same way that the same problems were solved for 2D interfaces: by abstracting the input hardware behind input primitives provided by the windowing system and compositing the output of applications within the windowing system before displaying it. To demonstrate the feasibility of this approach this thesis also presents a novel Wayland compositor which allows clients to create 3D interface contexts within a 3D interface space in the same way that traditional windowing systems allow applications to create 2D interface contexts (windows) within a 2D interface space (the desktop), as well as allowing unmodified 2D Wayland clients to window into the same 3D interface space and receive standard 2D input events. This implementation demonstrates the ability of consumer 3D interface hardware to support a 3D windowing system, the ability of this 3D windowing system to support applications with compelling 3D interfaces, the ability of this style of windowing system to be built on top of existing hardware accelerated graphics and windowing infrastructure, and the ability of such a windowing system to support unmodified 2D interface applications windowing into the same 3D windowing space as the 3D interface applications. This means that application developers could create compelling 3D interfaces with no knowledge of the hardware that supports them, that new hardware could be introduced without needing to integrate it with individual applications, and that users could mix whatever 2D and 3D applications they wish in an immersive 3D interface space regardless of the details of the underlying hardware

Improving everyday computing tasks with head-mounted displays

The proliferation of consumer-affordable head-mounted displays (HMDs) has brought a rash of entertainment applications for this burgeoning technology, but relatively little research has been devoted to exploring its potential home and office productivity applications. Can the unique characteristics of HMDs be leveraged to improve users’ ability to perform everyday computing tasks? My work strives to explore this question. One significant obstacle to using HMDs for everyday tasks is the fact that the real world is occluded while wearing them. Physical keyboards remain the most performant devices for text input, yet using a physical keyboard is difficult when the user can’t see it. I developed a system for aiding users typing on physical keyboards while wearing HMDs and performed a user study demonstrating the efficacy of my system. Building on this foundation, I developed a window manager optimized for use with HMDs and conducted a user survey to gather feedback. This survey provided evidence that HMD-optimized window managers can provide advantages that are difficult or impossible to achieve with standard desktop monitors. Participants also provided suggestions for improvements and extensions to future versions of this window manager. I explored the issue of distance compression, wherein users tend to underestimate distances in virtual environments relative to the real world, which could be problematic for window managers or other productivity applications seeking to leverage the depth dimension through stereoscopy. I also investigated a mitigation technique for distance compression called minification. I conducted multiple user studies, providing evidence that minification makes users’ distance judgments in HMDs more accurate without causing detrimental perceptual side effects. This work also provided some valuable insight into the human perceptual system. Taken together, this work represents valuable steps toward leveraging HMDs for everyday home and office productivity applications. I developed functioning software for this purpose, demonstrated its efficacy through multiple user studies, and also gathered feedback for future directions by having participants use this software in simulated productivity tasks

Spatial embodied augmented reality: design of AR for spatial productivity applications

Recently there has been a growth in commercial systems for Augmented Reality (AR), such as the Microsoft HoloLens and Magic Leap, which aim to subsume traditionally 2D productivity tasks performed by previous technologies through the use of a 3D User Interface (3DUI). However, there is currently a lack of research into how to design effective AR interfaces and no consensus on how to design general purpose 3DUIs. Augmented reality is an emergent technology with little prior design precedent. As AR becomes more widespread, it becomes apparent that new design paradigms are required to translate potential benefits of these new interfaces. This thesis aims to explore the design of AR productivity software that can exploit the spatial nature of augmented reality to complement the spatial nature of human interaction. To investigate this, three user studies were performed using an email client as a use case scenario. Each study focused on a different aspect in the design of an augmented reality system starting with user elicitation using informance design methods. This progressed to testing of alternative interfaces for three-dimensional document presentation using a formal lab experiment, finally ending with an observation of how users arrange documents in a simulated limited prototype system. The findings of this thesis include: • A new method of user elicitation for augmented reality interface design called “spatial informance design”. • Design recommendations of spatial interface augmentations for email. • Data to support that space can be used to triage email more effectively. • Different layouts of documents in space provide greater or lesser time, accuracy and memorability. • Presentation of document layouts with either an ego or exocentric view alters performance. • Users have a preference to use space over colour to group documents. • Users take advantage of space when a 3rd dimension is available. The conclusion of this thesis is that augmented reality has the potential to improve the user experience over the traditional two-dimensional GUI for knowledge work tasks

Enhanced Virtuality: Increasing the Usability and Productivity of Virtual Environments

Mit stetig steigender Bildschirmauflösung, genauerem Tracking und fallenden Preisen stehen Virtual Reality (VR) Systeme kurz davor sich erfolgreich am Markt zu etablieren. Verschiedene Werkzeuge helfen Entwicklern bei der Erstellung komplexer Interaktionen mit mehreren Benutzern innerhalb adaptiver virtueller Umgebungen. Allerdings entstehen mit der Verbreitung der VR-Systeme auch zusätzliche Herausforderungen: Diverse Eingabegeräte mit ungewohnten Formen und Tastenlayouts verhindern eine intuitive Interaktion. Darüber hinaus zwingt der eingeschränkte Funktionsumfang bestehender Software die Nutzer dazu, auf herkömmliche PC- oder Touch-basierte Systeme zurückzugreifen. Außerdem birgt die Zusammenarbeit mit anderen Anwendern am gleichen Standort Herausforderungen hinsichtlich der Kalibrierung unterschiedlicher Trackingsysteme und der Kollisionsvermeidung. Beim entfernten Zusammenarbeiten wird die Interaktion durch Latenzzeiten und Verbindungsverluste zusätzlich beeinflusst. Schließlich haben die Benutzer unterschiedliche Anforderungen an die Visualisierung von Inhalten, z.B. Größe, Ausrichtung, Farbe oder Kontrast, innerhalb der virtuellen Welten. Eine strikte Nachbildung von realen Umgebungen in VR verschenkt Potential und wird es nicht ermöglichen, die individuellen Bedürfnisse der Benutzer zu berücksichtigen. Um diese Probleme anzugehen, werden in der vorliegenden Arbeit Lösungen in den Bereichen Eingabe, Zusammenarbeit und Erweiterung von virtuellen Welten und Benutzern vorgestellt, die darauf abzielen, die Benutzerfreundlichkeit und Produktivität von VR zu erhöhen. Zunächst werden PC-basierte Hardware und Software in die virtuelle Welt übertragen, um die Vertrautheit und den Funktionsumfang bestehender Anwendungen in VR zu erhalten. Virtuelle Stellvertreter von physischen Geräten, z.B. Tastatur und Tablet, und ein VR-Modus für Anwendungen ermöglichen es dem Benutzer reale Fähigkeiten in die virtuelle Welt zu übertragen. Des Weiteren wird ein Algorithmus vorgestellt, der die Kalibrierung mehrerer ko-lokaler VR-Geräte mit hoher Genauigkeit und geringen Hardwareanforderungen und geringem Aufwand ermöglicht. Da VR-Headsets die reale Umgebung der Benutzer ausblenden, wird die Relevanz einer Ganzkörper-Avatar-Visualisierung für die Kollisionsvermeidung und das entfernte Zusammenarbeiten nachgewiesen. Darüber hinaus werden personalisierte räumliche oder zeitliche Modifikationen vorgestellt, die es erlauben, die Benutzerfreundlichkeit, Arbeitsleistung und soziale Präsenz von Benutzern zu erhöhen. Diskrepanzen zwischen den virtuellen Welten, die durch persönliche Anpassungen entstehen, werden durch Methoden der Avatar-Umlenkung (engl. redirection) kompensiert. Abschließend werden einige der Methoden und Erkenntnisse in eine beispielhafte Anwendung integriert, um deren praktische Anwendbarkeit zu verdeutlichen. Die vorliegende Arbeit zeigt, dass virtuelle Umgebungen auf realen Fähigkeiten und Erfahrungen aufbauen können, um eine vertraute und einfache Interaktion und Zusammenarbeit von Benutzern zu gewährleisten. Darüber hinaus ermöglichen individuelle Erweiterungen des virtuellen Inhalts und der Avatare Einschränkungen der realen Welt zu überwinden und das Erlebnis von VR-Umgebungen zu steigern

Réalité Augmentée et Environnement Collaboratif : Un Tour d'Horizon

National audienceLa Réalité Augmentée (RA) est généralement définie comme une branche dérivée de la Réalité Virtuelle. D'une façon plus générale, le concept de réalité augmentée regroupe une approche multidisciplinaire visant un mélange entre réel et virtuel. La forte potentialité induite par cette connexion promet un cadre adéquat pour l'interaction 3D ou les applications collaboratives. On présente dans cet article un tour d'horizon des principaux travaux menés à ce jour dans le cadre de l'image et de la RA et plus particulièrement le cadre collaboratif

Desarrollo de un escritorio digital para la captura, transcripción y gestión multimodal e interactiva de documentos manuscritos

[EN] This Master Thesis presents the design and implementation of the human-computer interface for the multidisciplinary project titled "Captura, transcripción y gestión multimodal e interactiva de documentos manuscritos". The objective of the interface is to help the user to manipulate the system, so it will be easy, intuitive and efficient. It has been developed a basic visualization and document management system, that allows the user to interact with the different parts of the system, as the document capture, processing, edition, printing, and storage[ES] El presente trabajo final de Máster consiste en el diseño e implementación de la interfaz humanocomputador para el proyecto multidisciplinar titulado “Captura, transcripción y gestión multimodal e interactiva de documentos manuscritos”. Dicha interfaz tiene como objetivo facilitar al usuario la utilización del sistema, de forma que pueda realizar las tareas de dicho proyecto de una forma sencilla, intuitiva y eficaz. Para ello se ha diseñado un sistema básico de visualización y gestión de documentos, que permite interaccionar con las distintas partes del sistema, como son la captura de documentos, su procesamiento, edición, impresión y almacenamiento.Vicente Parra, FJ. (2013). Desarrollo de un escritorio digital para la captura, transcripción y gestión multimodal e interactiva de documentos manuscritos. http://hdl.handle.net/10251/44381Archivo delegad

Design Patterns for Situated Visualization in Augmented Reality

Situated visualization has become an increasingly popular research area in the visualization community, fueled by advancements in augmented reality (AR) technology and immersive analytics. Visualizing data in spatial proximity to their physical referents affords new design opportunities and considerations not present in traditional visualization, which researchers are now beginning to explore. However, the AR research community has an extensive history of designing graphics that are displayed in highly physical contexts. In this work, we leverage the richness of AR research and apply it to situated visualization. We derive design patterns which summarize common approaches of visualizing data in situ. The design patterns are based on a survey of 293 papers published in the AR and visualization communities, as well as our own expertise. We discuss design dimensions that help to describe both our patterns and previous work in the literature. This discussion is accompanied by several guidelines which explain how to apply the patterns given the constraints imposed by the real world. We conclude by discussing future research directions that will help establish a complete understanding of the design of situated visualization, including the role of interactivity, tasks, and workflows.Comment: To appear in IEEE VIS 202

ARWin -- A Desktop Augmented Reality Window Manager

We present ARWin, a single user 3D augmented reality desktop. We explain our design considerations and system architecture and discuss a variety of applications and interaction techniques designed to take advantage of this new platform