3D Pointing with Everyday Devices: Speed, Occlusion, Fatigue

In recent years, display technology has evolved to the point where displays can be both non-stereoscopic and stereoscopic, and 3D environments can be rendered realistically on many types of displays. From movie theatres and shopping malls to conference rooms and research labs, 3D information can be deployed seamlessly. Yet, while 3D environments are commonly displayed in desktop settings, there are virtually no examples of interactive 3D environments deployed within ubiquitous environments, with the exception of console gaming. At the same time, immersive 3D environments remain - in users' minds - associated with professional work settings and virtual reality laboratories. An excellent opportunity for 3D interactive engagements is being missed not because of economic factors, but due to the lack of interaction techniques that are easy to use in ubiquitous, everyday environments. In my dissertation, I address the lack of support for interaction with 3D environments in ubiquitous settings by designing, implementing, and evaluating 3D pointing techniques that leverage a smartphone or a smartwatch as an input device. I show that mobile and wearable devices may be especially beneficial as input devices for casual use scenarios, where specialized 3D interaction hardware may be impractical, too expensive or unavailable. Such scenarios include interactions with home theatres, intelligent homes, in workplaces and classrooms, with movie theatre screens, in shopping malls, at airports, during conference presentations and countless other places and situations. Another contribution of my research is to increase the potential of mobile and wearable devices for efficient interaction at a distance. I do so by showing that such interactions are feasible when realized with the support of a modern smartphone or smartwatch. I also show how multimodality, when realized with everyday devices, expands and supports 3D pointing. In particular, I show how multimodality helps to address the challenges of 3D interaction: performance issues related to the limitations of the human motor system, interaction with occluded objects and related problem of perception of depth on non-stereoscopic screens, and user subjective fatigue, measured with NASA TLX as perceived workload, that results from providing spatial input for a prolonged time. I deliver these contributions by designing three novel 3D pointing techniques that support casual, "walk-up-and-use" interaction at a distance and are fully realizable using off-the-shelf mobile and wearable devices available today. The contributions provide evidence that democratization of 3D interaction can be realized by leveraging the pervasiveness of a device that users already carry with them: a smartphone or a smartwatch.4 month

Mobile Devices at the Cinema Theatre

The pre-show experience is a significant part of the movie industry. Moviegoers, on average arrive 24 min before when the previews start. Previews have been a part of the movie experience for more than a hundred years and are a culturally significant aspect of the whole experience. Over the last decade, the premovie in-theatre experience has grown to a $600 million industry. This growth continues to accelerate. Since 2012, this industry has increased by 150%. Consequently, there is an industry-wide demand for innovation in the pre-movie area. In this paper, we describe Paths, an innovative multiplayer real-time socially engaging game that we designed, developed and evaluated. An iterative refinement application development methodology was used to create the game. The game may be played on any smartphone and group interactions are viewed on the large theatre screen. This paper also reports on the quasiexperimental mixed method study with repeated measures that was conducted to ascertain the effectiveness of this new game. The results show that Paths is very engaging with elements of suspense, pleasant unpredictability and effective team building and crowd-pleasing characteristics

Collaborative System Design of Mixed Reality Communication for Medical Training

We present the design of a mixed reality (MR) telehealth training system that aims to close the gap between in-person and distance training and re-training for medical procedures. Our system uses real-time volumetric capture as a means for communicating and relating spatial information between the non-colocated trainee and instructor. The system's design is based on a requirements elicitation study performed in situ, at a medical school simulation training center. The focus is on the lightweight real-time transmission of volumetric data - meaning the use of consumer hardware, easy and quick deployment, and low-demand computations. We evaluate the MR system design by analyzing the workload for the users during medical training. We compare in-person, video, and MR training workloads. The results indicate that the overall workload for central line placement training with MR does not increase significantly compared to video communication. Our work shows that, when designed strategically together with domain experts, an MR communication system can be used effectively for complex medical procedural training without increasing the overall workload for users significantly. Moreover, MR systems offer new opportunities for teaching due to spatial information, hand tracking, and augmented communication

Providing Language Instructor with Artificial Intelligence Assistant

This paper presents the preliminary results of developing HAL for CALL, an artificial intelligence assistant for language instructor. The assistant consists of a chatbot, an avatar (a three-dimensional visualization of the chatbot), a voice (text-to-speech engine interface) and interfaces to external sources of language knowledge. Some techniques used in adapting freely available chatbot for the need of a language learning system are presented. Integration of HAL with Second Life virtual world is proposed. We will discuss technical challenges and possible future work direction

Modeldriven software product lines

Model-driven software product lines combine the abstractio