Toward multimodality: gesture and vibrotactile feedback in natural human computer interaction

In the present work, users’ interaction with advanced systems has been investigated in different application domains and with respect to different interfaces. The methods employed were carefully devised to respond to the peculiarities of the interfaces under examination. We could extract a set of recommendations for developers. The first application domain examined regards the home. In particular, we addressed the design of a gestural interface for controlling a lighting system embedded into a piece of furniture in the kitchen. A sample of end users was observed while interacting with the virtual simulation of the interface. Based on the videoanalysis of users’ spontaneous behaviors, we could derive a set of significant interaction trends The second application domain involved the exploration of an urban environment in mobility. In a comparative study, a haptic-audio interface and an audio-visual interface were employed for guiding users towards landmarks and for providing them with information. We showed that the two systems were equally efficient in supporting the users and they were both well- received by them. In a navigational task we compared two tactile displays each embedded in a different wearable device, i.e., a glove and a vest. Despite the differences in the shape and size, both systems successfully directed users to the target. The strengths and the flaws of the two devices were pointed out and commented by users. In a similar context, two devices supported Augmented Reality technology, i.e., a pair of smartglasses and a smartphone, were compared. The experiment allowed us to identify the circumstances favoring the use of smartglasses or the smartphone. Considered altogether, our findings suggest a set of recommendations for developers of advanced systems. First, we outline the importance of properly involving end users for unveiling intuitive interaction modalities with gestural interfaces. We also highlight the importance of providing the user the chance to choose the interaction mode better fitting the contextual characteristics and to adjust the features of every interaction mode. Finally, we outline the potential of wearable devices to support interactions on the move and the importance of finding a proper balance between the amount of information conveyed to the user and the size of the device

Heads Up! Supporting Maritime Navigation using Augmented Reality

Augmented Reality (AR) is a technology that shows potential for the improvement of maritime safety. Today, the ship bridge suffers from a lack of standardization and integration. Head-Mounted Displays (HMDs) may alleviate these challenges by showing information when relevant and enhancing operator mobility. Microsoft HoloLens 2 (HL2) is such a HMD. Prior research shows the potential of HMDs in the Maritime AR domain (Rowen et al., 2019). Limited research has been conducted however on the design of AR User Interfaces (UIs) for maritime applications leveraging HMDs. As a result, no framework exists to test new UI designs in the real world, which is necessary due to many variables that cannot be accurately modelled in a lab setting. This led to the research questions (RQs) 1. What makes an effective head-mounted AR UI for maritime navigation? (RQ1); and 2. How can HL2 be used as a ship bridge system? (RQ2) A Research through Design (RtD) process is detailed where a UI design and functional prototype was developed in collaboration with end-users. The prototype, named Sjør, implements the aforementioned interface, provides a framework for in-context UI testing and can be viewed as the next step towards standardizing AR UIs for the maritime industry. The design and development process led to three contributions to the Maritime AR domain. Firstly, a framework for the visualization of location-based data about points of interest on predefined canvases co-located in the real world was developed (Technology Readiness Level (TRL) 6), which runs on the HL2. This first contribution is defined in Section 4 and provides an answer to RQ2. Secondly, using this framework, an interface design (including interactions) is developed in collaboration with end-users and proposed as an answer to RQ1. This process is described in Section 5. The third contribution is a research agenda which provides insights into how contemporary and future research can leverage the developed framework. Section 7 discloses this research agenda.Master's Thesis in Interaction and Media DesignMIX350MASV-MI

Augmented Reality-based Indoor Navigation: A Comparative Analysis of Handheld Devices vs. Google Glass

© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. U. Rehman, & S. Cao. (2017). IEEE Transactions on Human-Machine Systems, 47(1), 140–151. https://doi.org/10.1109/THMS.2016.2620106Navigation systems have been widely used in outdoor environments, but indoor navigation systems are still in early development stages. In this paper, we introduced an augmented-reality-based indoor navigation application to assist people navigate in indoor environments. The application can be implemented on electronic devices such as a smartphone or a head-mounted device. In particular, we examined Google Glass as a wearable head-mounted device in comparison with handheld navigation aids including a smartphone and a paper map. We conducted both a technical assessment study and a human factors study. The technical assessment established the feasibility and reliability of the system. The human factors study evaluated human-machine system performance measures including perceived accuracy, navigation time, subjective comfort, subjective workload, and route memory retention. The results showed that the wearable device was perceived to be more accurate, but other performance and workload results indicated that the wearable device was not significantly different from the handheld smartphone. We also found that both digital navigation aids were better than the paper map in terms of shorter navigation time and lower workload, but digital navigation aids resulted in worse route retention. These results could provide empirical evidence supporting future designs of indoor navigation systems. Implications and future research were also discussed.This work was supported in part by NSERC Discovery Grant RGPIN-2015-04134

Mixed Reality Interiors: Exploring Augmented Reality Immersive Space Planning Design Archetypes for the Creation of Interior Spatial Volume 3D User Interfaces

Augmented reality is an increasingly relevant medium of interaction and media reception with the advances in user worn or hand-held input/output technologies endowing perception of the digital nested within and reactive to the native physical. Our interior spaces are becoming the media interface and this emergence affords designers the opportunity to delve further into crafting an aesthetics for the medium. Beyond having the virtual assets and applications in correct registration with the real-world environment, critical topics are addressed such as the compositional roles of virtual and physical design features including their purpose, modulation, interaction potentials and implementation into varying indoor settings. Examining and formulating methodologies for mixed reality interior 3D UI schemes derived from the convergence of digital media and interior design disciplines comprise the scope of this design research endeavor. A holistic approach is investigated to produce a framework for augmented reality 3D user interface interiors through research and development of pattern language systems for the balanced blending of complimentary digital and physical design elements. These foundational attributes serve in the creation, organization and exploration of interactive possibilities and implications of these hybrid futuristic spatial interface layouts.M.S., Digital Media -- Drexel University, 201

MRsive: an Augmented Reality Tool for Enhancing Wayfinding and Engagement With Art in Museums

Most museums use printed methods to support indoor navigation and visitor engagement. However, modern museum visitors’ needs are not always met using such static approaches. This thesis investigates how indoor wayfinding and visitor engagement in museums might be improved through augmented reality (AR). I design “MRsive”, a handheld AR tool using a user-centered design approach. The goal is twofold: simplifying the required cognitive effort in navigating the museum space, and boosting visitor engagement with artifacts through multisensory interactions. MRsive uses computer vision to detect visual features of the space and locate visitors indoors and recognize exhibits. To evaluate my design, I conduct user-testing at the Art Gallery of Ontario (AGO) followed by semi-structured interviews. The participants’ responses showed a considerable improvement in the speed, accuracy and ease-of-use when completing wayfinding or engagement tasks. I hope the findings will contribute to the future development of other AR systems with similar goals

A Flight Test Study to Assess the Utility of an Aircraft Referenced 3D Audio Display to Improve Pilot Performance under High Workload Conditions

A study to assess the utility of an aircraft referenced 3D audio display was undertaken to determine if there could be any improvements to pilot performance when operating under high workload conditions. Test subjects flew a general aviation light twin-engine aircraft under simulated single-pilot instrument flight rule conditions. Workload was elevated by ensuring each test subject had to execute an unexpected missed approach procedure and simultaneously handle a simulated engine failure. Subjective data was gathered using the NASA Task Load Index and a post-flight questionnaire on perceived performance, workload and situational awareness. Objective data on pilot performance was gathered using the research aircraft’s onboard instrumentation system. Within the limitations of having a low number (5) of test subjects available, subjective data results showed a perceived increase in situational awareness, performance, and a statistically significant reduction in workload. Although not statistically significant, the only objective impact to performance was a slight increase in heading control and course intercept. There was no corresponding performance increase in airspeed control, angle of bank control, or improvements to aircraft track. Overall, the results indicate that a 3D audio display would have utility and pilot acceptance as a supplemental navigational display, but would not result in any substantial improvements to pilot performance