A Review of Multimodal Interaction Technique in Augmented Reality Environment

Augmented Reality (AR) has proposed several types of interaction techniques such as 3D interactions, natural interactions, tangible interactions, spatial awareness interactions and multimodal interactions. Usually, interaction technique in AR involve unimodal interaction technique that only allows user to interact with AR content by using one modality such as gesture, speech, click, etc. Meanwhile, the combination of more than one modality is called multimodal. Multimodal can contribute to human and computer interaction more efficient and will enhance better user experience. This is because, there are a lot of issues have been found when user use unimodal interaction technique in AR environment such as fat fingers. Recent research has shown that multimodal interface (MMI) has been explored in AR environment and has been applied in various domain. This paper presents an empirical study of some of the key aspects and issues in multimodal interaction augmented reality, touching on the interaction technique and system framework. We reviewed the question of what are the interaction techniques that have been used to perform a multimodal interaction in AR environment and what are the integrated components applied in multimodal interaction AR frameworks. These two questions were used to be analysed in order to find the trends in multimodal field as a main contribution of this paper. We found that gesture, speech and touch are frequently used to manipulate virtual object. Most of the integrated component in MMI AR framework discussed only on the concept of the framework components or the information centred design between the components. Finally, we conclude this paper by providing ideas for future work involving this field

Object selection and scaling using multimodal interaction in mixed reality

Mixed Reality (MR) is the next evolution of human interacting with the computer as MR has the ability to combine the physical environment and digital environment and making them coexist with each other. Interaction is still a huge research area in Augmented Reality (AR) but very less in MR, this is due to current advanced MR display techniques still not robust and intuitive enough to let the user to naturally interact with 3D content. New techniques on user interaction have been widely studied, the advanced technique in interaction when the system able to invoke more than one input modalities. Multimodal interaction undertakes to deliver intuitive multiple objects manipulation with gestures. This paper discusses the multimodal interaction technique using gesture and speech which the proposed experimental setup to implement multimodal in the MR interface. The real hand gesture is combined with speech inputs in MR to perform spatial object manipulations. The paper explains the implementation stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, spatial manipulation for selection and scaling using multimodal interaction has been invoked, and this paper ends with a discussion

Collaborative and Multi-Modal Mixed Reality for Enhancing Cultural Learning in Virtual Heritage

No abstrac

Understanding, Modeling, and Simulating the Discrepancy Between Intended and Perceived Image Appearance on Optical See-Through Augmented Reality Displays

Augmented reality (AR) displays are transitioning from being primarily used in research and development settings, to being used by the general public. With this transition, these displays will be used by more people, in many different environments, and in many different contexts. Like other displays, the user\u27s perception of virtual imagery is influenced by the characteristics of the user\u27s environment, creating a discrepancy between the intended appearance and the perceived appearance of virtual imagery shown on the display. However, this problem is much more apparent for optical see-through AR displays, such as the HoloLens. For these displays, imagery is superimposed onto the user\u27s view of their environment, which can cause the imagery to become transparent and washed out in appearance from the user\u27s perspective. Any change in the user\u27s environment conditions or in the user\u27s position introduces changes to the perceived appearance of the AR imagery, and current AR displays do not adapt to maintain a consistent perceived appearance of the imagery being displayed. Because of this, in many environments the user may misinterpret or fail to notice information shown on the display. In this dissertation, I investigate the factors that influence user perception of AR imagery and demonstrate examples of how the user\u27s perception is affected for applications involving user interfaces, attention cues, and virtual humans. I establish a mathematical model that relates the user, their environment, their AR display, and AR imagery in terms of luminance or illuminance contrast. I demonstrate how this model can be used to classify the user\u27s viewing conditions and identify problems the user is prone to experience when in these conditions. I demonstrate how the model can be used to simulate changes in the user\u27s viewing conditions and to identify methods to maintain the perceived appearance of the AR imagery in changing conditions

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics

Augmented Reality in Sport Broadcasting

For a large portion of its history, sport broadcasting has been stagnant when it comes to incorporating new and innovative technologies. However, due to declining viewership and consumer desire for customizable content, augmented reality graphics have begun to be incorporated into multiple sport broadcast products. In fact, the UEFA Champions League, NBA, NFL, and NHL have all used or indicated their intention to utilize AR graphics in future broadcasts. Considering that media rights revenue is the main source of revenue to sport properties and organizations, it is important to carefully consider how the core product (the broadcast) is presented. The study examined consumer attitudes and intentions towards AR in sport broadcasts by utilizing three types of broadcasts of an NBA game. One of the broadcasts was a traditional broadcast format with no AR enhancement and the other two were enhanced with AR graphics, a coach-mode broadcast that featured AR player tracking and play diagramming while the other enhanced broadcast, mascot-mode, featured AR graphics similar to a video game with over-the-top animations. Results of the current study provide insight into consumer preferences towards AR in sport broadcasting and guidance to sport properties planning to utilize broadcast AR graphics. Specifically, that sport consumers were significantly more likely to re-view (p \u3c .05) and recommend via word of mouth (p \u3c .05) the coach-mode AR than the mascot-mode AR. Sport involvement was a significant factor for how sport fans perceive the AR broadcast types through incorporating the perspective of the elaboration likelihood model

Playing with Virtual Reality: Early Adopters of Commercial Immersive Technology

This dissertation examines early adopters of mass-marketed Virtual Reality (VR), as well as other immersive technologies, and the playful processes by which they incorporate the devices into their lives within New York City. Starting in 2016, relatively inexpensive head-mounted displays (HMDs) began to be manufactured and distributed by leaders in the game and information technology industries. However, even before these releases, developers and content creators were testing the devices through “development kits.” These de facto early adopters, who are distinctly commercially-oriented, acted as a launching point for the dissertation to scrutinize how, why and in what ways digital technologies spread to the wider public. Taking a multimethod approach that combines semi-structured interviews, two years of participant observation, media discourse analysis and autoethnography, the dissertation details a moment in the diffusion of an innovation and how publicity, social forces and industry influence adoption. This includes studying the media ecosystem which promotes and sustains VR, the role of New York City in framing opportunities and barriers for new users, and a description of meetups as important communities where devotees congregate. With Game Studies as a backdrop for analysis, the dissertation posits that the blurry relationship between labor and play held by most enthusiasts sustains the process of VR adoption. Their “playbor” colors not only the rhetoric and the focus of meetups, but also the activities, designs, and, most importantly, the financial and personal expenditures they put forth. Ultimately, play shapes the system of production by which adopters of commercial VR are introduced to the technology and, eventually, weave it into their lives. Situating play at the center of this system highlights that the assimilation of digital media is in part an embodied and irrational experience. It also suggests new models by which future innovations will spread to the public

The Benefits of Extended Reality for Technical Communication : Utilizing XR for Maintenance Documentation Creation and Delivery

The main goal of this dissertation is to explore the benefits of extended reality for technical communication. Both of these fields offer opportunities and also pose challenges to each other, and this dissertation provides insight into this relationship. The research was initiated by the author’s personal interest in both fields and also human-technology interaction and user needs in general. Even though this is an academic dissertation, it is first and foremost a practitioner’s view of these evolving technologies and their potential uses in industry and, specifically, in industrial maintenance and technical communication. Under the umbrella of extended reality and technical communication, this dissertation focuses on two main themes. The first part studies virtual reality as a technology to facilitate collaboration and digital content creation for technical documentation in industrial companies, and the second part explores the possibilities of augmented reality and smart glasses as a delivery channel for maintenance instructions. The developed concepts were tested by domain experts in user tests. The overall results of testing were positive, and domain experts expressed enthusiasm toward the concepts and technologies in general. The technical documentation process is an inherently collaborative process involving stakeholders from different teams and organizations, and virtual reality was evaluated to have a positive effect on that process, especially in the case of globally scattered teams. The developed tools were also rated positively for digital content creation. Therefore, virtual reality offers many benefits for technical documentation creation, an area where it has not been utilized until now. On the augmented reality side, domain experts were generally enthusiastic about the use of smart glasses even though the technologies are not yet mature enough for field use in industrial maintenance. Furthermore, the results show that content created in the technical communications industry standard, DITA XML, works well when delivered to smart glasses, and the same content can be single sourced to other delivery channels. The use of DITA XML, therefore, eliminates the need to tailor content for each delivery channel separately, and offers an effective way to create and update content for AR applications in industrial companies. This, in turn, can advance the use of AR technologies and related devices in field operations in industrial companies. In conclusion, the findings of this dissertation show that the fields of technical communication and extended reality have a significant amount of synergy. In this dissertation I establish use cases and guidelines for these areas