Augmented Reality Interfaces for Procedural Tasks

Procedural tasks involve people performing established sequences of activities while interacting with objects in the physical environment to accomplish particular goals. These tasks span almost all aspects of human life and vary greatly in their complexity. For some simple tasks, little cognitive assistance is required beyond an initial learning session in which a person follows one-time compact directions, or even intuition, to master a sequence of activities. In the case of complex tasks, procedural assistance may be continually required, even for the most experienced users. Approaches for rendering this assistance employ a wide range of written, audible, and computer-based technologies. This dissertation explores an approach in which procedural task assistance is rendered using augmented reality. Augmented reality integrates virtual content with a user's natural view of the environment, combining real and virtual objects interactively, and aligning them with each other. Our thesis is that an augmented reality interface can allow individuals to perform procedural tasks more quickly while exerting less effort and making fewer errors than other forms of assistance. This thesis is supported by several significant contributions yielded during the exploration of the following research themes: What aspects of AR are applicable and beneficial to the procedural task problem? In answering this question, we developed two prototype AR interfaces that improve procedural task accomplishment. The first prototype was designed to assist mechanics carrying out maintenance procedures under field conditions. An evaluation involving professional mechanics showed our prototype reduced the time required to locate procedural tasks and resulted in fewer head movements while transitioning between tasks. Following up on this work, we constructed another prototype that focuses on providing assistance in the underexplored psychomotor phases of procedural tasks. This prototype presents dynamic and prescriptive forms of instruction and was evaluated using a demanding and realistic alignment task. This evaluation revealed that the AR prototype allowed participants to complete the alignment more quickly and accurately than when using an enhanced version of currently employed documentation systems. How does the user interact with an AR application assisting with procedural tasks? The application of AR to the procedural task problem poses unique user interaction challenges. To meet these challenges, we present and evaluate a novel class of user interfaces that leverage naturally occurring and otherwise unused affordances in the native environment to provide a tangible user interface for augmented reality applications. This class of techniques, which we call Opportunistic Controls, combines hand gestures, overlaid virtual widgets, and passive haptics to form an interface that was proven effective and intuitive during quantitative evaluation. Our evaluation of these techniques includes a qualitative exploration of various preferences and heuristics for Opportunistic Control-based designs

SHIPBOARD 3-M PROGRAM SUPPLEMENTED BY AR TECHNOLOGY

The Navy maintenance program suffers from many inefficiencies, including poor labeling practices, difficult component identification, unspecific descriptions of component location in spaces for repair, and complicated diagrams. Maintenance programs onboard United States Naval Ships are a critical program to ensure they are prepared for combat and their duties by doing routine maintenance to equipment and keeping them in optimal working condition through repair. As mission difficulty and pace increases, these programs need to be carried out with fewer errors and more efficiently. Augmented reality (AR) technology can be used to identify and label all components in a space to assist with correctly identifying equipment and provide virtual instructions with critical, step-by-step information for conducting maintenance, inspections, repair work, and Damage Control (DC) events. Utilizing AR technology, Sailors or outside activity (i.e., contractors and shore-based repair facility Sailors) can enter a ship compartment and rapidly and accurately carry out a variety of maintenance program tasks. This technology would be particularly beneficial to inexperienced Sailors and outside activity by bolstering their limited knowledge and assisting them in the identification and prioritization of critical tasks and items, while simultaneously reducing the time required, and number of errors committed, while performing those tasks.Lieutenant, United States NavyApproved for public release. Distribution is unlimited

An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task

Many studies have confirmed the benefits of using Augmented Reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware for complex assembly tasks. For this research, previously published data using Desktop Model Based Instructions (MBI), Tablet MBI, and Tablet AR instructions were compared to new assembly data collected using AR instructions on the Microsoft HoloLens Head Mounted Display (HMD). Participants completed a mock wing assembly task, and measures like completion time, error count, Net Promoter Score, and qualitative feedback were recorded. The HoloLens condition yielded faster completion times than all other conditions. HoloLens users also had lower error rates than those who used the non-AR conditions. Despite the performance benefits of the HoloLens AR instructions, users of this condition reported lower net promoter scores than users of the Tablet AR instructions. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the user feedback favored the Tablet AR condition, the HoloLens condition resulted in significantly faster assembly times. As a result, it is recommended to use the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders better understand the benefits of different AR technology for manual assembly tasks

An Innovative Process to Select Augmented Reality (AR) Technology for Maintenance

Augmented Reality (AR) technology for maintenance aims to improve human performances by providing relevant information regarding both corrective and preventive maintenance. The development of an AR system involves the choice of a hardware, a development software and a visualisation method. These selections are challenging due to the wide choice of services and options available which result in fragmentation: different development processes and different user experiences. In order to ease the selection of an AR system for supporting maintenance operations, this paper proposes an innovative process. It guides the reader to identify the requirements and the constraints for any specific application through a number of questions developed in this study to help with the selection. This results in suggestions for the selection of the hardware, the development software and the visualisation method. The process is built based on a literature study, grey documents and experts interviews. Future works includes the validation of the selection process proposed in this project. It could be done by comparing the choices made using the proposed process with the choices made by experts for the same case study. Moreover, the decisional process could be extended to face the economical and ergonomics aspects related with the selection of an AR system. It could be done expanding the literature research including studies which investigate into the economical and ergonomics consequences of the application or AR for maintenance

From Paper Manual to AR Manual: Do We Still Need Text?

Abstract In this work, we proposed a method to reduce text in technical documentation, aiming at Augmented Reality manuals, where text must be reduced as much as possible. In fact, most of technical information is conveyed through other means such as CAD models, graphic signs, images, etc.. The method classifies technical instructions into two categories: instructions that can be presented with graphic symbols and instructions that should be presented with text. It is based on the analysis of the action verbs used in the instruction, and makes use of ASD Simplified Technical English (STE) for remaining text instructions and let them easier to translate into other languages

Effects of Augmented Reality on Student Achievement and Self-Efficacy in Vocational Education and Training

This study aimed to test the impact of augmented reality (AR) use on student achievement and self-efficacy in vocational education and training. For this purpose, a marker-based AR application, called HardwareAR, was developed. HardwareAR provides information about characteristics of hardware components, ports and assembly. The research design was quasi experimental with pre-test post-test that included a control group. The study was conducted with 46 undergraduate students in the Computer Hardware Course. Computer hardware course achievement test, motherboard assembly self-efficacy questionnaire and unstructured observation form were used in the study for data collection purposes. The control group learned the theoretical and applied information about motherboard assembly by using their textbooks (print material) while students in the experimental group used HardwareAR application for the same purpose. It was found that the use of AR had a positive impact on student achievement in motherboard assembly whereas it had no impact on students’ self-efficacy related to theoretical knowledge and assembly skills. On the other hand use of AR helped learners to complete the assembly process in a shorter time with less support. It is concluded that compared to control group students, experimental group students were more successful in computer hardware courses. This result shows that AR application can be effective in increasing achievement. It was concluded that AR application had no effect on students’ motherboard assembly theoretical knowledge self-efficacy and motherboard assembly skills self-efficacy. This result may have been affected from the fact that students had high levels of theoretical knowledge and assembly skills before the implementation. Observations showed that AR application enabled students to assemble motherboard in a shorter time with less support. It is thought that simultaneous interaction between virtual objects and real world provided by the AR application is effective in reducing assembly time. The students who were able to see the process steps and instructions directly with the help of HardwareAR application could complete the assembly by getting less help. Considering these results, it can be argued that, thanks to simultaneous interaction it provides, AR offers an important alternative for topics that need learner application and practice

An Augmented Reality Application for Personalised Diamond Shopping

This paper presents an Augmented Reality (AR) Application (App) developed for bespoke jewellery solutions. The App allows users to experience a visual representation of diamond rings through their mobile device. Customers could design and review bespoke jewellery without having to visit the jeweller’s shop and could communicate changes to the jeweller in real-time (saving time and money). The use of AR would allow customers to view their bespoke jewellery on their fingers to gain a better idea of how it would look once completed. Twenty seven participants evaluated the App by completing a questionnaire after using it. The application of Exploratory Factor Analysis resulted in four factors (compatibility, likeability, functionality and usability). Overall, the participants appreciated the AR App; regardless of their gender, age and experience

Experimentation of a New TeleAssistance System Using Augmented Reality

ISBN : 2-9515730-9-XInternational audienceT.A.C system was built in order to propose a new mode of remote communication. When somebody needs to be assisted on a manual task, classical technical support methods such as phone or visioconference rapidly show their limits in terms of human interaction. By allowing a distant expert to see exactly what an operator is seeing, he could interact with him in interactive time thanks to augmented reality via an adequate interaction paradigm named "Picking Outlining Adding". By "simulating" co-presence of the expert close to the operator through visual guidance information, ambiguity of language and difficulty of communication are avoided, and operations are easily performed. Scientific experimentation we have conducted and we describe in this article shows the teaching interest and the efficiency of this new mode of communication. The operator learns and operates more rapidly, in complete serenity, increasing reliability of his tasks. Moreover, throughout this paper, we show that the developed principles are sufficiently general to be applied to other use cases of tele-assistance that go beyond the originally planned industrial maintenance

Semi-automatic Design for Disassembly Strategy Planning: An Augmented Reality Approach

Abstract The mounting attention to environmental issues requires adopting better disassembly procedures at the product's End of Life. Planning and reckoning different disassembly strategies in the early stage of the design process can improve the development of sustainable products with an easy dismissing and recycling oriented approach. Nowadays many Computer Aided Process Planning software packages provide optimized assembly or disassembly sequences, but they are mainly based on a time and cost compression approach, neglecting the human factor. The environment we developed is based upon the integration of a CAD, an Augmented Reality tool, a Leap Motion Controller device, see-through glasses and an algorithm for disassembly strategies evaluation: this approach guarantees a more effective interaction with the 3D real and virtual assembly than an approach relying only on a CAD based disassembly sequence planning. In such a way, the operator may not test in a more natural and intuitive way automatic disassembly sequences, but he/she can also propose different strategies to improve the ergonomics. The methodology has been tested in a real case study to evaluate the strength points and criticalities of this approach