Overcoming the limitations of commodity augmented reality head mounted displays for use in product assembly

Numerous studies have shown the effectiveness of utilizing Augmented Reality (AR) to deliver work instructions for complex assemblies. Traditionally, this research has been performed using hand-held displays, such as smartphones and tablets, or custom-built Head Mounted Displays (HMDs). AR HMDs have been shown to be especially effective for assembly tasks as they allow the user to remain hands-free while receiving work instructions. Furthermore, in recent years a wave of commodity AR HMDs have come to market including the Microsoft HoloLens, Magic Leap One, Meta 2, and DAQRI Smart Glasses. These devices present a unique opportunity for delivering assembly instructions due to their relatively low cost and accessibility compared to custom built AR HMD solutions of the past. Despite these benefits, the technology behind these HMDs still contains many limitations including input, user interface, spatial registration, navigation and occlusion. To accurately deliver work instructions for complex assemblies, the hardware limitations of these commodity AR HMDs must be overcome. For this research, an AR assembly application was developed for the Microsoft HoloLens using methods specifically designed to address the aforementioned issues. Input and user interface methods were implemented and analyzed to maximize the usability of the application. An intuitive navigation system was developed to guide users through a large training environment, leading them to the current point of interest. The native tracking system of the HoloLens was augmented with image target tracking capabilities to stabilize virtual content, enhance accuracy, and account for spatial drift. This fusion of marker-based and marker-less tracking techniques provides a novel approach to display robust AR assembly instructions on a commodity AR HMD. Furthermore, utilizing this novel spatial registration approach, the position of real-world objects was accurately registered to properly occlude virtual work instructions. To render the desired effect, specialized computer graphics methods and custom shaders were developed and implemented for an AR assembly application. After developing novel methods to display work instructions on a commodity AR HMD, it was necessary to validate that these work instructions were being accurately delivered. Utilizing the sensors on the HoloLens, data was collected during the assembly process regarding head position, orientation, assembly step times, and an estimation of spatial drift. With the addition of wearable physiological sensor data, this data was fused together in a visualization application to validate instructions were properly delivered and provide an opportunity for an analysist to examine trends within an assembly session. Additionally, the spatial drift data was then analyzed to gain a better understanding of how spatial drift accumulates over time and ensure that the spatial registration mitigation techniques was effective. Academic research has shown that AR may substantial reduce cost for assembly operations through a reduction in errors, time, and cognitive workload. This research provides novel solutions to overcome the limitations of commodity AR HMDs and validate their use for product assembly. Furthermore, the research provided in this thesis demonstrates the potential of commodity AR HMDs and how their limitations can be mitigated for use in product assembly tasks

A novel context-aware augmented reality framework for maintenance systems

Augmented Reality (AR) bridges the gap between the real and the virtual world by bringing virtual information to a real environment as seamlessly as possible. The need for better perception of knowledge-intensive complex maintenance tasks and access to large amounts of documents and data makes the use of AR technology promising in a maintenance domain. Context-awareness enhances the usability of such AR applications, i.e. the output and behavior of the system will be adapted according to different contexts, such as the user location, preferences, devices, etc. to afford a higher level of personalization. The adaptation needs to be efficient in terms of performance and speed. This paper presents an optimized framework which combines context-awareness and AR for training and assisting technicians in maintaining equipment in an industrial context to improve field workers effectiveness. Ontology is used to model a maintenance context, and Semantic Web Rule Language (SWRL) provides logical reasoning. This optimized framework utilizes a behavior network to select a collection of suitable actions based on the current step of an ongoing task, and applies context-based inferred information from the ontology to each member of this collection. Evaluation results comparing the performance of the proposed framework with conventional ontology alone in a maintenance domain confirmed that the proposed framework in this research provides the same results as the ontology in terms of content, but it runs much faster in terms of run-time and performance. The proposed context-aware framework is quite valuable especially in terms of response time and performance for maintenance systems with a large number of maintenance activities

Remote maintenance assistance using real-time augmented reality authoring

Maintenance operations and lifecycle engineering have largely been considered one of the most expensive and time-consuming components for industrial equipment. Numerous organizations continually devote large quantities of resources towards maintaining equipment. As such, any optimizations that would reduce maintenance errors and expenses could lead to substantial time and cost savings. Unfortunately, there are often not enough specialists to meet the demand, forcing localized technicians to perform on-site maintenance on equipment outside their area of expertise. Augmented reality (AR) is one technology that has already been shown to improve the maintenance process. While powerful, AR has its own set of challenges, from content authoring to spatial perception. This work details a system that puts both the power of AR and the knowledge of a specialist directly into the hands of an on-site technician. An application was developed that enables a specialist to deliver AR instructions in real-time to assist a technician performing on-site maintenance. Using a novel and simplified authoring interface, specialists can create AR content in real-time, with little to no prior knowledge of augmented reality or the system itself. There has been ample research on different AR-supported processes, such as real-time authoring, video monitoring, and off-site assistance. However, much less work has been done to integrate them and leverage existing personnel knowledge to both author and deliver real-time AR instructions. This work details the development and implementation of such a system. A technical evaluation was also performed to ensure real-time connectivity in geographically distributed environments. Three network configurations were evaluated. A high-latency high-bandwidth network was used to represent a typical modern maintenance facility. A low-bandwidth network was evaluated to mimic older or more isolated maintenance environments. Lastly, a 4G LTE network was tested, showing the potential for the system to be used across global locations. Under all network configurations, the system effectively facilitated the complete disassembly of a hydraulic pump assembly

Study of Augmented Reality based manufacturing for further integration of quality control 4.0: a systematic literature review

Augmented Reality (AR) has gradually become a mainstream technology enabling Industry 4.0 and its maturity has also grown over time. AR has been applied to support different processes on the shop-floor level, such as assembly, maintenance, etc. As various processes in manufacturing require high quality and near-zero error rates to ensure the demands and safety of end-users, AR can also equip operators with immersive interfaces to enhance productivity, accuracy and autonomy in the quality sector. However, there is currently no systematic review paper about AR technology enhancing the quality sector. The purpose of this paper is to conduct a systematic literature review (SLR) to conclude about the emerging interest in using AR as an assisting technology for the quality sector in an industry 4.0 context. Five research questions (RQs), with a set of selection criteria, are predefined to support the objectives of this SLR. In addition, different research databases are used for the paper identification phase following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) methodology to find the answers for the predefined RQs. It is found that, in spite of staying behind the assembly and maintenance sector in terms of AR-based solutions, there is a tendency towards interest in developing and implementing AR-assisted quality applications. There are three main categories of current AR-based solutions for quality sector, which are AR-based apps as a virtual Lean tool, AR-assisted metrology and AR-based solutions for in-line quality control. In this SLR, an AR architecture layer framework has been improved to classify articles into different layers which are finally integrated into a systematic design and development methodology for the development of long-term AR-based solutions for the quality sector in the future

A systematic review of Augmented Reality content-related techniques for knowledge transfer in maintenance applications

Augmented Reality (AR) has experienced an increasing trend in applied research in the last few years. This emerging trend is focused in content-related challenges: mainly creation (Authoring), adaptation (Context-Awareness) and improvement (Interaction-Analysis) of augmented content. Research in these techniques has enabled Academia to recognise Augmented Reality capability for knowledge transfer, either from AR systems to users or between users. But to the best of author’s knowledge, there are no specific literature review in these areas, neither on their relations with AR knowledge transfer ability. Therefore, this paper aims to identify these relations through an analysis of state-of-the-art techniques in Authoring (A), Context-Awareness (CA) and Interaction-Analysis (IA) in the context of maintenance applications. In order to do so, a Systematic Literature Review (SLR) has been conducted on 74 application-relevant papers from 2012 to 2017. It comprised a thematic analysis to establish the relation between maintenance applications, research in A, CA and IA and AR knowledge transfer modes. Its results helped to classify AR maintenance-applications by technological readiness levels. They also revealed the potential of AR for users’ knowledge capture, and future research required for full knowledge management capabilities. Furthermore, the SLR method proposed could be extended to correlate AR systems and applications by their knowledge management capabilities in any AR application context

A NOVEL AUGMENTED REALITY BASED SYSTEM FOR PROVIDING MAINTENANCE ASSISTANCE

Ph.DDOCTOR OF PHILOSOPH

Human Computer Interaction and Emerging Technologies

The INTERACT Conferences are an important platform for researchers and practitioners in the field of human-computer interaction (HCI) to showcase their work. They are organised biennially by the International Federation for Information Processing (IFIP) Technical Committee on Human–Computer Interaction (IFIP TC13), an international committee of 30 member national societies and nine Working Groups. INTERACT is truly international in its spirit and has attracted researchers from several countries and cultures. With an emphasis on inclusiveness, it works to lower the barriers that prevent people in developing countries from participating in conferences. As a multidisciplinary field, HCI requires interaction and discussion among diverse people with different interests and backgrounds. The 17th IFIP TC13 International Conference on Human-Computer Interaction (INTERACT 2019) took place during 2-6 September 2019 in Paphos, Cyprus. The conference was held at the Coral Beach Hotel Resort, and was co-sponsored by the Cyprus University of Technology and Tallinn University, in cooperation with ACM and ACM SIGCHI. This volume contains the Adjunct Proceedings to the 17th INTERACT Conference, comprising a series of selected papers from workshops, the Student Design Consortium and the Doctoral Consortium. The volume follows the INTERACT conference tradition of submitting adjunct papers after the main publication deadline, to be published by a University Press with a connection to the conference itself. In this case, both the Adjunct Proceedings Chair of the conference, Dr Usashi Chatterjee, and the lead Editor of this volume, Dr Fernando Loizides, work at Cardiff University which is the home of Cardiff University Press

Ontology-based augmented reality content-related techniques and their impact in knowledge capture and re-use within maintenance diagnosis

This PhD thesis aims to study ontology-based AR content-related methods and their impact in knowledge transfer, capture and re-use for cost-effective human knowledge integration in digital diagnostic systems. Industry 4.0 has revealed the importance of maintainers’ knowledge capture and re-use in diagnostics systems for providing satisfactory solutions in cases where those systems cannot (e.g. nofault-found). Augmented Reality (AR) utilises content-related techniques to transfer knowledge to maintainers for improving efficiency and effectiveness of diagnosis tasks. Academic literature has shown that AR can also be utilised for knowledge capture and re-use, but this has only been demonstrated in simple, step-by-step repair operations. In diagnosis research, ontology-based methods are applied to capture and re-use knowledge from unstructured and heterogenous sources like humans. Nevertheless, these methods have not made use of AR potential to contextualise knowledge and so, improve efficiency and effectiveness of knowledge capture and re-use diagnosis operations...[cont.]Manufacturin