Designing procedure execution tools with emerging technologies for future astronauts

NASA’s human spaceflight efforts are moving towards long-duration exploration missions requiring asynchronous communication between onboard crew and an increasingly remote ground support. In current missions aboard the International Space Station, there is a near real-time communication loop between Mission Control Center and astronauts. This communication is essential today to support operations, maintenance, and science requirements onboard, without which many tasks would no longer be feasible. As NASA takes the next leap into a new era of human space exploration, new methods and tools compensating for the lack of continuous, real-time communication must be explored. The Human-Computer Interaction Group at NASA Ames Research Center has been investigating emerging technologies and their applicability to increase crew autonomy in missions beyond low Earth orbit. Interactions using augmented reality and the Internet of Things have been researched as possibilities to facilitate usability within procedure execution operations. This paper outlines four research efforts that included technology demonstrations and usability studies with prototype procedure tools implementing emerging technologies. The studies address habitat feedback integration, analogous procedure testing, task completion management, and crew training. Through these technology demonstrations and usability studies, we find that low-to medium-fidelity prototypes, evaluated early in the design process, are both effective for garnering stakeholder buy-in and developing requirements for future systems. In this paper, we present the findings of the usability studies for each project and discuss ways in which these emerging technologies can be integrated into future human spaceflight operations

Study of Augmented Reality based manufacturing for further integration of quality control 4.0: a systematic literature review

Augmented Reality (AR) has gradually become a mainstream technology enabling Industry 4.0 and its maturity has also grown over time. AR has been applied to support different processes on the shop-floor level, such as assembly, maintenance, etc. As various processes in manufacturing require high quality and near-zero error rates to ensure the demands and safety of end-users, AR can also equip operators with immersive interfaces to enhance productivity, accuracy and autonomy in the quality sector. However, there is currently no systematic review paper about AR technology enhancing the quality sector. The purpose of this paper is to conduct a systematic literature review (SLR) to conclude about the emerging interest in using AR as an assisting technology for the quality sector in an industry 4.0 context. Five research questions (RQs), with a set of selection criteria, are predefined to support the objectives of this SLR. In addition, different research databases are used for the paper identification phase following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) methodology to find the answers for the predefined RQs. It is found that, in spite of staying behind the assembly and maintenance sector in terms of AR-based solutions, there is a tendency towards interest in developing and implementing AR-assisted quality applications. There are three main categories of current AR-based solutions for quality sector, which are AR-based apps as a virtual Lean tool, AR-assisted metrology and AR-based solutions for in-line quality control. In this SLR, an AR architecture layer framework has been improved to classify articles into different layers which are finally integrated into a systematic design and development methodology for the development of long-term AR-based solutions for the quality sector in the future

Comparative Evaluation of Augmented Reality-based Assistance for Procedural Tasks: A Simulated Control Room Study

This is an Accepted Manuscript of an article published by Taylor & Francis in Behaviour & Information Technology in 2019, available online: https://doi.org/10.1080/0144929X.2019.1660805This research explores the design, implementation, and evaluation of a prototype augmented reality application that assists operators in performing procedural tasks in control room settings. Our prototype uses a tablet display to supplement an operator’s natural view of existing control panel elements with sequences of interactive visual and attention guiding cues. An experiment, conducted using a nuclear power plant simulator, examined university students completing both standard and emergency operating procedures. The augmented reality condition was compared against two other conditions – a paper-based procedure condition using paper manuals and a computer-based procedure condition using digital procedures presented on a desktop display. The results demonstrated that the augmented reality -based procedure system had benefits in terms of reduced mental workload in comparison to the other two conditions. Regarding task completion time, accuracy, and situation awareness, the augmented reality condition had no significant difference when compared against the computer-based procedure condition but performed better than the paper-based procedure condition. It was also found that the augmented reality condition resulted in fewer intra-team inquiry communication exchanges in comparison to both paper-based and computer-based conditions. The augmented reality condition, however, yielded poorer memory retention score when assessed against the other two conditions

Let’s augment the future together!:Augmented reality troubleshooting support for IT/OT rolling stock failures

The railway industry is moving to a socio-technological system that relies on computer-controlled and human-machine interfaces. Opportunities arise for creating new services and commercial business cases by using technological innovations and traffic management systems. The convergence of Information Technology (IT) with Operational Technology (OT) is critical for cost-effective and reliable railway operations. However, this convergence introduces complexities, leading to more intricate rolling stock system failures. Hence, operators necessitate assistance in their troubleshooting and maintenance strategy to simplify the decision-making and action-taking processes. Augmented Reality (AR) emerges as a pivotal tool for troubleshooting within this context. AR enhances the operator’s ability to visualize, contextualize, and understand complex data by overlaying real-time and virtual information onto physical objects. AR supports the identification of IT/OT rolling stock system failures, offers troubleshooting directions, and streamlines maintenance procedures, ultimately enhancing decision-making and action-taking processes. This thesis investigates how AR can support operators in navigating troubleshooting and maintenance challenges posed by IT/OT rolling stock system failures in the railway industry

A systematic review of augmented reality applications in maintenance

Augmented Reality (AR) technologies for supporting maintenance operations have been an academic research topic for around 50 years now. In the last decade, major progresses have been made and the AR technology is getting closer to being implemented in industry. In this paper, the advantages and disadvantages of AR have been explored and quantified in terms of Key Performance Indicators (KPI) for industrial maintenance. Unfortunately, some technical issues still prevent AR from being suitable for industrial applications. This paper aims to show, through the results of a systematic literature review, the current state of the art of AR in maintenance and the most relevant technical limitations. The analysis included filtering from a large number of publications to 30 primary studies published between 1997 and 2017. The results indicate a high fragmentation among hardware, software and AR solutions which lead to a high complexity for selecting and developing AR systems. The results of the study show the areas where AR technology still lacks maturity. Future research directions are also proposed encompassing hardware, tracking and user-AR interaction in industrial maintenance is proposed

A field study on the impacts of implementing concepts and elements of industry 4.0 in the biopharmaceutical sector

This study proposes a field study, based on a literature review, about the applications and impacts of Industry 4.0 (I4.0) in the biopharmaceutical sector. The world is facing a new industrial revolution and the central idea is the integration between the virtual and the real world through elements that will allow for a greater degree of automation and digitization of processes. The production of medicines via biological processes is a booming domain in the pharmaceutical sector, that involves extraordinary technological challenges. The fieldwork, carried out between August 2019 and February 2020, involved semi-structured interviews with managers of pharmaceutical companies and specialists in the I4.0 theme. The interviews allowed for the identification of trends and key benefits and barriers for implementing I4.0 in the biopharmaceutical sector. While the perceptions were considerably diversified, benefits in productivity, competitiveness and quality ranked among the most scored items. The main barriers, highlighted by the interviewees, refer to the need to break organizational cultural standards, the regulatory requirements, the lack of organizational strategies for implementation, and the lack of qualified professionals. This work offers a contribution to the biopharmaceutical sector and reinforces the imminent need for companies to adapt to this new reality.publishe

Human factor in intelligent manufacturing systems - knowledge acquisition and motivation

Abstract People play a central role in intelligent manufacturing systems because of two reasons: their knowledge is indispensable to create and improve intelligent manufacturing systems; and their motivation is very important to identify and solve causes of the problems which may occur in order to prevent them in the future. Therefore, adequate learning methods are required to accomplish these two goals: empower and motivate people. In this paper innovative methods such as learning by doing, simulations and virtual reality will be presented as the ways to transfer the knowledge about intelligent manufacturing systems and to increase motivation concerning their improvements

Overcoming the limitations of commodity augmented reality head mounted displays for use in product assembly

Numerous studies have shown the effectiveness of utilizing Augmented Reality (AR) to deliver work instructions for complex assemblies. Traditionally, this research has been performed using hand-held displays, such as smartphones and tablets, or custom-built Head Mounted Displays (HMDs). AR HMDs have been shown to be especially effective for assembly tasks as they allow the user to remain hands-free while receiving work instructions. Furthermore, in recent years a wave of commodity AR HMDs have come to market including the Microsoft HoloLens, Magic Leap One, Meta 2, and DAQRI Smart Glasses. These devices present a unique opportunity for delivering assembly instructions due to their relatively low cost and accessibility compared to custom built AR HMD solutions of the past. Despite these benefits, the technology behind these HMDs still contains many limitations including input, user interface, spatial registration, navigation and occlusion. To accurately deliver work instructions for complex assemblies, the hardware limitations of these commodity AR HMDs must be overcome. For this research, an AR assembly application was developed for the Microsoft HoloLens using methods specifically designed to address the aforementioned issues. Input and user interface methods were implemented and analyzed to maximize the usability of the application. An intuitive navigation system was developed to guide users through a large training environment, leading them to the current point of interest. The native tracking system of the HoloLens was augmented with image target tracking capabilities to stabilize virtual content, enhance accuracy, and account for spatial drift. This fusion of marker-based and marker-less tracking techniques provides a novel approach to display robust AR assembly instructions on a commodity AR HMD. Furthermore, utilizing this novel spatial registration approach, the position of real-world objects was accurately registered to properly occlude virtual work instructions. To render the desired effect, specialized computer graphics methods and custom shaders were developed and implemented for an AR assembly application. After developing novel methods to display work instructions on a commodity AR HMD, it was necessary to validate that these work instructions were being accurately delivered. Utilizing the sensors on the HoloLens, data was collected during the assembly process regarding head position, orientation, assembly step times, and an estimation of spatial drift. With the addition of wearable physiological sensor data, this data was fused together in a visualization application to validate instructions were properly delivered and provide an opportunity for an analysist to examine trends within an assembly session. Additionally, the spatial drift data was then analyzed to gain a better understanding of how spatial drift accumulates over time and ensure that the spatial registration mitigation techniques was effective. Academic research has shown that AR may substantial reduce cost for assembly operations through a reduction in errors, time, and cognitive workload. This research provides novel solutions to overcome the limitations of commodity AR HMDs and validate their use for product assembly. Furthermore, the research provided in this thesis demonstrates the potential of commodity AR HMDs and how their limitations can be mitigated for use in product assembly tasks

Towards the use of augmented reality techniques for assisted acceptance sampling

Acceptance sampling is a statistical procedure for accepting or rejecting production lots according to the result of a sample inspection. Formalizing the concept of assisted acceptance sampling, this article suggests the use of consolidated tools for reducing the risk of human errors in acceptance sampling activities. To this purpose, the application of augmented reality techniques may represent a profitable and sustainable solution. An augmented reality–based prototype system is described in detail and tested by an experimental plan. The major original contributions of this work are (a) introducing the new paradigm of assisted acceptance sampling and (b) developing a preliminary application in an industrial-like environment. This application is a first step towards the realization of a complete assisted acceptance sampling system