Utilizing Virtual and Augmented Reality to Augment Real World Operational Training to Improve Proficiency

NPS NRP Executive SummaryRecent years have been marked by the emergence of affordable off-the-shelf solutions in Virtual Reality (VR) and Augmented Reality (AR) technology. Those advances created a unique opportunity for exercising transformational approaches to personalized, just-in-time relevant training of large numbers of people. The research studies suggest that the systemic application of novel technical solutions in training will have a major positive impact on operational readiness regardless of community size or Department of Defense service demographic. These findings are especially relevant during times of financial constraints, or environmental limitations such as pandemics. We propose a series of research activities to identify best in class contemporary VR/AR training solutions used by the U.S. Navy and map those systems' capabilities to the U.S. Coast Guard's (USCG) needs. Although the research questions addressed in this effort are focused on current U.S. Navy (USN) training solutions and practices, there is sufficient evidence to support the transferability of training effectiveness across services, resulting in a tenfold return on taxpayer investment. Further, this VR/AR research compliments congressionally mandated interoperability requirements for USN and USCG, while simultaneously addressing USCG training gaps/shortfalls. The research will include a review of current literature dedicated to training effectiveness of VR and AR systems, discussions with stakeholders, and data collection that provide understanding about the USCG's training requirements. Our investigation will emphasize the benefits that each system offers, include advantages, disadvantages, potential obstacles to large-scale adoption, and highlight the best use cases.Office of Naval Research (ONR)ASN(RDA) - Research, Development, and AcquisitionThis research is supported by funding from the Naval Postgraduate School, Naval Research Program (PE 0605853N/2098). https://nps.edu/nrpChief of Naval Operations (CNO)Approved for public release. Distribution is unlimited.

X-RAY VISION: APPLICATION OF AUGMENTED REALITY IN AVIATION MAINTENANCE TO SIMPLIFY TASKS INHIBITED BY OCCLUSION

This thesis examined the potential applications of augmented or mixed reality (AR/MR) technology and leveraging them in the context of the aviation maintenance community. Specifically, we examined whether using the 3D mapping and real-time space tracking technology of devices like the Microsoft HoloLens 2 can be used to make maintenance tasks easier in environments where the maintainer is not able to see into their workspace. With the complexities of aircraft construction, the prevalence of narrow, tight fitting spaces that are blocked by walls or obstructions is common. In the past, aviation maintainers have had to rely on memorizing 2D diagrams and feeling around dark, cramped spaces in order to determine where certain parts are located. Previous research in the field of AR primarily focuses on comparing AR methods to traditional methods for different types of tasks in simulacra. There is a lack of research in the specific application of AR that addresses occlusion introduced into these tasks. By conducting trials of simulated maintenance in an occluded area using AR technology, we found that the novice maintainer increased the accuracy of performance and decreased maintenance time when compared to traditional methods, while providing a subjectively easier method for instruction.Captain, United States Marine CorpsApproved for public release. Distribution is unlimited

The Impacts of Using Augmented Reality to Support Aircraft Maintenance

The United States Air Force (USAF) expends significant resources to address the rise in aviation mishaps derived from an overworked, understaffed maintenance community, and high operational environment. Currently, paper-based technical orders (T.O.) are utilized by maintainers to accomplish aircraft inspections, servicing, and maintenance tasks. As technology advances, many civilian agencies have begun to leverage augmented reality (AR) to improve organizational proficiency. This research seeks to identify if the inclusion of AR within aircraft maintenance will positively or negatively affect maintenance task accuracy and completion time. A single variable randomized complete block design (RCBD), within-subject design of experiment (DOE) asses the differences between a treatment group (AR-enabled T.O.) contrary to the control group (paper-based T.O.). Results conclude AR-enabled T.O.s designed from the AF perspective will reduce simple task errors, but will not impact total task completion time. Differentiation from prior findings, application specificity, will impact AR effectiveness and utilization within the organization employed. Additionally, experimental research revealed the need to address current AF infrastructure barriers before implementation of the technology within the organization

Augmented Reality Future Step Visualization for Robust Surgical Telementoring

Introduction Surgical telementoring connects expert mentors with trainees performing urgent care in austere environments. However, such environments impose unreliable network quality, with significant latency and low bandwidth. We have developed an augmented reality telementoring system that includes future step visualization of the medical procedure. Pregenerated video instructions of the procedure are dynamically overlaid onto the trainee's view of the operating field when the network connection with a mentor is unreliable. Methods Our future step visualization uses a tablet suspended above the patient's body, through which the trainee views the operating field. Before trainee use, an expert records a “future library” of step-by-step video footage of the operation. Videos are displayed to the trainee as semitransparent graphical overlays. We conducted a study where participants completed a cricothyroidotomy under telementored guidance. Participants used one of two telementoring conditions: conventional telestrator or our system with future step visualization. During the operation, the connection between trainee and mentor was bandwidth throttled. Recorded metrics were idle time ratio, recall error, and task performance. Results Participants in the future step visualization condition had 48% smaller idle time ratio (14.5% vs. 27.9%, P < 0.001), 26% less recall error (119 vs. 161, P = 0.042), and 10% higher task performance scores (rater 1 = 90.83 vs. 81.88, P = 0.008; rater 2 = 88.54 vs. 79.17, P = 0.042) than participants in the telestrator condition. Conclusions Future step visualization in surgical telementoring is an important fallback mechanism when trainee/mentor network connection is poor, and it is a key step towards semiautonomous and then completely mentor-free medical assistance systems

Virginia Digital Shipbuilding Program (VDSP): Building an Agile Modern Workforce to Improve Performance in the Shipbuilding and Ship Repair Industry

Industry 4.0 is the latest stage in the Industrial Revolution and is reflected in the digital transformation and use of emergent technologies including the Internet of Things, Big Data, Robotic automation of processes, 3D printing and additive manufacturing, drones and Artificial Intelligence (AI) in the manufacturing industry. The implementation of these technologies in the Shipbuilding and Ship Repair Industry is currently in a nascent stage. Considering this, there is huge potential to increase cost savings, decrease production timelines, and drive down inefficiencies in Lifecyle management of ships. However, the implementation of these Industry 4.0 technologies is hindered by a noticeable gap in workforce capability and capacity. The shipbuilding and ship repair industry is projected to lose approximately 33% of skilled workforce and 48% of management by 2028. With an aging workforce and an incoming digital generation that excels in tech savviness, flexibility, global thinking, and multi-tasking it is crucial to be innovative in workforce development. The Virginia Digital Shipbuilding Program responds to this need by providing a process and platform to address education, training, adoption of innovative new technology and the ability to provide real-time solutions to current and future industry problems. This paper will focus on the three pillars of Digital Shipbuilding – Career Pathway Mapping and Curriculum Development, Outreach and Workforce Development, and Research and Development. Additionally, this paper will address how the team is ensuring that stackable, transferable education and certification processes are implemented between military and industry to facilitate the transition of veterans to the civilian workforce

Chapter 07. Message Design for Healthcare Simulation

Simulation is a teaching method that motivates and engages learners (Ifenthaler et al. 2020; Martin & Betrus, 2019), can provide better student performance outcomes compared to traditional classroom education (Chernikova et al. 2020; D’Angleo et al., 2014; Di Natale et al., 2020; Gralnick & Levy, 2011; Pellas et al., 2019), and can even replace some real-world learning experiences (Alexander et al., 2015). Instructional designers who develop healthcare simulations should use relevant learning theory and instructional message design to ensure that the simulation is learner-centered and based in evidence-based practice to maximize the potential for the learning experience

The Role of the Human Operator in the Third Offset Strategy

New gadgets are useful, but our personnel are the most critical component to continued operational success. The Third Offset Strategy provides a near-unique opportunity to leap forward in optimizing human performance. We must establish priorities, tangible deliverables, and a timeline

AN ANALYSIS OF THE MARINE CORPS AVIATION TRAINING SYSTEM REQUIREMENTS LIFE CYCLE

This capstone applied project examines the suitability of the current requirements life cycle for Marine Corps aviation training systems, including new programs and upgrades. Methodology includes a comprehensive review of existing policies and processes as well as interviews with key stakeholders. Analysis has identified weaknesses in the areas of training-focused requirements generation as well as portfolio management across Marine Corps training system programs. Recommendations include integrating modeling and simulation (M&S) expertise into the Training Management Process (TMP) and full implementation of Training Systems Certification (TSC) and Systematic Team Assessment of Readiness Training (START) tools to improve requirement relevancy to training needs as well as improved portfolio management for Marine Corps training systems led by Marine Corps Training and Education Command (TECOM). This capstone applied project concludes with recommendations for further study related to these matters.Major, United States Marine CorpsMajor, United States Marine CorpsApproved for public release. Distribution is unlimited