Mixed Reality Improves Education and Training in Assembly Processes

Mixed reality is the outcome of blending the physical world with the digital world, made possible by technological advancement. Mixed reality is the next evolution in human, computer, and environment interaction. Augmented reality (AR) uses a virtual model of the real world, augmented by using a computer to see the real environment through a special display device. Current education and training systems in the engineering maintenance field are still insufficiently directed at the psychomotor skills in learning about machine parts, which makes them less effective for trainees. The oil and gas industry always face problems related to inefficiency due to downtime of critical equipment. This study was conducted at designing and developing a virtual reality (VR) and augmented reality (AR) system as a learning and training platform. This work also reviewed AR applications for machine part maintenance and assembly. An AR system was modelled and developed using the following software: CATIA, Blender, Unity and Vuforia. The effectiveness of using the AR technique in an education and training process was evaluated with 20 respondents among university students. The results showed that using this AR app enhanced the participant's understanding according to certain criteria and can be adopted as a learning method

Authoring Software for Augmented Reality Applications for the Use of Maintenance and Training Process

Augmented Reality (AR) in the last decade has increased the popularity on various areas, such as education, advertising, maintenance, marketing and entertainment. On the area of maintenance and education specially we have been researching the benefits of the use of augmented reality bring us, and we have discover that the transfer of knowledge is faster than the traditional methods, and help to companies to train their employees faster and better. Many of the AR applications are custom made to the client needs, and to make an application of AR involves different types of skills such as programming, designing, modeling, animating, texturing. Given the high cost of these or the lack of some of these abilities, the need of programs of authoring has increase that permit to users create AR processes using just the GUI without having learn how to program. This papers describes the program developed ManAR an authoring program that permits the user to create an AR process for maintenance and training. The application permit to companies to create process assisted by augmented reality to train and use on the field. The application links tridimensional models to a mark, and make use of pictures, texts and videos, to enhance the experience, and finally visualize the final product on tablets. Also other benefit is to access relevant information such as times, errors of the employees to improve AR process or to know how the users are progressing with their training.Ramirez, H.; González Mendívil, E.; Ramirez Flores, P.; Contero, M. (2013). Authoring Software for Augmented Reality Applications for the Use of Maintenance and Training Process. Procedia Computer Science. 25:189-193. doi:10.1016/j.procs.2013.11.023S1891932

Comparative study of AR versus video tutorials for minor maintenance operations

[EN] Augmented Reality (AR) has become a mainstream technology in the development of solutions for repair and maintenance operations. Although most of the AR solutions are still limited to specific contexts in industry, some consumer electronics companies have started to offer pre-packaged AR solutions as alternative to video-based tutorials (VT) for minor maintenance operations. In this paper, we present a comparative study of the acquired knowledge and user perception achieved with AR and VT solutions in some maintenance tasks of IT equipment. The results indicate that both systems help users to acquire knowledge in various aspects of equipment maintenance. Although no statistically significant differences were found between AR and VT solutions, users scored higher on the AR version in all cases. Moreover, the users explicitly preferred the AR version when evaluating three different usability and satisfaction criteria. For the AR version, a strong and significant correlation was found between the satisfaction and the achieved knowledge. Since the AR solution achieved similar learning results with higher usability scores than the video-based tutorials, these results suggest that AR solutions are the most effective approach to substitute the typical paper-based instructions in consumer electronics.This work has been supported by Spanish MINECO and EU ERDF programs under grant RTI2018-098156-B-C55.Morillo, P.; García García, I.; Orduña, JM.; Fernández, M.; Juan, M. (2020). Comparative study of AR versus video tutorials for minor maintenance operations. Multimedia Tools and Applications. 79(11-12):7073-7100. https://doi.org/10.1007/s11042-019-08437-9S707371007911-12Ahn J, Williamson J, Gartrell M, Han R, Lv Q, Mishra S (2015) Supporting healthy grocery shopping via mobile augmented reality. ACM Trans Multimedia Comput Commun Appl 12(1s):16:1–16:24. https://doi.org/10.1145/2808207Anderson TW (2011) Anderson–darling tests of goodness-of-fit. Springer, Berlin, pp 52–54. https://doi.org/10.1007/978-3-642-04898-2_118Awad N, Lewandowski SE, Decker EW (2015) Event management system for facilitating user interactions at a venue. US Patent App. 14/829,382Azuma R (1997) A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6(4):355–385Baird K, Barfield W (1999) Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality 4:250–259Ballo P (2018) Hardware and software for ar/vr development. In: Augmented and virtual reality in libraries, pp 45–55. LITA guidesBarrile V, Fotia A, Bilotta G (2018) Geomatics and augmented reality experiments for the cultural heritage. Applied Geomatics. https://doi.org/10.1007/s12518-018-0231-5Billinghurst M, Duenser A (2012) Augmented reality in the classroom. Computer 45(7):56–63. https://doi.org/10.1109/MC.2012.111Bowman DA, McMahan RP (2007) Virtual reality: how much immersion is enough? Computer 40(7)Brown TA (2015) Confirmatory factor analysis for applied research. Guilford PublicationsDodge Y. (ed) (2008) Kruskal-Wallis test. Springer, New York. https://doi.org/10.1007/978-0-387-32833-1_216Elmunsyah H, Hidayat WN, Asfani K (2019) Interactive learning media innovation: utilization of augmented reality and pop-up book to improve user’s learning autonomy. J Phys Conf Ser 1193(012):031. https://doi.org/10.1088/1742-6596/1193/1/012031Entertainment L (2017) Dolphin Player. https://play.google.com/store/apps/details?id=com.broov.player. Online; Accessed 09-September-2017Fletcher J, Belanich J, Moses F, Fehr A, Moss J (2017) Effectiveness of augmented reality & augmented virtuality. In: MODSIM Modeling & simulation of systems and applications) world conferenceFraga-Lamas P, Fernández-Caramés TM, Blanco-Novoa O, Vilar-Montesinos MA (2018) A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access 6:13,358–13,375. https://doi.org/10.1109/ACCESS.2018.2808326Furió D, Juan MC, Seguí I, Vivó R (2015) Mobile learning vs. traditional classroom lessons: a comparative study. J Comput Assist Learn 31(3):189–201. https://doi.org/10.1111/jcal.12071Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ 23(6):778–798. https://doi.org/10.1080/10494820.2013.815221Gimeno J, Morillo P, Orduña JM, Fernández M (2013) A new ar authoring tool using depth maps for industrial procedures. Comput Ind 64(9):1263–1271. https://doi.org/10.1016/j.compind.2013.06.012Holzinger A, Kickmeier-Rust MD, Albert D (2008) Dynamic media in computer science education; content complexity and learning performance: is less more? Educational Technology & Society 11(1):279–290Hornbaek K (2013) Some whys and hows of experiments in human–computer interaction. Foundations and TrendsⓇ in Human–Computer Interaction 5(4):299–373. https://doi.org/10.1561/1100000043Huang J, Liu S, Xing J, Mei T, Yan S (2014) Circle & search: Attribute-aware shoe retrieval. ACM Trans Multimedia Comput Commun Appl 11 (1):3:1–3:21. https://doi.org/10.1145/2632165Jiang S, Wu Y, Fu Y (2018) Deep bidirectional cross-triplet embedding for online clothing shopping. ACM Trans Multimedia Comput Commun Appl 14(1):5:1–5:22. https://doi.org/10.1145/3152114Kim SK, Kang SJ, Choi YJ, Choi MH, Hong M (2017) Augmented-reality survey: from concept to application. KSII Transactions on Internet and Information Systems 11:982–1004. https://doi.org/10.3837/tiis.2017.02.019Langlotz T, Zingerle M, Grasset R, Kaufmann H, Reitmayr G (2012) Ar record&replay: Situated compositing of video content in mobile augmented reality. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI ’12. ACM, New York, pp 318–326, DOI https://doi.org/10.1145/2414536.2414588, (to appear in print)Martin-SanJose JF, Juan MC, Mollá R, Vivó R (2017) Advanced displays and natural user interfaces to support learning. Interact Learn Environ 25(1):17–34. https://doi.org/10.1080/10494820.2015.1090455Massey FJ (1951) The kolmogorov-Smirnov test for goodness of fit. J Am Stat Assoc 46(253):68–78van der Meij H, van der Meij J, Voerman T, Duipmans E (2018) Supporting motivation, task performance and retention in video tutorials for software training. Educ Technol Res Dev 66(3):597–614. https://doi.org/10.1007/s11423-017-9560-zvan der Meij J, van der Meij H (2015) A test of the design of a video tutorial for software training. J Comput Assist Learn 31 (2):116–132. https://doi.org/10.1111/jcal.12082Mestre LS (2012) Student preference for tutorial design: a usability study. Ref Serv Rev 40(2):258–276. https://doi.org/10.1108/00907321211228318Mohr P, Kerbl B, Donoser M, Schmalstieg D, Kalkofen D (2015) Retargeting technical documentation to augmented reality. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ’15. ACM, New York, pp 3337–3346, DOI https://doi.org/10.1145/2702123.2702490, (to appear in print)Mohr P, Mandl D, Tatzgern M, Veas E, Schmalstieg D, Kalkofen D (2017) Retargeting video tutorials showing tools with surface contact to augmented reality. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ’17. ACM, New York, pp 6547–6558, DOI https://doi.org/10.1145/3025453.3025688, (to appear in print)Montgomery DC, Runger GC (2003) Applied statistics and probability for engineers. Wiley, New YorkMorillo P, Orduña JM, Casas S, Fernández M (2019) A comparison study of ar applications versus pseudo-holographic systems as virtual exhibitors for luxury watch retail stores. Multimedia Systems. https://doi.org/10.1007/s00530-019-00606-yMorse JM (2000) Determining sample size. Qual Health Res 10(1):3–5. https://doi.org/10.1177/104973200129118183Muñoz-Montoya F, Juan M, Mendez-Lopez M, Fidalgo C (2019) Augmented reality based on slam to assess spatial short-term memory. IEEE Access 7:2453–2466. https://doi.org/10.1109/ACCESS.2018.2886627Neuhäuser M (2011) Wilcoxon–Mann–Whitney test. Springer, Berlin, pp 1656–1658Neumann U, Majoros A (1998) Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. In: Inproceedings of the IEEE virtual reality annual international symposium (VR ’98), pp 4–11no JJA, Juan MC, Gil-Gómez JA, Mollá R. (2014) A comparative study using an autostereoscopic display with augmented and virtual reality. Behaviour & Information Technology 33(6):646–655. https://doi.org/10.1080/0144929X.2013.815277Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215–228Quint F, Loch F (2015) Using smart glasses to document maintenance processes. Mensch und Computer 2015–WorkshopbandRadkowski R, Herrema J, Oliver J (2015) Augmented reality-based manual assembly support with visual features for different degrees of difficulty. International Journal of Human–Computer Interaction 31(5):337–349. https://doi.org/10.1080/10447318.2014.994194Regenbrecht H, Schubert T (2002) Measuring presence in augmented reality environments: design and a first test of a questionnaire, Porto, PortugalRobertson J (2012) Likert-type scales, statistical methods, and effect sizes. Commun ACM 55(5):6–7. https://doi.org/10.1145/2160718.2160721Rodríguez-Andrés D, Juan MC, Méndez-López M, Pérez-Hernández E, Lluch J (2016) Mnemocity task: Assessment of childrens spatial memory using stereoscopy and virtual environments. PLos ONE 1(8). https://doi.org/10.1371/journal.pone.0161858Sanna A, Manuri F, Lamberti F, Paravati G, Pezzolla P (2015) Using handheld devices to support augmented reality-based maintenance and assembly tasks. In: 2015 IEEE International conference on consumer electronics (ICCE), pp. 178–179. https://doi.org/10.1109/ICCE.2015.7066370Schmidt S, Ehrenbrink P, Weiss B, Voigt-Antons J, Kojic T, Johnston A, Moller S (2018) Impact of virtual environments on motivation and engagement during exergames. In: 2018 Tenth international conference on quality of multimedia experience (qoMEX), pp 1–6. https://doi.org/10.1109/QoMEX.2018.8463389Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’03. ACM, New York, pp 73–80, DOI https://doi.org/10.1145/642611.642626, (to appear in print)Tomás JM, Oliver A, Galiana L, Sancho P, Lila M (2013) Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling: A Multidisciplinary Journal 20(2):299–313. https://doi.org/10.1080/10705511.2013.769394Uva AE, Gattullo M, Manghisi VM, Spagnulo D, Cascella GL, Fiorentino M (2017) Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. The Int J Adv Manuf Technol: 1–13Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Advances in Manufacturing 4(1):1–22. https://doi.org/10.1007/s40436-015-0131-4Westerfield G, Mitrovic A, Billinghurst M (2015) Intelligent augmented reality training for motherboard assembly. Int J Artif Intell Educ 25(1):157–172. https://doi.org/10.1007/s40593-014-0032-xWiedenmaier S, Oehme O, Schmidt L, Luczak H (2003) Augmented reality (ar) for assembly processes - design and experimental evaluation. International Journal of Human-Computer Interaction 16(3):497–514Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence: Teleoperators and Virtual Environments 7(3):225–240Wu HK, Lee SWY, Chang HY, Liang JC (2013) Current status, opportunities and challenges of augmented reality in education. Computers & Education 62:41–49. https://doi.org/10.1016/j.compedu.2012.10.024Yim MYC, Chu SC, Sauer PL (2017) Is augmented reality technology an effective tool for e-commerce? an interactivity and vividness perspective. Journal of Interactive Marketing 39(http://www.sciencedirect.com/science/article/pii/S1094996817300336):89–103. https://doi.org/10.1016/j.intmar.2017.04.001Yuan ML, Ong SK, Nee AYC (2008) Augmented reality for assembly guidance using a virtual interactive tool. Int J Prod Res 46(7):1745–1767. https://doi.org/10.1080/0020754060097293

Research Avenues on use of Augmented Reality in Education

The use of Innovative technology in education enhances the grasping ability of the student to gain knowledge proactively and provides a platform for a constructive process of learning and understanding. Augmented Reality (AR) plays an essential role in active learning and critical thinking in the current information age because technology enables students to interact with the virtual world with an immersive experience. Moreover, the integration of AR in education has attracted researcher’s attention towards AR due to its immersive, naturalistic experience. Augmented reality plays a vital role in Medical Science, the Aviation industry, the Advertising industry, the Printing Industry, Maintenance, Tourism, Education, the Automobile industry and many more upcoming industries. The use of AR is going to be spread in the coming days. This paper comprises an overview and the study of augmented reality in different sectors. On emphasising the uses of AR in the education field, to give a real-life interactive experience to the user on his mobile. The review narrates the ability of AR, and applications of AR in the field of education such as science education, Industrial training, and biomedical education. The review summarises the potential of technology integration

Applied virtual reality at the Research Triangle Institute

Virtual Reality (VR) is a way for humans to use computers in visualizing, manipulating and interacting with large geometric data bases. This paper describes a VR infrastructure and its application to marketing, modeling, architectural walk through, and training problems. VR integration techniques used in these applications are based on a uniform approach which promotes portability and reusability of developed modules. For each problem, a 3D object data base is created using data captured by hand or electronically. The object's realism is enhanced through either procedural or photo textures. The virtual environment is created and populated with the data base using software tools which also support interactions with and immersivity in the environment. These capabilities are augmented by other sensory channels such as voice recognition, 3D sound, and tracking. Four applications are presented: a virtual furniture showroom, virtual reality models of the North Carolina Global TransPark, a walk through the Dresden Fraunenkirche, and the maintenance training simulator for the National Guard

Mixed Reality Technology Applications in Construction Equipment Operator Training

Abstract: This paper provides information on Mixed Reality (MR), and more specifically Augmented Reality (AR) and their potential applications in heavy construction equipment and operator training. Mixed Reality involves the use of special display and tracking technologies that are capable of seamlessly merging digital (virtual) content into a real environment. Conceptual designs and application scenarios of two AR systems for construction are presented: an AR-based equipment management system (AR EMS) for maintenance, and repair of heavy equipment fleet, and an AR-based operator training system (AR OTS) that trains the novice operators in a real worksite environment populated with virtual materials and instructions

An Investigation of Acceptance and E-Readiness for the Application of Virtual Reality and Augmented Reality Technologies to Maintenance Training in the Manufacturing Industry

Virtual Reality (VR) and Augmented Reality (AR) technologies offer new ways of providing training in manufacturing maintenance. The adoption of modern maintenance training practices has the potential to create efficiencies in terms of cost and time to train, while enhancing the quality of learning and maintenance outputs. However, in order to utilise the potential improvements that VR and AR offer in a manufacturing maintenance context, it is first important to understand the specific factors associated with VR and AR readiness and user requirement. The paper will firstly describe the results from a number of interviews conducted within a range of manufacturing companies in the North East of England to establish the state of e-technology readiness and acceptance, with specific emphasis on VR and AR applications. The results will identify how VR and AR might be utilised, relative to the company’s needs. Secondly, a new ‘model’ for maintenance training utilising VR/AR technologies will be described, based upon the initial findings and analyses combining cognitive behavioural models, real world data, and learning theory

SCALABLE AR FOR BIM ON TELECOMMUNICATION NETWORK SITES

A growing number of research works, experiments and applications is investigating the potential at the intersection of augmented reality (AR) and the architecture, engineering, and construction (AEC) industry. Project management, project communication, collaborative design, maintenance and construction progress documentation, construction site safety, and training are some of the cases that can benefit from blending real and virtual views through mobile devices. In recent years, research also highlighted how mixed reality and building information modeling (BIM) could cooperate to provide effective communication between multiple agents and closer interaction between digital information and the building site. Nevertheless, consolidated applications in these fields are still limited, especially when compared to other areas of AR adoption. This paper presents the development of an AR-based mobile app for exploring telecommunications tower sites and interacting with a related BIM database. The project aims to provide easy-to-use tools to maintain both the physical assets and an up-to-date model. We discuss critical issues in developing a scalable and interoperable application, supporting the feasibility study of similar solutions in the AEC sector

Interaction Design of Augmented Education Environments - Augmented and Mixed Reality for performance and training support of Aviation / Automotive Technicians.

"Augmented reality (AR),Mixed Reality (MR) and their mix and combination with other disruptive technologies offer an enormous potential for supporting instructors and trainees in modern education and working environments such as of aircraft maintenance technicians or automotive service technicians. In this paper we investigate and show some examples on how the performance and training of such instructors and trainees can be actively supported. Furthermore we will discuss the new challenges for training designers. The augmentation of the physical world with interactive, context-aware information (e.g. 2D and 3D content) provides multifaceted possibilities, on various ubiquitous and pervasive computing environments. While there is still the broad opinion that these concepts are just situated in the world of science fiction (SciFi) and SciFi movies, we will relate these techniques to existing technologies and prototypes in research. Terms like outernet, print + or 2.0, augmented goggles, wearable technology are not just remaining pure buzzwords anymore. We will demonstrate how different prototypes applying low cost rapid prototyping methods can be applied as powerful performance assistance and training support instruments, whereby discussing the requirements and user-needs analysis phases as well as human–computer interaction and interaction design issues, user modelling, usability engineering, prototyping and evaluation issues. Different scenarios are possible and provide the basis to generate storyboards. One of the key factors is hereby to analyse typical tasks and activities of users and utilize familiar user interaction paradigms for accessing information, such as using a book or assisting the work with task sheets. For example by augmenting the material that is printed in the book with additional graphical 3D interactive information which can be viewed and manipulated by the instructor and/or trainee, one can provide a link between traditional learning and technology-enhanced learning. Basing on theoretical and empirical research it is possible then to design via first moodboards and scribbles relevant prototypes. A qualitative and quantitative analysis can be used to define a basic design process for such new environments and settings. Moreover, MR and AR along with Mobile Tagging (MT) combined with Pervasive Computing provide the possibility to realize a Physical World Connection (PWC) between Reality and Virtuality. In the field of aviation and automotive industry, this offers manifold possibilities for maintenance and service personnel to get access to assistive technologies in a very intuitive way to enhance their operation, work, training, and knowledge. Assistance for the large variety of job tasks can be provided to a certain extent by offering augmentation of the different senses like vision and audition, providing a media-rich interface. Although the roots of Mixed Reality and Augmented Reality are based on prototype applications in the aircraft industry in the early 1990s, the impact of these emerging technologies on special target groups has not yet been investigated and validated by many research groups. With a specific focus on these user communities, applications are considerably more influenced by both usefulness and usability of technology. Consequently, it is argued that key issues in developing such applications are the tracking methodology, the display technology, interaction (devices and framework) and most of all ensuring good usability. In this paper, a concrete example in a aviation and automotive environment will be presented as a case study for investigating and validating these key issues. Preliminary results of semi-structured interviews and observations in real training and work settings indicate a lack of information concerning existence of such technologies and environments, but show big interest and potential for such educational and workplace innovations, while concrete visions or user requirements for future augmented education environments remain open and are subject of our further research steps