Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks

Blattgerste J, Strenge B, Renner P, Pfeiffer T, Essig K. Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments. PETRA '17. New York, NY, USA: ACM; 2017: 75-82.Augmented Reality (AR) gains increased attention as a means to provide assistance for different human activities. Hereby the suitability of AR does not only depend on the respective task, but also to a high degree on the respective device. In a standardized assembly task, we tested AR-based in-situ assistance against conventional pictorial instructions using a smartphone, Microsoft HoloLens and Epson Moverio BT-200 smart glasses as well as paper-based instructions. Participants solved the task fastest using the paper instructions, but made less errors with AR assistance on the Microsoft HoloLens smart glasses than with any other system. Methodically we propose operational definitions of time segments and other optimizations for standardized benchmarking of AR assembly instructions

An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task

Many studies have confirmed the benefits of using Augmented Reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware for complex assembly tasks. For this research, previously published data using Desktop Model Based Instructions (MBI), Tablet MBI, and Tablet AR instructions were compared to new assembly data collected using AR instructions on the Microsoft HoloLens Head Mounted Display (HMD). Participants completed a mock wing assembly task, and measures like completion time, error count, Net Promoter Score, and qualitative feedback were recorded. The HoloLens condition yielded faster completion times than all other conditions. HoloLens users also had lower error rates than those who used the non-AR conditions. Despite the performance benefits of the HoloLens AR instructions, users of this condition reported lower net promoter scores than users of the Tablet AR instructions. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the user feedback favored the Tablet AR condition, the HoloLens condition resulted in significantly faster assembly times. As a result, it is recommended to use the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders better understand the benefits of different AR technology for manual assembly tasks

Interaktiiviset kokoonpano-ohjeet

Industrial products are increasingly varying, and the assembly of customized or unique products is slow, expensive, and prone to errors. Conventional static assembly drawings and instructions are suboptimal in supporting complex and dynamic assembly operations. The main objective of the study was to investigate if interactive assembly instructions could substitute the current documents instructing assembly in the case company. Two approaches, 3D instructions and augmented reality (AR) instructions, were developed based on literature review. 3D instructions presented the assembly procedure in steps in which the assembly of the parts is animated. The instructions were based directly on the 3D model of the assembly object. AR instructions utilized the same assembly sequence as 3D instructions. AR instructions were viewed using a head-mounted display, which presented the assembly step animations spatially overlaid on the physical assembly. The developed instructions were evaluated in a user study. The tests were observed by the author, and the participants answered to a post-study questionnaire that concerned subjective efficiency and user acceptance. Both AR instructions and 3D instructions received positive feedback and were evaluated more efficient than the currently used assembly drawings. The features of the interactive assembly drawings address directly the problems of the current assembly documents. Hence, it was concluded that interactive assembly instructions could be used instead of the current assembly drawings and work instructions. However, the complexity of the case company products require that the instructions must be configurable to enable their implementation.Teolliset tuotteet kehittyvät jatkuvasti monipuolisemmin muunneltaviksi, ja samalla niiden kokoonpano muuttuu hankalammaksi ja kalliimmaksi. Perinteiset kuviin ja tekstiin perustuvat kokoonpanokuvat ja työohjeet ovat monin tavoin riittämättömiä ohjeistamaan monimutkaisia ja dynaamisia kokoonpanotehtäviä. Tässä työssä tavoitteena oli tutkia, voisiko interaktiivisilla kokoonpano-ohjeilla korvata kohdeyrityksessä nykyisin käytössä olevat työohjeet ja kokoonpanokuvat. Työssä kehitettiin aikaisempien tutkimusten pohjalta kaksi erilaista interaktiivista ohjeistustapaa. 3D-ohjeet opastavat kokoonpanoa vaihe vaiheelta näyttäen jokaisen osan asennuksen animoidusti. 3D-ohjeet luodaan suoraan kokoonpanon 3D-mallin pohjalta. Toiseksi menetelmäksi valikoitui lisättyä todellisuutta (augmented reality, AR) hyödyntävät ohjeet. AR-ohjeet perustuvat 3D-ohjeita varten luotuihin vaiheistuksiin sekä animaatioihin. AR-ohjeita katsotaan silmikkonäytöllä, joka näyttää ohjeiden virtuaaliset komponentit todellisen kokoonpanon päällä. Ohjeiden toimivuutta testattiin käyttäjäkokeissa. Testeissä havainnoitiin koehenkilöiden toimintaa, ja lisäksi he vastasivat kyselyyn. Kyselyllä selvitettiin, miten tehokkaana koehenkilöt pitivät testattuja ohjeita verrattuna heidän tavallisesti käyttämiin kokoonpanokuviin. Sekä AR- että 3D-ohjeet saivat positiivista palautetta, ja koehenkilöt kokivat niiden toimivan tavallisia kokoonpanokuvia paremmin. Interaktiiviset ohjeet ja niiden tärkeimmät ominaisuudet vastaavat nykyisten kokoonpanokuvien ja työohjeiden ongelmakohtiin. Työn johtopäätöksenä voidaankin todeta, että interaktiiviset kokoonpano-ohjeet sopisivat korvaamaan nykyiset kokoonpanokuvat sekä työohjeet. Tuotteiden monimutkaisuus kuitenkin edellyttää, että ohjeet pitää pystyä konfiguroimaan varianttikohtaisesti

Smart operators: How augmented and virtual technologies are affecting the worker's performance in manufacturing contexts

Purpose: The correct interaction between the workforce and augmented, virtual, and mixed reality technologies represents a crucial aspect of the success of the smart factory. This interaction is, indeed, affected by the variability of human behavior and its reliability, which can strongly influence the quality, safety, and productivity standards. For this reason, this paper aims to provide a clear and complete analysis of the impacts of these technologies on the performance of operators. Design/methodology/approach: A Systematic Literature Review (SLR) was conducted to identify peer-reviewed papers that focused on the implementation of augmented and virtual technologies in manufacturing systems and their effects on human performance. Findings: In total, 61 papers were selected and thoroughly analyzed. The findings of this study reveal that Augmented, Virtual and Mixed Reality can be applied for several applications in manufacturing systems with different types of devices, that involve various advantages and disadvantages. The worker’s performance that are influencing by the use of these technologies are above all time to complete a task, error rate and mental and physical workload. Originality/value: Over the years Augmented, Virtual and Mixed Reality technologies in manufacturing systems have been investigated by researchers. Several studies mostly focused on technological issues, have been conducted. The role of the operator, whose tasks may be influenced positively or negatively by the use of new devices, has been hardly ever analyzed and a deep analysis of human performance affected by these technologies is missing. This study represents a preliminary analysis to fill this gap. The results obtained from the SLR allowed us to develop a conceptual framework that investigates the current state-of-the-art knowledge about the topic and highlights gaps in the current researchPeer Reviewe

The Analysis of design and manufacturing tasks using haptic and immersive VR - Some case studies

The use of virtual reality in interactive design and manufacture has been researched extensively but the practical application of this technology in industry is still very much in its infancy. This is surprising as one would have expected that, after some 30 years of research commercial applications of interactive design or manufacturing planning and analysis would be widespread throughout the product design domain. One of the major but less well known advantages of VR technology is that logging the user gives a great deal of rich data which can be used to automatically generate designs or manufacturing instructions, analyse design and manufacturing tasks, map engineering processes and, tentatively, acquire expert knowledge. The authors feel that the benefits of VR in these areas have not been fully disseminated to the wider industrial community and - with the advent of cheaper PC-based VR solutions - perhaps a wider appreciation of the capabilities of this type of technology may encourage companies to adopt VR solutions for some of their product design processes. With this in mind, this paper will describe in detail applications of haptics in assembly demonstrating how user task logging can lead to the analysis of design and manufacturing tasks at a level of detail not previously possible as well as giving usable engineering outputs. The haptic 3D VR study involves the use of a Phantom and 3D system to analyse and compare this technology against real-world user performance. This work demonstrates that the detailed logging of tasks in a virtual environment gives considerable potential for understanding how virtual tasks can be mapped onto their real world equivalent as well as showing how haptic process plans can be generated in a similar manner to the conduit design and assembly planning HMD VR tool reported in PART A. The paper concludes with a view as to how the authors feel that the use of VR systems in product design and manufacturing should evolve in order to enable the industrial adoption of this technology in the future

Comparative study of AR versus video tutorials for minor maintenance operations

[EN] Augmented Reality (AR) has become a mainstream technology in the development of solutions for repair and maintenance operations. Although most of the AR solutions are still limited to specific contexts in industry, some consumer electronics companies have started to offer pre-packaged AR solutions as alternative to video-based tutorials (VT) for minor maintenance operations. In this paper, we present a comparative study of the acquired knowledge and user perception achieved with AR and VT solutions in some maintenance tasks of IT equipment. The results indicate that both systems help users to acquire knowledge in various aspects of equipment maintenance. Although no statistically significant differences were found between AR and VT solutions, users scored higher on the AR version in all cases. Moreover, the users explicitly preferred the AR version when evaluating three different usability and satisfaction criteria. For the AR version, a strong and significant correlation was found between the satisfaction and the achieved knowledge. Since the AR solution achieved similar learning results with higher usability scores than the video-based tutorials, these results suggest that AR solutions are the most effective approach to substitute the typical paper-based instructions in consumer electronics.This work has been supported by Spanish MINECO and EU ERDF programs under grant RTI2018-098156-B-C55.Morillo, P.; García García, I.; Orduña, JM.; Fernández, M.; Juan, M. (2020). Comparative study of AR versus video tutorials for minor maintenance operations. Multimedia Tools and Applications. 79(11-12):7073-7100. https://doi.org/10.1007/s11042-019-08437-9S707371007911-12Ahn J, Williamson J, Gartrell M, Han R, Lv Q, Mishra S (2015) Supporting healthy grocery shopping via mobile augmented reality. ACM Trans Multimedia Comput Commun Appl 12(1s):16:1–16:24. https://doi.org/10.1145/2808207Anderson TW (2011) Anderson–darling tests of goodness-of-fit. Springer, Berlin, pp 52–54. https://doi.org/10.1007/978-3-642-04898-2_118Awad N, Lewandowski SE, Decker EW (2015) Event management system for facilitating user interactions at a venue. US Patent App. 14/829,382Azuma R (1997) A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6(4):355–385Baird K, Barfield W (1999) Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality 4:250–259Ballo P (2018) Hardware and software for ar/vr development. In: Augmented and virtual reality in libraries, pp 45–55. LITA guidesBarrile V, Fotia A, Bilotta G (2018) Geomatics and augmented reality experiments for the cultural heritage. Applied Geomatics. https://doi.org/10.1007/s12518-018-0231-5Billinghurst M, Duenser A (2012) Augmented reality in the classroom. Computer 45(7):56–63. https://doi.org/10.1109/MC.2012.111Bowman DA, McMahan RP (2007) Virtual reality: how much immersion is enough? Computer 40(7)Brown TA (2015) Confirmatory factor analysis for applied research. Guilford PublicationsDodge Y. (ed) (2008) Kruskal-Wallis test. Springer, New York. https://doi.org/10.1007/978-0-387-32833-1_216Elmunsyah H, Hidayat WN, Asfani K (2019) Interactive learning media innovation: utilization of augmented reality and pop-up book to improve user’s learning autonomy. J Phys Conf Ser 1193(012):031. https://doi.org/10.1088/1742-6596/1193/1/012031Entertainment L (2017) Dolphin Player. https://play.google.com/store/apps/details?id=com.broov.player. Online; Accessed 09-September-2017Fletcher J, Belanich J, Moses F, Fehr A, Moss J (2017) Effectiveness of augmented reality & augmented virtuality. In: MODSIM Modeling & simulation of systems and applications) world conferenceFraga-Lamas P, Fernández-Caramés TM, Blanco-Novoa O, Vilar-Montesinos MA (2018) A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access 6:13,358–13,375. https://doi.org/10.1109/ACCESS.2018.2808326Furió D, Juan MC, Seguí I, Vivó R (2015) Mobile learning vs. traditional classroom lessons: a comparative study. J Comput Assist Learn 31(3):189–201. https://doi.org/10.1111/jcal.12071Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ 23(6):778–798. https://doi.org/10.1080/10494820.2013.815221Gimeno J, Morillo P, Orduña JM, Fernández M (2013) A new ar authoring tool using depth maps for industrial procedures. Comput Ind 64(9):1263–1271. https://doi.org/10.1016/j.compind.2013.06.012Holzinger A, Kickmeier-Rust MD, Albert D (2008) Dynamic media in computer science education; content complexity and learning performance: is less more? Educational Technology & Society 11(1):279–290Hornbaek K (2013) Some whys and hows of experiments in human–computer interaction. Foundations and TrendsⓇ in Human–Computer Interaction 5(4):299–373. https://doi.org/10.1561/1100000043Huang J, Liu S, Xing J, Mei T, Yan S (2014) Circle & search: Attribute-aware shoe retrieval. ACM Trans Multimedia Comput Commun Appl 11 (1):3:1–3:21. https://doi.org/10.1145/2632165Jiang S, Wu Y, Fu Y (2018) Deep bidirectional cross-triplet embedding for online clothing shopping. ACM Trans Multimedia Comput Commun Appl 14(1):5:1–5:22. https://doi.org/10.1145/3152114Kim SK, Kang SJ, Choi YJ, Choi MH, Hong M (2017) Augmented-reality survey: from concept to application. KSII Transactions on Internet and Information Systems 11:982–1004. https://doi.org/10.3837/tiis.2017.02.019Langlotz T, Zingerle M, Grasset R, Kaufmann H, Reitmayr G (2012) Ar record&replay: Situated compositing of video content in mobile augmented reality. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI ’12. ACM, New York, pp 318–326, DOI https://doi.org/10.1145/2414536.2414588, (to appear in print)Martin-SanJose JF, Juan MC, Mollá R, Vivó R (2017) Advanced displays and natural user interfaces to support learning. Interact Learn Environ 25(1):17–34. https://doi.org/10.1080/10494820.2015.1090455Massey FJ (1951) The kolmogorov-Smirnov test for goodness of fit. J Am Stat Assoc 46(253):68–78van der Meij H, van der Meij J, Voerman T, Duipmans E (2018) Supporting motivation, task performance and retention in video tutorials for software training. Educ Technol Res Dev 66(3):597–614. https://doi.org/10.1007/s11423-017-9560-zvan der Meij J, van der Meij H (2015) A test of the design of a video tutorial for software training. J Comput Assist Learn 31 (2):116–132. https://doi.org/10.1111/jcal.12082Mestre LS (2012) Student preference for tutorial design: a usability study. Ref Serv Rev 40(2):258–276. https://doi.org/10.1108/00907321211228318Mohr P, Kerbl B, Donoser M, Schmalstieg D, Kalkofen D (2015) Retargeting technical documentation to augmented reality. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ’15. ACM, New York, pp 3337–3346, DOI https://doi.org/10.1145/2702123.2702490, (to appear in print)Mohr P, Mandl D, Tatzgern M, Veas E, Schmalstieg D, Kalkofen D (2017) Retargeting video tutorials showing tools with surface contact to augmented reality. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ’17. ACM, New York, pp 6547–6558, DOI https://doi.org/10.1145/3025453.3025688, (to appear in print)Montgomery DC, Runger GC (2003) Applied statistics and probability for engineers. Wiley, New YorkMorillo P, Orduña JM, Casas S, Fernández M (2019) A comparison study of ar applications versus pseudo-holographic systems as virtual exhibitors for luxury watch retail stores. Multimedia Systems. https://doi.org/10.1007/s00530-019-00606-yMorse JM (2000) Determining sample size. Qual Health Res 10(1):3–5. https://doi.org/10.1177/104973200129118183Muñoz-Montoya F, Juan M, Mendez-Lopez M, Fidalgo C (2019) Augmented reality based on slam to assess spatial short-term memory. IEEE Access 7:2453–2466. https://doi.org/10.1109/ACCESS.2018.2886627Neuhäuser M (2011) Wilcoxon–Mann–Whitney test. Springer, Berlin, pp 1656–1658Neumann U, Majoros A (1998) Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. In: Inproceedings of the IEEE virtual reality annual international symposium (VR ’98), pp 4–11no JJA, Juan MC, Gil-Gómez JA, Mollá R. (2014) A comparative study using an autostereoscopic display with augmented and virtual reality. Behaviour & Information Technology 33(6):646–655. https://doi.org/10.1080/0144929X.2013.815277Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215–228Quint F, Loch F (2015) Using smart glasses to document maintenance processes. Mensch und Computer 2015–WorkshopbandRadkowski R, Herrema J, Oliver J (2015) Augmented reality-based manual assembly support with visual features for different degrees of difficulty. International Journal of Human–Computer Interaction 31(5):337–349. https://doi.org/10.1080/10447318.2014.994194Regenbrecht H, Schubert T (2002) Measuring presence in augmented reality environments: design and a first test of a questionnaire, Porto, PortugalRobertson J (2012) Likert-type scales, statistical methods, and effect sizes. Commun ACM 55(5):6–7. https://doi.org/10.1145/2160718.2160721Rodríguez-Andrés D, Juan MC, Méndez-López M, Pérez-Hernández E, Lluch J (2016) Mnemocity task: Assessment of childrens spatial memory using stereoscopy and virtual environments. PLos ONE 1(8). https://doi.org/10.1371/journal.pone.0161858Sanna A, Manuri F, Lamberti F, Paravati G, Pezzolla P (2015) Using handheld devices to support augmented reality-based maintenance and assembly tasks. In: 2015 IEEE International conference on consumer electronics (ICCE), pp. 178–179. https://doi.org/10.1109/ICCE.2015.7066370Schmidt S, Ehrenbrink P, Weiss B, Voigt-Antons J, Kojic T, Johnston A, Moller S (2018) Impact of virtual environments on motivation and engagement during exergames. In: 2018 Tenth international conference on quality of multimedia experience (qoMEX), pp 1–6. https://doi.org/10.1109/QoMEX.2018.8463389Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’03. ACM, New York, pp 73–80, DOI https://doi.org/10.1145/642611.642626, (to appear in print)Tomás JM, Oliver A, Galiana L, Sancho P, Lila M (2013) Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling: A Multidisciplinary Journal 20(2):299–313. https://doi.org/10.1080/10705511.2013.769394Uva AE, Gattullo M, Manghisi VM, Spagnulo D, Cascella GL, Fiorentino M (2017) Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. The Int J Adv Manuf Technol: 1–13Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Advances in Manufacturing 4(1):1–22. https://doi.org/10.1007/s40436-015-0131-4Westerfield G, Mitrovic A, Billinghurst M (2015) Intelligent augmented reality training for motherboard assembly. Int J Artif Intell Educ 25(1):157–172. https://doi.org/10.1007/s40593-014-0032-xWiedenmaier S, Oehme O, Schmidt L, Luczak H (2003) Augmented reality (ar) for assembly processes - design and experimental evaluation. International Journal of Human-Computer Interaction 16(3):497–514Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence: Teleoperators and Virtual Environments 7(3):225–240Wu HK, Lee SWY, Chang HY, Liang JC (2013) Current status, opportunities and challenges of augmented reality in education. Computers & Education 62:41–49. https://doi.org/10.1016/j.compedu.2012.10.024Yim MYC, Chu SC, Sauer PL (2017) Is augmented reality technology an effective tool for e-commerce? an interactivity and vividness perspective. Journal of Interactive Marketing 39(http://www.sciencedirect.com/science/article/pii/S1094996817300336):89–103. https://doi.org/10.1016/j.intmar.2017.04.001Yuan ML, Ong SK, Nee AYC (2008) Augmented reality for assembly guidance using a virtual interactive tool. Int J Prod Res 46(7):1745–1767. https://doi.org/10.1080/0020754060097293

Smart augmented reality instructional system for mechanical assembly

Quality and efficiency are pivotal indicators of a manufacturing company. Many companies are suffering from shortage of experienced workers across the production line to perform complex assembly tasks such as assembly of an aircraft engine. This could lead to a significant financial loss. In order to further reduce time and error in an assembly, a smart system consisting of multi-modal Augmented Reality (AR) instructions with the support of a deep learning network for tool detection is introduced. The multi-modal smart AR is designed to provide on-site information including various visual renderings with a fine-tuned Region-based Convolutional Neural Network, which is trained on a synthetic tool dataset. The dataset is generated using CAD models of tools augmented onto a 2D scene without the need of manually preparing real tool images. By implementing the system to mechanical assembly of a CNC carving machine, the result has shown that the system is not only able to correctly classify and localize the physical tools but also enables workers to successfully complete the given assembly tasks. With the proposed approaches, an efficiently customizable smart AR instructional system capable of sensing, characterizing the requirements, and enhancing worker\u27s performance effectively has been built and demonstrated --Abstract, page iii