Study of Augmented Reality based manufacturing for further integration of quality control 4.0: a systematic literature review

Augmented Reality (AR) has gradually become a mainstream technology enabling Industry 4.0 and its maturity has also grown over time. AR has been applied to support different processes on the shop-floor level, such as assembly, maintenance, etc. As various processes in manufacturing require high quality and near-zero error rates to ensure the demands and safety of end-users, AR can also equip operators with immersive interfaces to enhance productivity, accuracy and autonomy in the quality sector. However, there is currently no systematic review paper about AR technology enhancing the quality sector. The purpose of this paper is to conduct a systematic literature review (SLR) to conclude about the emerging interest in using AR as an assisting technology for the quality sector in an industry 4.0 context. Five research questions (RQs), with a set of selection criteria, are predefined to support the objectives of this SLR. In addition, different research databases are used for the paper identification phase following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) methodology to find the answers for the predefined RQs. It is found that, in spite of staying behind the assembly and maintenance sector in terms of AR-based solutions, there is a tendency towards interest in developing and implementing AR-assisted quality applications. There are three main categories of current AR-based solutions for quality sector, which are AR-based apps as a virtual Lean tool, AR-assisted metrology and AR-based solutions for in-line quality control. In this SLR, an AR architecture layer framework has been improved to classify articles into different layers which are finally integrated into a systematic design and development methodology for the development of long-term AR-based solutions for the quality sector in the future

Augmented reality and its aspects: a case study for heating systems

Mestrado de dupla diplomação com a UTFPR - Universidade Tecnológica Federal do ParanáThanks to the advances of technology in various domains, and the mixing between real and virtual worlds. Allowed this master’s thesis to explore concepts related to virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR). The development and comparison of Android applications and Microsoft HoloLens aimed to solve a deadlock in the recognition of instructions by the users. We used an interactive manual of assembly and disassembly for taps of residential heaters. Therefore, this work deals with three main parts. Firstly, the exploration of the concepts of VR, AR, MR, and XR. Secondly, 3D modeling and animations techniques. Finally, the development of applications using Vuforia, Wikitude, and MRTK. The users tried our application “HeaterGuideAR” to verify the effectiveness of the instruction passed by the interactive manual. Only a few users had some difficulties at the beginning of the trials. Thus, it was necessary to provide aid tools. However, other users were able to disassemble the faucet without any external help. We suggest continuing this work with more explorations, models, and situations.Graças aos últimos avanços tecnológicos em diversas áreas deram a possibilidade de fazer a mistura do mundo real com o virtual. É com este intuito que esta tese de mestrado veio expor os conceitos relacionados à realidade virtual (RV), realidade aumentada (RA), realidade mista (RM) e realidade estendida (RE). O desenvolvimento e comparação de aplicativos Android e Microsoft HoloLens teve como objetivo resolver um impasse no entendimento de instruções por parte dos usuários. Utilizamos um manual interativo para montagem e desmontagem de torneiras de aquecedores residenciais. Este trabalho, portanto, lida com três partes principais. Na primeira, a exploração dos conceitos de RV, RA, RM e RE. Na segunda, modelagem 3D e técnicas de animações. E por fim, o desenvolvimento de aplicações usando Vuforia, Wikitude e MRTK. A aplicação “HeaterGuideAR” foi testada pelos usuários afim de verificar a eficácia da instrução passada pelo manual interativo. Apenas alguns usuários tiveram algumas dificuldades no início dos testes. Sendo que, foi necessário fornecer algumas ferramentas de auxílio. Mesmo assim, outros usuários conseguiram desmontar a torneira sem ajuda externa. Sugerimos continuar este trabalho com mais explorações, modelos e situações.Mestrado de dupla diplomação com a UTFPR - Universidade Tecnológica Federal do Paran

A Framework for Mobile Augmented Reality in Urban Maintenance

Mobile handheld devices such as smartphones have become increasingly powerful in modern times. Because of this, there has been a surge in 3D graphics-heavy mobile applications that aim to provide immersive experiences. An example of this phenomenon would be Augmented Reality (AR) applications, which have been increasingly popular and offer a wide array of use-cases. The ability to merge the real world with the virtual world seamlessly using the built-in camera of the smartphone brings a whole new world of possibilities, which makes it interesting to explore how such a technology could be used to solve real-world problems. This dissertation focuses on applying this technology in the field of urban maintenance. To do so, a mobile AR application was developed, designed to be used by urban maintenance workers as a field-assistance tool. Using any standard smartphone camera, the developed system can accurately detect any equipment and augment it with relevant information and step-by-step instructions on how to do any required maintenance jobs. Alongside this mobile application, a desktop application was also developed with the purposes of creating and authoring the data and augmentations that should be displayed during a given job, called Archer. Lastly, this dissertation proposes a novel approach to automatically detect and minimize the amount of points (checkpoints) at which the application will ask the user to perform a new equipment recognition, which are useful in order to maintain tracking stability as the user modifies the real-world object during the course of the job. The experiments and user tests conducted during the final stages of this dissertation demonstrate the accuracy and practicality of the developed systems, proving that they can effectively be used to greatly improve the workflow of urban maintenance workers.Dispositivos móveis tais como os smartphones têm-se tornado cada vez mais poderosos nos tempos modernos. Como tal, tem havido um grande aumento na quantidade de aplicações móveis 3D com o intuito de fornecer experiências imersivas. Um exemplo desse fenómeno são as aplicações de Realidade Aumentada (RA), as quais se têm tornado cada vez mais populares, oferecendo um vasto leque de casos de uso. A habilidade de fundir o mundo real com o mundo virtual através da câmara de um smartphone traz todo um novo mundo de possibilidades, o que torna interessante a exploração de como esta tecnologia pode ser usada para resolver problemas no mundo real. Esta dissertação foca-se na aplicação desta tecnologia na área da manutenção urbana. Nesse sentido, foi desenvolvida uma aplicação móvel de RA projetada para ser usada como uma ferramenta de assistência em campo por trabalhadores de manutenção urbana. Usando qualquer câmara de smartphone, este sistema consegue detetar qualquer equipamento de forma precisa e aumentá-lo digitalmente com informação relevante e instruções passo-a-passo de como fazer qualquer trabalho de manutenção. Juntamente com esta aplicação móvel, também foi desenvolvida uma aplicação para desktop — chamada Archer — com o intuito de criar e validar os dados e os objetos digitais que serão apresentados na aplicação mobile durante o curso de um trabalho de manutenção. Por fim, esta dissertação apresenta uma nova solução para a deteção e minimização automática dos pontos (checkpoints) em que a aplicação móvel deverá pedir ao utilizador para efetuar um novo reconhecimento do equipamento, os quais são úteis para manter um tracking fiável e estável à medida que o utilizador vai modificando o equipamento durante um trabalho. As experiências e testes com utilizadores efetuados na fase final desta dissertação demonstram a precisão e praticidade dos sistemas desenvolvidos, provando que estes podem efetivamente ser usados para melhorar o workflow dos trabalhadores de manutenção urbana

Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training

Deep learning (DL) algorithms have achieved significantly high performance in object detection tasks. At the same time, augmented reality (AR) techniques are transforming the ways that we work and connect with people. With the increasing popularity of online and hybrid learning, we propose a new framework for improving students’ learning experiences with electrical engineering lab equipment by incorporating the abovementioned technologies. The DL powered automatic object detection component integrated into the AR application is designed to recognize equipment such as multimeter, oscilloscope, wave generator, and power supply. A deep neural network model, namely MobileNet-SSD v2, is implemented for equipment detection using TensorFlow’s object detection API. When a piece of equipment is detected, the corresponding AR-based tutorial will be displayed on the screen. The mean average precision (mAP) of the developed equipment detection model is 81.4%, while the average recall of the model is 85.3%. Furthermore, to demonstrate practical application of the proposed framework, we develop a multimeter tutorial where virtual models are superimposed on real multimeters. The tutorial includes images and web links as well to help users learn more effectively. The Unity3D game engine is used as the primary development tool for this tutorial to integrate DL and AR frameworks and create immersive scenarios. The proposed framework can be a useful foundation for AR and machine-learning-based frameworks for industrial and educational training

Tracking Support for Collaborative Warehousing using Head-mounted Augmented Reality

In this thesis, we show that the conventional workflows for tracking objects in a warehouse can be enhanced by a new Augmented Reality (AR) service. In particular, we have designed a Unity implementation of an AR toolkit for the Microsoft HoloLens based on a "indoor greenhouse" scenario, which enables the physical locations of plants (together with plant data such as nutritional status) to be tracked as they are moved by greenhouse workers. As part of this case study, we found that a considerable latency occurs when sharing the physical positions of the plant tray between multiple users in a collaborative scenario. To deal with this, we have improved Microsoft's Local Anchor Transfer method which only enables one "spatial anchor" (corresponding to the information of the physical location of an object) to be transferred for one object. Subsequently, we have designed a new One-For-All-Shared-Experiences (OFALL-SE) method to enable many objects to be tracked by one Spatial Anchor. Our results show that use of the OFALL-SE method means that the transmission latencies can be greatly reduced. Although our OFALL-SE method can implement a near-real-time transmission of the spatial anchor when tracking multiple objects, its associated spatial limitations mean that it can only track objects through small regions of space. Following the recommendations of Microsoft, their Azure Spatial Anchor (ASA) service can overcome this limitation and used in a large space, but it requires a new toolkit which is independent with our OFALL-SE technique. ASA also requires external implementations by developers and it also lacks security on the user's anchor information. Thus, we have designed a new technique based on our OFALL-SE technique - Spatial Anchor Based Indoor Asset Tracking (SABIAT) - that achieves similar system features as ASA service such as multiple spatial anchors management over a large area. SABIAT can be implemented as a standalone application or integrated into other AR applications to provide location-based services such as our OFALL-SE method. With this new technique, We also created a standalone AR application - AR-IPS, which show that objects can hop between different spatial anchors, and the historical positions of multiple objects can be tracked, visualized and recorded

Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage

Tesis por compendio[ES] La RA consiste en la superposición de elementos virtuales sobre el entorno real, de manera que el usuario percibe estos elementos como si formaran parte de la realidad que está visualizando. Las aplicaciones de RA en dispositivos móviles permiten visualizar el contenido virtual a través de la cámara del dispositivo. La RA es una herramienta de divulgación muy potente ya que permite añadir a la realidad cualquier tipo de información, desde un simple texto informativo a un modelo 3D interactivo. Tiene infinitas utilidades, puede servir de guía en un museo, puede mostrar la recreación de un monumento destruido, o como en el caso de estudio aquí presentado, ayudar a la interpretación de pinturas rupestres. Esta tesis parte de la idea de que la RA puede mejorar mucho la interpretación del arte rupestre sin alterar ni dañar las pinturas. Puede servir para atraer a un público mayor, dar a conocer la historia de las pinturas rupestres y que al mismo tiempo el visitante tenga una experiencia mucho más enriquecedora. A lo largo de la tesis se ha estudiado en profundidad la técnica de visualización de RA mediante dispositivos móviles. Se han analizado las diferentes librerías de programación mediante casos de estudio en entornos reales y examinado los factores que pueden afectar al reconocimiento de las pinturas. Se ha desarrollado una aplicación de RA aplicada a un caso real de pinturas rupestres y posteriormente ha sido evaluada por un grupo de personas. Finalmente, se ha estudiado el efecto de la luz solar y sus cambios a lo largo del día sobre el reconocimiento de imágenes en entornos al aire libre. Este trabajo proporciona un punto de partida para el desarrollo de aplicaciones de RA aplicadas a la difusión del patrimonio cultural, especialmente centrado en el arte rupestre, un entorno que sufre de unas dificultades añadidas debido a su localización, dificultad de reconocimiento de puntos característicos en las pinturas y los cambios en la luz solar, problemas que se han tratado de resolver a lo largo del estudio. Las principales conclusiones han sido muy favorables, partiendo de librerías de programación disponibles y gratuitas. Se han podido desarrollar un conjunto de aplicaciones de RA en diferentes lugares. Las valoraciones han sido muy positivas, los usuarios que han probado las aplicaciones afirman que la interpretación de las pinturas les resulta más fácil y consiguen entender mejor el propósito de las mismas. El principal inconveniente encontrado es la falta de conocimiento sobre esta técnica y la pérdida de realismo en algunos casos debido a la oclusión, es decir, que los objetos virtuales no se posicionen por detrás de los objetos reales. La buena noticia es que esta tecnología evoluciona muy rápido y durante el desarrollo de la tesis ha habido avances muy grandes, entre ellos, el desarrollo de nuevas librerías de programación desarrolladas por Google y Apple, que proporcionan las herramientas necesarias para crear aplicaciones muy potentes e immersivas, donde el usuario se sentirá parte de los entornos creados.[CA] La RA consisteix en la superposició d'elements virtuals sobre l'entorn real, de manera que l'usuari percep aquests elements com si formaren part de la realitat que està visualitzant. Les aplicacions de RA en dispositius mòbils permeten visualitzar el contingut virtual a través de la cambra del dispositiu. La RA és una eina de divulgació molt potent ja que permet afegir a la realitat qualsevol tipus d'informació, des d'un simple text informatiu a un model 3D interactiu. Té infinites utilitats, pot servir de guia en un museu, pot mostrar la recreació d'un monument destruït, o com en el cas d'estudi ací presentat, ajudar a la interpretació de pintures rupestres. Aquesta tesi parteix de la idea que la RA pot millorar molt la interpretació de l'art rupestre sense alterar ni danyar les pintures. Pot servir per a atraure a un públic major, donar a conéixer la història de les pintures rupestres i que al mateix temps el visitant tinga una experiència molt més enriquidora. Al llarg de la tesi s'ha estudiat en profunditat la tècnica de visualització de RA mitjançant dispositius mòbils. S'han analitzat les diferents llibreries de programació mitjançant casos d'estudi en entorns reals i analitzat els factors que poden afectar el reconeixement de les pintures. S'ha desenvolupat una aplicació de RA aplicada a un cas real de pintures rupestres i posteriorment ha sigut avaluada per un grup de persones. Finalment, s'ha estudiat l'efecte de la llum solar i els seus canvis al llarg del dia sobre el reconeixement d'imatges en entorns a l'aire lliure. Aquest treball proporciona un punt de partida per al desenvolupament d'aplicacions de RA aplicades a la difusió del patrimoni cultural, especialment centrat en l'art rupestre, un entorn que pateix d'unes dificultats afegides a causa de la seua localització, dificultat de reconeixement de punts característics en les pintures i els canvis en la llum solar, problemes que s'han tractat de resoldre al llarg de l'estudi. Les principals conclusions han sigut molt favorables, partint de llibreries de programació disponibles i gratuïtes. S'han pogut desenvolupar un conjunt d'aplicacions de RA en diferents llocs. Les valoracions han sigut molt positives, els usuaris que han provat les aplicacions afirmen que la interpretació de les pintures els resulta més fàcil i aconsegueixen entendre millor el propòsit d'aquestes. El principal inconvenient trobat és la falta de coneixement sobre aquesta tècnica i la perduda de realisme en alguns casos a causa de l'oclusió, és a dir, que els objectes virtuals no es posicionen per darrere dels objectes reals. La bona notícia és que aquesta tecnologia evoluciona molt ràpid i durant el desenvolupament de la tesi hi ha hagut avanços molt grans, entre ells, el desenvolupament de noves llibreries de programació per Google i Apple, que proporcionen les eines necessàries per a crear aplicacions molt potents i immersives, on l'usuari se sentirà part dels entorns creats.[EN] AR consists of superimposing virtual elements on the real environment, so that the user perceives these elements as if they were part of the reality they are looking at. AR applications on smartphones allow virtual content to be visualised through the device's camera. AR is a very powerful tool for dissemination as it allows any type of information to be added to reality, from a simple informative text to an interactive 3D model. It can be used as a guide in a museum, it can show the recreation of a destroyed monument, or, as in the case study presented here, it can help in the interpretation of cave paintings. This thesis is based on the idea that AR can greatly enhance the interpretation of rock art without affecting or damaging the paintings. It can be used to attract a wider audience, to introduce the history of the rock art paintings and at the same time provide the visitor with a much more enriching experience. Throughout the thesis, the technique of AR visualisation using mobile devices has been studied in-depth. The different programming libraries have been analysed by means of case studies in real environments as well as the factors that can affect the paintings recognition. An AR application applied to a real case of rock art paintings has been developed and subsequently evaluated by a group of people. Finally, the effect of sunlight and its changes throughout the day on image recognition in outdoor environments has been studied. This work provides a starting point for the AR applications development applied to the dissemination of cultural heritage, especially focused on rock art, an environment that suffers from additional difficulties due to its location, the difficulty of characteristic points recognition and changes in sunlight, problems that have been tried to solve throughout the study. The main outcomes have been very favourable, using freely available programming libraries, and it has been possible to develop a set of AR applications in different places. The evaluations have been very positive, with users who have tested the applications confirming that the interpretation of the paintings is easier for them and they can better understand the purpose of the paintings. The major drawback is the lack of knowledge about this technique and the loss of realism in some cases due to occlusion, i.e. the virtual objects are not positioned behind the real objects. The good news is that this technology is evolving very fast and during the development of the thesis there have been great advances, among them, the development of new programming libraries developed by Google and Apple, which provide the necessary tools to create very powerful and immersive applications, where the user will feel part of the virtual environments created.Blanco Pons, S. (2021). Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/178895TESISCompendi

Augmented Reality Application in Manufacturing Industry: Maintenance and Non-destructive Testing (NDT) Use Cases

In recent years, a structural transformation of the manufacturing industry has been occurring as a result of the digital revolution. Thus, digital tools are now systematically used throughout the entire value chain, from design to production to marketing, especially virtual and augmented reality. Therefore, the purpose of this paper is to review, through concrete use cases, the progress of these novel technologies and their use in the manufacturing industry

Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence

Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and perform seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences using MAR devices to provide universal access to digital content. Over the past 20 years, several MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discuss the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks, including supported platforms and corresponding features such as tracking, feature extraction, and sensing capabilities; and (4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields and the current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.Peer reviewe