Guidance and surroundings awareness in outdoor handheld augmented reality

Handheld and wearable devices are becoming ubiquitous in our lives and augmented reality technology is stepping out of the laboratory environment and becoming ready to be used by anyone with portable devices. The success of augmented reality applications for pedestrians depends on different factors including a reliable guidance system and preventing risks. We show that different guidance systems can be supplementary to provide directions to a point of interest and offer clues that help the user find the augmented data when they get close to the location they have to visit. We tested the helpfulness of a map with the points of interest, an image preview of the next point of interest to visit, and an arrow showing the direction to it. The results show that the effectiveness of these guidance systems depend on the distance to the point of interest and the accuracy of the data obtained from the Global Positioning System. We also measured the total time that participants spent looking at the screen, as well as the perceived elapsed time as a measurement of real world dissociation. Finally, we discuss preliminary results to minimize the risk of accidents when using augmented reality applications in an outdoor urban environment

Overcoming the limitations of commodity augmented reality head mounted displays for use in product assembly

Numerous studies have shown the effectiveness of utilizing Augmented Reality (AR) to deliver work instructions for complex assemblies. Traditionally, this research has been performed using hand-held displays, such as smartphones and tablets, or custom-built Head Mounted Displays (HMDs). AR HMDs have been shown to be especially effective for assembly tasks as they allow the user to remain hands-free while receiving work instructions. Furthermore, in recent years a wave of commodity AR HMDs have come to market including the Microsoft HoloLens, Magic Leap One, Meta 2, and DAQRI Smart Glasses. These devices present a unique opportunity for delivering assembly instructions due to their relatively low cost and accessibility compared to custom built AR HMD solutions of the past. Despite these benefits, the technology behind these HMDs still contains many limitations including input, user interface, spatial registration, navigation and occlusion. To accurately deliver work instructions for complex assemblies, the hardware limitations of these commodity AR HMDs must be overcome. For this research, an AR assembly application was developed for the Microsoft HoloLens using methods specifically designed to address the aforementioned issues. Input and user interface methods were implemented and analyzed to maximize the usability of the application. An intuitive navigation system was developed to guide users through a large training environment, leading them to the current point of interest. The native tracking system of the HoloLens was augmented with image target tracking capabilities to stabilize virtual content, enhance accuracy, and account for spatial drift. This fusion of marker-based and marker-less tracking techniques provides a novel approach to display robust AR assembly instructions on a commodity AR HMD. Furthermore, utilizing this novel spatial registration approach, the position of real-world objects was accurately registered to properly occlude virtual work instructions. To render the desired effect, specialized computer graphics methods and custom shaders were developed and implemented for an AR assembly application. After developing novel methods to display work instructions on a commodity AR HMD, it was necessary to validate that these work instructions were being accurately delivered. Utilizing the sensors on the HoloLens, data was collected during the assembly process regarding head position, orientation, assembly step times, and an estimation of spatial drift. With the addition of wearable physiological sensor data, this data was fused together in a visualization application to validate instructions were properly delivered and provide an opportunity for an analysist to examine trends within an assembly session. Additionally, the spatial drift data was then analyzed to gain a better understanding of how spatial drift accumulates over time and ensure that the spatial registration mitigation techniques was effective. Academic research has shown that AR may substantial reduce cost for assembly operations through a reduction in errors, time, and cognitive workload. This research provides novel solutions to overcome the limitations of commodity AR HMDs and validate their use for product assembly. Furthermore, the research provided in this thesis demonstrates the potential of commodity AR HMDs and how their limitations can be mitigated for use in product assembly tasks

Reorganization of the Warehouse Activities Processes with the Framing of their Problems and Solutions

Purpose: The aim of this study is to examine and identify goods processing operations in a warehouse and propose a reorganization of tasks to minimize processing time.   Theoretical framework: Current literature highlights the need to reorganize and adapt the operations chain (LOMBA Cédric, 2018). Some areas still need to be studied in depth to improve the supply chain. The literature on warehouse management has not sufficiently studied partial time management or closed-loop time management in the supply chain (CLSC) from goods-in to goods-out. The potential links between the risks of lost time and the management of the quality of the goods handling service have not been studied in depth.   Design/Methodology/Approach: The in-depth analysis of recent international studies published on warehouse management and the supply chain enables us to identify the studies that have the greatest impact on research.   Findings:  The result of this work is the development of a double typology, one linked to the chain of operations and the other to the constraints linked to the CLSC in a warehouse. Tools will be applied to diagnose the constraints and factors adversely goods handling operations.   Research, Practical & Social implications: To help logisticians and researchers, we have proposed an operational reformulation of a warehouse, enriched with a logical framework linking the problems and their potential solutions.   Originality/Value: The article proposes an original vision to improve the spatio-temporal organization of warehouses, by synthesizing its different entities, operations, and processes while presenting all the existing problems and identifying their possible solutions

An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task

Many studies have confirmed the benefits of using Augmented Reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware for complex assembly tasks. For this research, previously published data using Desktop Model Based Instructions (MBI), Tablet MBI, and Tablet AR instructions were compared to new assembly data collected using AR instructions on the Microsoft HoloLens Head Mounted Display (HMD). Participants completed a mock wing assembly task, and measures like completion time, error count, Net Promoter Score, and qualitative feedback were recorded. The HoloLens condition yielded faster completion times than all other conditions. HoloLens users also had lower error rates than those who used the non-AR conditions. Despite the performance benefits of the HoloLens AR instructions, users of this condition reported lower net promoter scores than users of the Tablet AR instructions. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the user feedback favored the Tablet AR condition, the HoloLens condition resulted in significantly faster assembly times. As a result, it is recommended to use the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders better understand the benefits of different AR technology for manual assembly tasks

Evaluation of graphical user interfaces for augmented reality based manual assembly support

Augmented reality (AR) technology is advancing rapidly and promises benefits to a wide variety of applications&mdashincluding manual assembly and maintenance tasks. This thesis addresses the design of user interfaces for AR applications, focusing specifically on information presentation interface elements for assembly tasks. A framework was developed and utilized to understand and classify these elements, as well as to evaluate numerous existing AR assembly interfaces from literature. Furthermore, a user study was conducted to investigate the strengths and weaknesses of concrete and abstract AR interface elements in an assembly scenario, as well as to compare AR assembly instructions against common paper-based assembly instructions. The results of this study supported, at least partially, the three hypotheses that concrete AR elements are more suitable to convey part manipulation information than abstract AR elements, that concrete AR and paper-based instructions lead to faster assembly times than abstract AR instructions alone, and that concrete AR instructions lead to greater increases in user confidence than paper-based instructions. The study failed to support the hypothesis that abstract AR elements are more suitable for part identification than concrete AR elements. Finally, the study results and hypothesis conclusions are used to suggest future work regarding interface element design for AR assembly applications

Harnessing Technology: analysis of emerging trends affecting the use of technology in education – October 2009

Research to support the delivery and development of Harnessing Technology: Next Generation Learning 2008–1

Augmented reality for order picking using wearable computers with head-mounted displays

Order picking - the task of assembling orders in a warehouse is still the biggest expenditure in modern warehouses, therefore it is worth it focusing new research on the improvement of technologies and strategies which support this task. The recent development and the enhanced availability of head-mounted displays (HMDs) prompts further research regarding HMDs as a supporting technology for industrial applications. Based on research in related topics and the precise analysis of the use case we composed a concept for a system that supports the order picking process with the use of HMDs. In order to evaluate the concept a prototype that implemented the features of the concept, was built. The range of functionalities contains real time visual user guiding, visual task support, context and much more important step awareness that exempts the user from additional work steps and instant feedback in case of wrong actions of the user which enables direct reaction to errors. Tough a comparative user study, we compared the prototype with two traditional order picking methods and another order picking support system that uses projections for user guiding. The results show, that our prototype shows the lowest occurrence of long therm errors with the downside of a higher task completion time. However the study proved that the concept is worth being taken into consideration and be improved in further research.Komissionierung, die Aufgabe, Bestellungen in einem Lager zusammenzustellen, ist immer noch der größte Kostenpunkt in modernen Warenlagern. Deshalb lohnt es sich, Forschung auf die Verbesserung von Technologien und Strategien, die diese Aufgabe unterstützen zu fokussieren. Der jüngste Aufstieg und die verbesserte Verfügbarkeit von Head-Mounted Displays (HMDs) fordert Nachforschung im Bezug auf HMDs als unterstützende Technologie für industrielle Anwendungen. Basierend auf Forschung in verwandten Themen und der genauen Analyse des Anwendungsfalles haben wir ein Konzept zusammengestellt, welches den Komissionierprozess mit Hilfe von HMDs unterstützt. Um dieses Konzept auszuwerten wurde ein Prototyp gebaut, der die Eigenschaften des Konzepts umsetzt. Der Umfang der Funktionalitäten umfasst visuelle Echtzeitbenutzerführung, visuelle Aufgabenunterstützung, Bewusstsein für den aktuellen Kontext und, viel wichtiger den momentanen Arbeitsschritt, was den Benutzer von zusätzlichen Arbeitsschritten befreit und sofortige Rückmeldung im Falle falscher Handlungen des Benutzers, was direkte Reaktion auf Fehler ermöglicht. Dieser Prototyp wurde in einer Benutzerstudie mit zwei traditionellen Systemen zu Komissionierungsunterstützung und einem weiteren System, das Projektionen zur Benutzerführung benutzt verglichen. Die Ergebnisse zeigen, dass mit unserem Prototyp am wenigsten Langzeitfehler auftreten, was eine längere Zeit zur Erledigung der Aufgabe zur Folge hat. Die Studie hat bewiesen, dass dieses Konzept in zukünftiger Forschung mit beachtet werden sollte und sich eine Weiterentwicklung lohnt

Développement d'un système en réalité augmentée d'assistance aux opérateurs de machinerie lourde dans une cour à bois

De nos jours, la Réalité augmentée (RA) prend une place de plus en plus prépondérante dans l’industrie en fournissant de l’assistance contextuelle à un niveau jamais atteint précédemment. En particulier, dans le domaine de la foresterie, les cours à bois sont des zones de stockage de paquets de bois, où évoluent des engins de machinerie lourde réalisant la manutention. Dans notre étude, on s’attache à développer une application en RA capable d’assister les opérateurs des machines évoluant dans la cour. On utilise pour cela le visiocasque HoloLens de Microsoft. Ce visiocasque possède un dispositif interne pour mesurer sa position basé sur une centrale inertielle et des caméras analysant l’environnement. Mais nos tests ont montré que ce dispositif était incapable de fournir une position fiable à l’intérieur d’un véhicule en mouvement. En conséquence nous avons conçu un système capable de mesurer la position et l’orientation d’un visiocasque à l’intérieur d’un véhicule évoluant en extérieur avec la précision requise pour obtenir une qualité d’expérience satisfaisante en RA. Nous n’avons pas pu mettre en pratique ce système car nous n’avions pas accès au matériel haute précision nécessaire. Nous avons cependant développé un prototype applicatif fonctionnant en intérieur en utilisant les informations du système de positionnement embarqué de l’HoloLens afin de mettre en pratique et valider les solutions de design d’interface que nous avons conçues. En parallèle, nous avons aussi mené une réflexion pour déterminer comment instrumenter les chargeurs à fourche servant à la manutention dans la cour à bois afin qu’ils puissent s’interfacer avec le reste du système d’assistance aux opérateurs. La manutention dans une cour à bois étant rarement la tâche d’un seul chargeur, nous avons ainsi conçu et mis en pratique une architecture réseau permettant d’interconnecter tous les chargeurs instrumentés de la cour avec un serveur d’inventaire qui était développé en parallèle avec notre étude. De plus, nous nous sommes assurés de la robustesse du système face à des connexions instables telles que rencontrées en utilisant des liens sans-fil à bord de véhicules en mouvement. Nous avons donc conçu un système complet d’assistance en RA aux opérateurs de machinerie lourde dans une cour à bois, et nous en avons mis en pratique l’interface ainsi que l’ensemble de l’architecture réseautique.----------ABSTRACT: Nowadays, augmented reality (AR) has taken a central role in industrial settings by providing contextual help at a level never reached before. For example, in the domain of forestry, timber yards are storage areas for wood stacks. In these yards, one can find heavy equipment tasked with handling material. In our study, we aim to develop an AR application that can assist the operators working in the timber yard. For this purpose, we elected to use the HoloLens head-mounted display (HMD) sold by Microsoft. The HMD has an embedded positioning device that uses an inertial measurement unit (IMU) and environment sensing cameras. However, our results showed this device is not able to provide a reliable position when used inside a moving vehicle. Thus we designed an attitude measurement system able to locate and give the orientation of an HMD sitting inside a vehicle driving outdoors, with enough precision to provide a satisfying experience in AR. However, we were not able to fully implement this system due to lacking access the high performance equipment required. Nonetheless, we developed an application prototype designed to work indoors using the embedded HoloLens positioning system. We did so in order to implement and validate the user interface we designed. We also lead a reflection in parallel, to determine a way to instrument the fork-loaders used for handling in the timber yard. That way, they would be able to interface with the rest of the operator assistance system. Handling in a timber yard is rarely – if ever – a one-loader task. Thus we designed and implemented a network architecture allowing for connections between the instrumented loaders and an inventory server that was developed in parallel of our project. Moreover, we made sure the system was robust against unstable connections, as those can be common when using a wireless link in a moving vehicle. In summary, we designed an entire system to assist heavy equipment operators in a timber yard, and we implemented the interface and user experience, as well as the whole network architecture

Human factors in instructional augmented reality for intravehicular spaceflight activities and How gravity influences the setup of interfaces operated by direct object selection

In human spaceflight, advanced user interfaces are becoming an interesting mean to facilitate human-machine interaction, enhancing and guaranteeing the sequences of intravehicular space operations. The efforts made to ease such operations have shown strong interests in novel human-computer interaction like Augmented Reality (AR). The work presented in this thesis is directed towards a user-driven design for AR-assisted space operations, iteratively solving issues arisen from the problem space, which also includes the consideration of the effect of altered gravity on handling such interfaces.Auch in der bemannten Raumfahrt steigt das Interesse an neuartigen Benutzerschnittstellen, um nicht nur die Mensch-Maschine-Interaktion effektiver zu gestalten, sondern auch um einen korrekten Arbeitsablauf sicherzustellen. In der Vergangenheit wurden wiederholt Anstrengungen unternommen, Innenbordarbeiten mit Hilfe von Augmented Reality (AR) zu erleichtern. Diese Arbeit konzentriert sich auf einen nutzerorientierten AR-Ansatz, welcher zum Ziel hat, die Probleme schrittweise in einem iterativen Designprozess zu lösen. Dies erfordert auch die Berücksichtigung veränderter Schwerkraftbedingungen

Augmented Reality Interfaces for Procedural Tasks

Procedural tasks involve people performing established sequences of activities while interacting with objects in the physical environment to accomplish particular goals. These tasks span almost all aspects of human life and vary greatly in their complexity. For some simple tasks, little cognitive assistance is required beyond an initial learning session in which a person follows one-time compact directions, or even intuition, to master a sequence of activities. In the case of complex tasks, procedural assistance may be continually required, even for the most experienced users. Approaches for rendering this assistance employ a wide range of written, audible, and computer-based technologies. This dissertation explores an approach in which procedural task assistance is rendered using augmented reality. Augmented reality integrates virtual content with a user's natural view of the environment, combining real and virtual objects interactively, and aligning them with each other. Our thesis is that an augmented reality interface can allow individuals to perform procedural tasks more quickly while exerting less effort and making fewer errors than other forms of assistance. This thesis is supported by several significant contributions yielded during the exploration of the following research themes: What aspects of AR are applicable and beneficial to the procedural task problem? In answering this question, we developed two prototype AR interfaces that improve procedural task accomplishment. The first prototype was designed to assist mechanics carrying out maintenance procedures under field conditions. An evaluation involving professional mechanics showed our prototype reduced the time required to locate procedural tasks and resulted in fewer head movements while transitioning between tasks. Following up on this work, we constructed another prototype that focuses on providing assistance in the underexplored psychomotor phases of procedural tasks. This prototype presents dynamic and prescriptive forms of instruction and was evaluated using a demanding and realistic alignment task. This evaluation revealed that the AR prototype allowed participants to complete the alignment more quickly and accurately than when using an enhanced version of currently employed documentation systems. How does the user interact with an AR application assisting with procedural tasks? The application of AR to the procedural task problem poses unique user interaction challenges. To meet these challenges, we present and evaluate a novel class of user interfaces that leverage naturally occurring and otherwise unused affordances in the native environment to provide a tangible user interface for augmented reality applications. This class of techniques, which we call Opportunistic Controls, combines hand gestures, overlaid virtual widgets, and passive haptics to form an interface that was proven effective and intuitive during quantitative evaluation. Our evaluation of these techniques includes a qualitative exploration of various preferences and heuristics for Opportunistic Control-based designs