Human factors in instructional augmented reality for intravehicular spaceflight activities and How gravity influences the setup of interfaces operated by direct object selection

In human spaceflight, advanced user interfaces are becoming an interesting mean to facilitate human-machine interaction, enhancing and guaranteeing the sequences of intravehicular space operations. The efforts made to ease such operations have shown strong interests in novel human-computer interaction like Augmented Reality (AR). The work presented in this thesis is directed towards a user-driven design for AR-assisted space operations, iteratively solving issues arisen from the problem space, which also includes the consideration of the effect of altered gravity on handling such interfaces.Auch in der bemannten Raumfahrt steigt das Interesse an neuartigen Benutzerschnittstellen, um nicht nur die Mensch-Maschine-Interaktion effektiver zu gestalten, sondern auch um einen korrekten Arbeitsablauf sicherzustellen. In der Vergangenheit wurden wiederholt Anstrengungen unternommen, Innenbordarbeiten mit Hilfe von Augmented Reality (AR) zu erleichtern. Diese Arbeit konzentriert sich auf einen nutzerorientierten AR-Ansatz, welcher zum Ziel hat, die Probleme schrittweise in einem iterativen Designprozess zu lösen. Dies erfordert auch die Berücksichtigung veränderter Schwerkraftbedingungen

Realidade aumentada para produção assistida em ambiente industrial

Smart factories are becoming more and more common and Augmented Reality (AR) is a pillar of the transition to Industry 4.0 and smart manufacturing. AR can improve many industrial processes such as training, maintenance, assembly, quality control, remote collaboration and others. AR has the potential to revolutionize the way information is accessed, used and exchanged, extending user’s perception and improving their performance. This work proposes a Pervasive AR tool, created in collaboration with industrial partners, to support the training of operators on industrial shop floors while performing production operations. A Human-Centered Design (HCD) methodology was used to identify operators’ difficulties, challenges, and define requirements. After initial meetings with stakeholders, an AR prototype was designed and developed to allow the configuration and visualization of AR content on the shop floor. Several meetings and user studies were conducted to evaluate the developed tools and improve their usability and features. Comparisons between the proposed Head Mounted Display (HMD) solution, the method currently being used in the shopfloor and alternative AR solutions (mobile based) were conducted. The results of user studies suggest that the proposed AR system can significantly improve the performance (up to 70% when compared with the method currently used in the shop floor) of novice operators.Fábricas inteligentes estão a tornar-se cada vez mais comuns e a Realidade Aumentada (Augmented Reality) é essencial para a transição para a Indústria 4.0 e para a produção inteligente. A AR pode ser usada para melhorar muitos processos industriais, tais como treino, assistência, montagem, controlo de qualidade, colaboração remota, entre outros. A AR tem potencial para revolucionar a maneira como a informação é acedida, usada e partilhada, expandindo a perceção do utilizador e melhorando a sua performance. Este trabalho propõe uma ferramenta de AR Pervasiva, criada em colaboração com parceiros da indústria, para ajudar no treino de operadores de chão de fábrica em tarefas de produção fabril. Para identificar as dificuldades, desafios e definir requisitos, foi seguida uma metodologia de Desenho Centrada no Utilizador (HCD). Depois de vários encontros com o público-alvo, um protótipo de AR foi desenhado e desenvolvido para permitir a configuração e visualização de conteúdo em AR na linha de montagem de uma fábrica. Diversas reuniões e testes com utilizadores foram realizados de modo a avaliar as ferramentas desenvolvidas e melhorar a usabilidade e as suas funcionalidades. Foram também realizadas comparações entre a solução de AR proposta, o método atualmente utilizado na linha de produção e uma solução alternativa de AR para dispositivos móveis. Os resultados dos testes de utilizador realizados sugerem que a solução proposta pode melhorar substancialmente a eficiência (até 70% quando comparado com método atualmente utilizado na linha de produção) de novos operadores.Mestrado em Engenharia de Computadores e Telemátic

Evaluating the use of augmented reality to facilitate assembly

Assembly is the process in which two or more objects are joined together through particular sequences and operations. Current practice utilises two-dimensional (2D) drawings as the main visualisation means to guide assembly. Other visualisation means such as three-dimensional (3D) manual and Virtual Reality (VR) technology have also been applied to assist in assembly. As an emerging technology, Augmented Reality (AR) integrates 3D images of virtual objects into a real-world workspace. The insertion of digitalised information into the real-world workspace using AR can provide workers with the means to implement correct assembly procedures with improved accuracy and reduced errors. Despite the substantial application of AR in assembly; related research has rarely been explored from a human cognitive perspective. The limited available cognitive research concerning the applications of AR visualisation means in assembly highlights the need for a structured methodology of addressing cognitive and useability issues for the application potentials of AR technology to be fully realised.This dissertation reviews the issues and discrepancies in using four types of visualisation means (2D drawings, 3D manual prints, VR, and AR) for guiding assembly, and investigates potential cognitive theories to underpin the benefits of animated AR in assembly. A theoretical framework is then put forward, which summarises existing mechanisms concerning visual-spatial information processing and THE Working Memory (WM) processing in the context of spatial cognition theory, active vision theory and THE WM theory, and raises the to-be-validated aspects of the above theories when transferring from the psychological arena to practical instances. Moreover, the dissertation formulates the methodology of configuring a prototype-animated AR system, and devising particular assembly tasks that are normally guided by reference to documentation and a test-bed with a series of experiments.Two experiments were conducted with three testing scenarios: experiment I concerns the evaluation in the first and second scenarios, while experiment II concerns the third scenario. In scenario 1, a small scale LEGO model was used as the assembly and experimental tester task to compare 3D manual prints and AR. This scenario measured the task performance and cognitive workload of using the system for assembly. The second scenario applied the knowledge gained from scenario 1 to the real construction piping assembly. Comparisons were then made as to productivity improvements, cost reduction and the reduction of rework between 2D isometric drawings and AR. Common findings from both scenarios revealed that the AR visualisation yielded shorter task completion time, less assembly errors and lower total task load. Evaluation from the real construction scenario also indicated that the animated AR visualisation significantly shortened the completion time (original time and rework time), payment to assemblers and cost on correcting erroneous assembly.Questionnaire feedback (including NASA task load index) (Hart 2006, 908) revealed that the animated AR visualisation better aided assembly comprehension, and better facilitated information retrieval and collaboration between human and guidance medium. Using the same LEGO tester task, the third scenario measured the training effects of using 3D manual prints and AR among novice assemblers. The results revealed that the learning curve of novice assemblers was reduced (faster learning) and task performance relevant to working memory was increased when implementing AR training. Useability evaluation was conducted based on classical useability methods, to assess the user interface regarding system improvements

In-Situ Instructions Exceed Side-by-Side Instructions in Augmented Reality Assisted Assembly

Blattgerste J, Renner P, Strenge B, Pfeiffer T. In-Situ Instructions Exceed Side-by-Side Instructions in Augmented Reality Assisted Assembly. In: Proceedings of the 11th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA'18). New York, NY, USA: ACM; 2018: 133-140.Driven by endeavors towards Industry 4.0, there is increasing interest in augmented reality (AR) as an approach for assistance in areas like picking, assembly and maintenance. In this work our focus is on AR-based assistance in manual assembly. The design space for AR instructions in this context includes, e.g., side-by-side, 3D or projected 2D presentations. In previous research, the low quality of the AR devices available at the respective time had a significant impact on performance evaluations. Today, a proper and up-to-date comparison of different presentation approaches is missing. This paper presents an improved 3D in-situ instruction and compares it to previously presented techniques. All instructions are implemented on up-to-date AR hardware, namely the Microsoft HoloLens. To support reproducible research, the comparison is made using a standardized benchmark scenario. The results show, contrary to previous research, that in-situ instructions on state-of-the-art AR glasses outperform side-by-side instructions in terms of errors made, task completion time, and perceived task load

Human behavior understanding for worker-centered intelligent manufacturing

“In a worker-centered intelligent manufacturing system, sensing and understanding of the worker’s behavior are the primary tasks, which are essential for automatic performance evaluation & optimization, intelligent training & assistance, and human-robot collaboration. In this study, a worker-centered training & assistant system is proposed for intelligent manufacturing, which is featured with self-awareness and active-guidance. To understand the hand behavior, a method is proposed for complex hand gesture recognition using Convolutional Neural Networks (CNN) with multiview augmentation and inference fusion, from depth images captured by Microsoft Kinect. To sense and understand the worker in a more comprehensive way, a multi-modal approach is proposed for worker activity recognition using Inertial Measurement Unit (IMU) signals obtained from a Myo armband and videos from a visual camera. To automatically learn the importance of different sensors, a novel attention-based approach is proposed to human activity recognition using multiple IMU sensors worn at different body locations. To deploy the developed algorithms to the factory floor, a real-time assembly operation recognition system is proposed with fog computing and transfer learning. The proposed worker-centered training & assistant system has been validated and demonstrated the feasibility and great potential for applying to the manufacturing industry for frontline workers. Our developed approaches have been evaluated: 1) the multi-view approach outperforms the state-of-the-arts on two public benchmark datasets, 2) the multi-modal approach achieves an accuracy of 97% on a worker activity dataset including 6 activities and achieves the best performance on a public dataset, 3) the attention-based method outperforms the state-of-the-art methods on five publicly available datasets, and 4) the developed transfer learning model achieves a real-time recognition accuracy of 95% on a dataset including 10 worker operations”--Abstract, page iv