Towards Enabling Human-Robot Collaboration in Industry: Identification of Current Implementation Barriers

Human-robot collaboration (HRC) is designed to combine the repeatability and precision of robots with the flexibility and adaptability of human workers. However, despite being researched for several years, HRC applications are still not broadly adopted in the industry. This study aims to identify current barriers to HRC adoption in the industry from a practical perspective. Therefore, a qualitative explorative approach based on semi-structured interviews with knowledgeable industry experts was chosen. The study was conducted in cooperation between IMT Nord Europe and the Technical University of Munich in France and Germany. Thereby, several experts from various backgrounds in areas such as robot manufacturing, system integration, and robot application in manufacturing were interviewed. These interviews are inductively analysed, and the findings are compared to the state-of-the-art in scientific HRC research. The study offers insights into the practical barriers to HRC adoption resulting from the technical, economic, social, and normative dimensions as well as the trade-offs between them. Based on these insights, opportunities for future research are identified

Wire Harness Assembly Process Supported by Collaborative Robots: Literature Review and Call for R&D

The wire harness assembly process is a complicated manufacturing activity, which is becoming more complex because of the evolving nature of mechatronic and electronic products that require more connectors, sensors, controllers, communication networking, etc. Furthermore, the demand for wire harnesses continues to grow in all industries worldwide as the majority of equipment, appliances, machinery, vehicles, etc., are becoming "smart" (i.e., more mechatronic or electronic). Moreover, most of the wire harness assembly process tasks are done manually, and most of these are considered non-ergonomic for human assembly workers. Hence, the wire harness manufacturing industry is faced with the challenge of increasing productivity while improving the occupational health of its human assembly workers. The purpose of this paper is to conduct a literature review exploring the state of the use of collaborative robots in the wire harness assembly process due to their potential to reduce current occupational health problems for human assembly workers and increase the throughput of wire harness assembly lines, and to provide main findings, discussion, and further research directions for collaborative robotics in this application domain. Eleven papers were found in the scientific literature. All papers demonstrated the potential of collaborative robots to improve the productivity of wire harness assembly lines, and two of these in particular on the ergonomics of the wire harness assembly process. None of the papers reviewed presented a cost-benefit or a cycle time analysis to qualitatively and/or quantitatively measure the impact of the incorporation of collaborative robots in the wire harness assembly process. This represents an important area of opportunity for research with relevance to industry. Three papers remark on the importance of the integration of computer vision systems into a collaborative wire harness assembly process to make this more versatile as many types of wire harnesses exist. The literature review findings call for further research and technological developments in support of the wire harness manufacturing industry and its workers in four main categories: (i) Collaborative Robotics and Grippers, (ii) Ergonomics, (iii) Computer Vision Systems, and (iv) Implementation Methodologies

On cognitive assistant robots for reducing variability in industrial human-robot activities

In the industrial domain, one important research activity for cognitive robotics is the development of assistant robots. In this work, we show how the use of a cognitive assistant robot can contribute to (i) improving task effectiveness and productivity, (ii) providing autonomy for the human supervisor to make decisions, providing or improving human operators’ skills, and (iii) giving feedback to the human operator in the loop. Our approach is evaluated on variability reduction in a manual assembly system. The overall study and analysis are performed on a model of the assembly system obtained using the Functional Resonance Analysis Method (FRAM) and tested in a robotic simulated scenario. Results show that a cognitive assistant robot is a useful partner in the role of improving the task effectiveness of human operators and supervisors.This work has been co-financed by the European Regional Development Fund of the European Union in the framework of the ERDF Operational Program of Catalonia 2014-2020, grant number 001-P-001643. Cecilio Angulo has been partly supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 825619 (AI4EU).Peer ReviewedPostprint (published version

Human Digital Twin: A Survey

Digital twin has recently attracted growing attention, leading to intensive research and applications. Along with this, a new research area, dubbed as "human digital twin" (HDT), has emerged. Similar to the conception of digital twin, HDT is referred to as the replica of a physical-world human in the digital world. Nevertheless, HDT is much more complicated and delicate compared to digital twins of any physical systems and processes, due to humans' dynamic and evolutionary nature, including physical, behavioral, social, physiological, psychological, cognitive, and biological dimensions. Studies on HDT are limited, and the research is still in its infancy. In this paper, we first examine the inception, development, and application of the digital twin concept, providing a context within which we formally define and characterize HDT based on the similarities and differences between digital twin and HDT. Then we conduct an extensive literature review on HDT research, analyzing underpinning technologies and establishing typical frameworks in which the core HDT functions or components are organized. Built upon the findings from the above work, we propose a generic architecture for the HDT system and describe the core function blocks and corresponding technologies. Following this, we present the state of the art of HDT technologies and applications in the healthcare, industry, and daily life domain. Finally, we discuss various issues related to the development of HDT and point out the trends and challenges of future HDT research and development

Collaboration in Co-located Automotive Applications

Virtual reality systems offer substantial potential in supporting decision processes based purely on computer-based representations and simulations. The automotive industry is a prime application domain for such technology, since almost all product parts are available as three-dimensional models. The consideration of ergonomic aspects during assembly tasks, the evaluation of humanmachine interfaces in the car interior, design decision meetings as well as customer presentations serve as but a few examples, wherein the benefit of virtual reality technology is obvious. All these tasks require the involvement of a group of people with different expertises. However, current stereoscopic display systems only provide correct 3D-images for a single user, while other users see a more or less distorted virtual model. This is a major reason why these systems still face limited acceptance in the automotive industry. They need to be operated by experts, who have an advanced understanding of the particular interaction techniques and are aware of the limitations and shortcomings of virtual reality technology. The central idea of this thesis is to investigate the utility of stereoscopic multi-user systems for various stages of the car development process. Such systems provide multiple users with individual and perspectively correct stereoscopic images, which are key features and serve as the premise for the appropriate support of collaborative group processes. The focus of the research is on questions related to various aspects of collaboration in multi-viewer systems such as verbal communication, deictic reference, embodiments and collaborative interaction techniques. The results of this endeavor provide scientific evidence that multi-viewer systems improve the usability of VR-applications for various automotive scenarios, wherein co-located group discussions are necessary. The thesis identifies and discusses the requirements for these scenarios as well as the limitations of applying multi-viewer technology in this context. A particularly important gesture in real-world group discussions is referencing an object by pointing with the hand and the accuracy which can be expected in VR is made evident. A novel two-user seating buck is introduced for the evaluation of ergonomics in a car interior and the requirements on avatar representations for users sitting in a car are identified. Collaborative assembly tasks require high precision. The novel concept of a two-user prop significantly increases the quality of such a simulation in a virtual environment and allows ergonomists to study the strain on workers during an assembly sequence. These findings contribute toward an increased acceptance of VR-technology for collaborative development meetings in the automotive industry and other domains.Virtual-Reality-Systeme sind ein innovatives Instrument, um mit Hilfe computerbasierter Simulationen Entscheidungsprozesse zu unterstützen. Insbesondere in der Automobilbranche spielt diese Technologie eine wichtige Rolle, da heutzutage nahezu alle Fahrzeugteile in 3D konstruiert werden. Im Entwicklungsbereich der Automobilindustrie werden Visualisierungssysteme darüber hinaus bei der Untersuchung ergonomischer Aspekte von Montagevorgängen, bei der Bewertung der Mensch-Maschine-Schnittstelle im Fahrzeuginterieur, bei Designentscheidungen sowie bei Kundenpräsentationen eingesetzt. Diese Entscheidungsrunden bedürfen der Einbindung mehrerer Experten verschiedener Fachbereiche. Derzeit verfügbare stereoskopische Visualisierungssysteme ermöglichen aber nur einem Nutzer eine korrekte Stereosicht, während sich für die anderen Teilnehmer das 3D-Modell verzerrt darstellt. Dieser Nachteil ist ein wesentlicher Grund dafür, dass derartige Systeme bisher nur begrenzt im Automobilbereich anwendbar sind. Der Fokus dieser Dissertation liegt auf der Untersuchung der Anwendbarkeit stereoskopischer Mehrbenutzer-Systeme in verschiedenen Stadien des automobilen Entwicklungsprozesses. Derartige Systeme ermöglichen mehreren Nutzern gleichzeitig eine korrekte Stereosicht, was eine wesentliche Voraussetzung für die Zusammenarbeit in einer Gruppe darstellt. Die zentralen Forschungsfragen beziehen sich dabei auf die Anforderungen von kooperativen Entscheidungsprozessen sowie den daraus resultierenden Aspekten der Interaktion wie verbale Kommunikation, Gesten sowie virtuelle Menschmodelle und Interaktionstechniken zwischen den Nutzern. Die Arbeit belegt, dass stereoskopische Mehrbenutzersysteme die Anwendbarkeit virtueller Techniken im Automobilbereich entscheidend verbessern, da sie eine natürliche Kommunikation zwischen den Nutzern fördern. So ist die Unterstützung natürlicher Gesten beispielsweise ein wichtiger Faktor und es wird dargelegt, welche Genauigkeit beim Zeigen mit der realen Hand auf virtuelle Objekte erwartet werden kann. Darüber hinaus werden Anforderungen an virtuelle Menschmodelle anhand einer Zweibenutzer-Sitzkiste identifiziert und untersucht. Diese Form der Simulation, bei der die Nutzer nebeneinander in einem Fahrzeugmodell sitzen, dient vor allem der Bewertung von Mensch-Maschine-Schnittstellen im Fahrzeuginterieur. Des Weiteren wird das neue Konzept eines Mehrbenutzer-Werkzeugs in die Arbeit mit einbezogen, da hier verdeutlicht wird wie die Simulation von Montagevorgängen in virtuellen Umgebungen mit passivem haptischem Feedback zu ergonomischen Verbesserungen entsprechender Arbeitsvorgänge in der Realität beitragen kann. Diese Konzepte veranschaulichen wie VR-Systeme zur Unterstützung kollaborativer Prozesse in der Automobilbranche und darüber hinaus eingesetzt werden können

Dynamic Speed and Separation Monitoring with On-Robot Ranging Sensor Arrays for Human and Industrial Robot Collaboration

This research presents a flexible and dynamic implementation of Speed and Separation Monitoring (SSM) safety measure that optimizes the productivity of a task while ensuring human safety during Human-Robot Collaboration (HRC). Unlike the standard static/fixed demarcated 2D safety zones based on 2D scanning LiDARs, this research presents a dynamic sensor setup that changes the safety zones based on the robot pose and motion. The focus of this research is the implementation of a dynamic SSM safety configuration using Time-of-Flight (ToF) laser-ranging sensor arrays placed around the centers of the links of a robot arm. It investigates the viability of on-robot exteroceptive sensors for implementing SSM as a safety measure. Here the implementation of varying dynamic SSM safety configurations based on approaches of measuring human-robot separation distance and relative speeds using the sensor modalities of ToF sensor arrays, a motion-capture system, and a 2D LiDAR is shown. This study presents a comparative analysis of the dynamic SSM safety configurations in terms of safety, performance, and productivity. A system of systems (cyber-physical system) architecture for conducting and analyzing the HRC experiments was proposed and implemented. The robots, objects, and human operators sharing the workspace are represented virtually as part of the system by using a digital-twin setup. This system was capable of controlling the robot motion, monitoring human physiological response, and tracking the progress of the collaborative task. This research conducted experiments with human subjects performing a task while sharing the robot workspace under the proposed dynamic SSM safety configurations. The experiment results showed a preference for the use of ToF sensors and motion capture rather than the 2D LiDAR currently used in the industry. The human subjects felt safe and comfortable using the proposed dynamic SSM safety configuration with ToF sensor arrays. The results for a standard pick and place task showed up to a 40% increase in productivity in comparison to a 2D LiDAR

Human-Robot Collaboration in Automotive Assembly

In the past decades, automation in the automobile production line has significantly increased the efficiency and quality of automotive manufacturing. However, in the automotive assembly stage, most tasks are still accomplished manually by human workers because of the complexity and flexibility of the tasks and the high dynamic unconstructed workspace. This dissertation is proposed to improve the level of automation in automotive assembly by human-robot collaboration (HRC). The challenges that eluded the automation in automotive assembly including lack of suitable collaborative robotic systems for the HRC, especially the compact-size high-payload mobile manipulators; teaching and learning frameworks to enable robots to learn the assembly tasks, and how to assist humans to accomplish assembly tasks from human demonstration; task-driving high-level robot motion planning framework to make the trained robot intelligently and adaptively assist human in automotive assembly tasks. The technical research toward this goal has resulted in several peer-reviewed publications. Achievements include: 1) A novel collaborative lift-assist robot for automotive assembly; 2) Approaches of vision-based robot learning of placing tasks from human demonstrations in assembly; 3) Robot learning of assembly tasks and assistance from human demonstrations using Convolutional Neural Network (CNN); 4) Robot learning of assembly tasks and assistance from human demonstrations using Task Constraint-Guided Inverse Reinforcement Learning (TC-IRL); 5) Robot learning of assembly tasks from non-expert demonstrations via Functional Objective-Oriented Network (FOON); 6) Multi-model sampling-based motion planning for trajectory optimization with execution consistency in manufacturing contexts. The research demonstrates the feasibility of a parallel mobile manipulator, which introduces novel conceptions to industrial mobile manipulators for smart manufacturing. By exploring the Robot Learning from Demonstration (RLfD) with both AI-based and model-based approaches, the research also improves robots’ learning capabilities on collaborative assembly tasks for both expert and non-expert users. The research on robot motion planning and control in the dissertation facilitates the safety and human trust in industrial robots in HRC

Programming Robots by Demonstration using Augmented Reality

O mundo está a viver a quarta revolução industrial, a Indústria 4.0; marcada pela crescente inteligência e automação dos sistemas industriais. No entanto, existem tarefas que são muito complexas ou caras para serem totalmente automatizadas, seria mais eficiente se a máquina pudesse trabalhar com o ser humano, não apenas partilhando o mesmo espaço de trabalho, mas como colaboradores úteis. O foco da investigação para solucionar esse problema está em sistemas de interação homem-robô, percebendo em que aplicações podem ser úteis para implementar e quais são os desafios que enfrentam. Neste contexto, uma melhor interação entre as máquinas e os operadores pode levar a múltiplos benefícios, como menos, melhor e mais fácil treino, um ambiente mais seguro para o operador e a capacidade de resolver problemas mais rapidamente. O tema desta dissertação é relevante na medida em que é necessário aprender e implementar as tecnologias que mais contribuem para encontrar soluções para um trabalho mais simples e eficiente na indústria. Assim, é proposto o desenvolvimento de um protótipo industrial de um sistema de interação homem-máquina através de Realidade Estendida, no qual o objetivo é habilitar um operador industrial sem experiência em programação, a programar um robô colaborativo utilizando o Microsoft HoloLens 2. O sistema desenvolvido é dividido em duas partes distintas: o sistema de tracking, que regista o movimento das mãos do operador, e o sistema de tradução da programação por demonstração, que constrói o programa a ser enviado ao robô para que ele se mova. O sistema de monitorização e supervisão é executado pelo Microsoft HoloLens 2, utilizando a plataforma Unity e Visual Studio para programá-lo. A base do sistema de programação por demonstração foi desenvolvida em Robot Operating System (ROS). Os robôs incluídos nesta interface são Universal Robots UR5 (robô colaborativo) e ABB IRB 2600 (robô industrial). Adicionalmente, a interface foi construída para incorporar facilmente mais robôs.The world is living the fourth industrial revolution, Industry 4.0; marked by the increasing intelligence and automation of manufacturing systems. Nevertheless, there are types of tasks that are too complex or too expensive to be fully automated, it would be more efficient if the machine were able to work with the human, not only by sharing the same workspace but also as useful collaborators. A possible solution to that problem is on human-robot interactions systems, understanding the applications where they can be helpful to implement and what are the challenges they face. In this context a better interaction between the machines and the operators can lead to multiples benefits, like less, better, and easier training, a safer environment for the operator and the capacity to solve problems quicker. The focus of this dissertation is relevant as it is necessary to learn and implement the technologies which most contribute to find solutions for a simpler and more efficient work in industry. This dissertation proposes the development of an industrial prototype of a human machine interaction system through Extended Reality (XR), in which the objective is to enable an industrial operator without any programming experience to program a collaborative robot using the Microsoft HoloLens 2. The system itself is divided into two different parts: the tracking system, which records the operator's hand movement, and the translator of the programming by demonstration system, which builds the program to be sent to the robot to execute the task. The monitoring and supervision system is executed by the Microsoft HoloLens 2, using the Unity platform and Visual Studio to program it. The programming by demonstration system's core was developed in Robot Operating System (ROS). The robots included in this interface are Universal Robots UR5 (collaborative robot) and ABB IRB 2600 (industrial robot). Moreover, the interface was built to easily add other robots

A socio-technical approach for assistants in human-robot collaboration in industry 4.0

The introduction of technologies disruptive of Industry 4.0 in the workplace integrated through human cyber-physical systems causes operators to face new challenges. These are reflected in the increased demands presented in the operator's capabilities physical, sensory, and cognitive demands. In this research, cognitive demands are the most interesting. In this perspective, assistants are presented as a possible solution, not as a tool but as a set of functions that amplify human capabilities, such as exoskeletons, collaborative robots for physical capabilities, virtual and augmented reality for sensory capabilities. Perhaps chatbots and softbots for cognitive capabilities, then the need arises to ask ourselves: How can operator assistance systems 4.0 be developed in the context of industrial manufacturing? In which capacities does the operator need more assistance? From the current paradigm of systematization, different approaches are used within the context of the workspace in industry 4.0. Thus, the functional resonance analysis method (FRAM) is used to model the workspace from the sociotechnical system approach, where the relationships between the components are the most important among the functions to be developed by the human-robot team. With the use of simulators for both robots and robotic systems, the behavior of the variability of the human-robot team is analyzed. Furthermore, from the perspective of cognitive systems engineering, the workspace can be studied as a joint cognitive system, where cognition is understood as distributed, in a symbiotic relationship between the human and technological agents. The implementation of a case study as a human-robot collaborative workspace allows evaluating the performance of the human-robot team, the impact on the operator's cognitive abilities, and the level of collaboration achieved in the human-robot team through a set of metrics and proven methods in other areas, such as cognitive systems engineering, human-machine interaction, and ergonomics. We conclude by discussing the findings and outlook regarding future research questions and possible developments.La introducción de tecnologías disruptivas de Industria 4.0 en el lugar de trabajo integradas a través de sistemas ciberfísicos humanos hace que los operadores enfrenten nuevos desafíos. Estos se reflejan en el aumento de las demandas presentadas en las capacidades físicas, sensoriales y cognitivas del operador. En esta investigación, las demandas cognitivas son las más interesantes. En esta perspectiva, los asistentes se presentan como una posible solución, no como una herramienta sino como un conjunto de funciones que amplifican las capacidades humanas, como exoesqueletos, robots colaborativos para capacidades físicas, realidad virtual y aumentada para capacidades sensoriales. Quizás chatbots y softbots para capacidades cognitivas, entonces surge la necesidad de preguntarnos: ¿Cómo se pueden desarrollar los sistemas de asistencia al operador 4.0 en el contexto de la fabricación industrial? ¿En qué capacidades el operador necesita más asistencia? A partir del paradigma actual de sistematización, se utilizan diferentes enfoques dentro del contexto del espacio de trabajo en la industria 4.0. Así, se utiliza el método de análisis de resonancia funcional (FRAM) para modelar el espacio de trabajo desde el enfoque del sistema sociotécnico, donde las relaciones entre los componentes son las más importantes entre las funciones a desarrollar por el equipo humano-robot. Con el uso de simuladores tanto para robots como para sistemas robóticos se analiza el comportamiento de la variabilidad del equipo humano-robot. Además, desde la perspectiva de la ingeniería de sistemas cognitivos, el espacio de trabajo puede ser estudiado como un sistema cognitivo conjunto, donde la cognición se entiende distribuida, en una relación simbiótica entre los agentes humanos y tecnológicos. La implementación de un caso de estudio como un espacio de trabajo colaborativo humano-robot permite evaluar el desempeño del equipo humano-robot, el impacto en las habilidades cognitivas del operador y el nivel de colaboración alcanzado en el equipo humano-robot a través de un conjunto de métricas y métodos probados en otras áreas, como la ingeniería de sistemas cognitivos, la interacción hombre-máquina y la ergonomía. Concluimos discutiendo los hallazgos y las perspectivas con respecto a futuras preguntas de investigación y posibles desarrollos.Postprint (published version

Trusted Artificial Intelligence in Manufacturing; Trusted Artificial Intelligence in Manufacturing

The successful deployment of AI solutions in manufacturing environments hinges on their security, safety and reliability which becomes more challenging in settings where multiple AI systems (e.g., industrial robots, robotic cells, Deep Neural Networks (DNNs)) interact as atomic systems and with humans. To guarantee the safe and reliable operation of AI systems in the shopfloor, there is a need to address many challenges in the scope of complex, heterogeneous, dynamic and unpredictable environments. Specifically, data reliability, human machine interaction, security, transparency and explainability challenges need to be addressed at the same time. Recent advances in AI research (e.g., in deep neural networks security and explainable AI (XAI) systems), coupled with novel research outcomes in the formal specification and verification of AI systems provide a sound basis for safe and reliable AI deployments in production lines. Moreover, the legal and regulatory dimension of safe and reliable AI solutions in production lines must be considered as well. To address some of the above listed challenges, fifteen European Organizations collaborate in the scope of the STAR project, a research initiative funded by the European Commission in the scope of its H2020 program (Grant Agreement Number: 956573). STAR researches, develops, and validates novel technologies that enable AI systems to acquire knowledge in order to take timely and safe decisions in dynamic and unpredictable environments. Moreover, the project researches and delivers approaches that enable AI systems to confront sophisticated adversaries and to remain robust against security attacks. This book is co-authored by the STAR consortium members and provides a review of technologies, techniques and systems for trusted, ethical, and secure AI in manufacturing. The different chapters of the book cover systems and technologies for industrial data reliability, responsible and transparent artificial intelligence systems, human centered manufacturing systems such as human-centred digital twins, cyber-defence in AI systems, simulated reality systems, human robot collaboration systems, as well as automated mobile robots for manufacturing environments. A variety of cutting-edge AI technologies are employed by these systems including deep neural networks, reinforcement learning systems, and explainable artificial intelligence systems. Furthermore, relevant standards and applicable regulations are discussed. Beyond reviewing state of the art standards and technologies, the book illustrates how the STAR research goes beyond the state of the art, towards enabling and showcasing human-centred technologies in production lines. Emphasis is put on dynamic human in the loop scenarios, where ethical, transparent, and trusted AI systems co-exist with human workers. The book is made available as an open access publication, which could make it broadly and freely available to the AI and smart manufacturing communities