Horváth, Gergely

Kardos, Csaba

Kemény, Zsolt

Kovács, András

Pataki, Balázs

Váncza, József

Horváth, G.

Kardos, Cs.

Kemény, Zs.

Kovács, A.

Pataki, B.E.

Váncza, J.

Repository of the Academy's Library

Multi-Modal Interfaces for Human–Robot Communication in Collaborative Assembly  by Gergely Horváth, Csaba Kardos, Zsolt Kemény, András Kovács, Balázs E. Pataki and József Váncza Human–robot collaboration (HRC) in production, especially in assembly, has the potential to improve flexibility and competitiveness. However, there are still numerous challenges to address before HRC can be realised. Beyond the essential problem of safety, the efficient sharing of work and workspace between human and robot also requires new interfaces for communication. The SYMBIO-TIC H2020 project is developing a dynamic context-aware and bi-directional, multi-modal communication system that supports human operators in collaborative assembly.  Introduction The main goal of the SYMBIO-TIC H2020 project is to provide a safe, dynamic, intuitive and cost effective working environment, hosting immersive and symbiotic collaboration between human workers and robots [1]. In such a dynamic environment, a key to boosting the efficiency of human workers is supporting them with context-dependent work instructions, delivered via communication modalities that suit the actual context. Workers, in turn, should be able to control the robot or other components of the production system by using the most convenient modality, thus lifting the limitations of traditional interfaces such as push buttons installed at fixed locations. As part of the SYMIBIO-TIC project, we are developing a system that addresses these needs. Context-awareness in human–robot collaboration To harness the flexibility of an HRC production environment, it is essential that the worker assistance system delivers information to the human worker that suits the actual context of production. In order to gather the information describing the context, data related to both the worker (individual properties, location, activity) and to the process under execution is required. This information is provided to the worker assistance system by three connected systems, which together form the HRC ecosystem, namely (1) the workcell-level task execution and control (unit controller, UC), (2) the shopfloor-level scheduling (cockpit), and (3) the mobile worker identification (MWI) systems [2]. The process execution context is defined by the state of the task execution in the UC. The identification and location of the worker by the MWI is essential in order to trigger the worker assistance system and to properly utilise the devices around the worker. Actions of the worker have to be captured either directly by devices available to the worker, or the sensors deployed in the workcell, registering the worker’s activity context. The properties of the worker, such as skills or preferences of instruction delivery, define the final format of the instructions delivered. Automatically generated work instructions The complexity of managing work instructions in production environments characterised by shortening product life cycles and increasing product variety, as well as the requirement to fully exploit the available context data in customised instructions, calls for the automated generation of human work instructions. A method for this, relying on a feature-based representation of the assembly process [3], computer-aided process planning (CAPP) techniques, and a hierarchical template for the instructions, has been proposed in [4][3]. The method generates textual work instructions (with potential audio interpretation using text-to-speech tools) and X3D animations of the process tailored to the skill level and the language preferences of the individual worker. The presentation of the instruction can be further customised in real time by the instruction delivery system: e.g., the selection of the modality and the device, as well as the font size and the sound volume, can be adjusted according to the current context. Multi-modal communication Traditionally, worker assistance is provided by visual media, mostly in the form of text or images. The currently prevailing digital assistance systems hence focus on delivering visual instructions to the workers. However, in a HRC setting, it is also necessary to provide bi-directional interfaces that allow the workers to control the robots and other equipment participating in the production process. The worker assistance system that we have developed is designed to deliver various forms of visual work instructions, such as text, images, static and animated 3D content and videos. Audio instructions are also supported: by using text-to-speech software, the textual instructions can be transformed as well. Instruction delivery is implemented as an HTML5 webpage, which supports embedding multi-media content and also allows multiple devices to be used for both visual and audio content, such as smartphones, AR-glasses, computer screens, or tablets. Our web-based solution for input interfaces provides the classic button-like input channels, which are still required in most industrial scenarios. Potentially promising contactless technologies are also integrated into the system. Interpreting audio commands shows great potential as it is not only contactless, but also hands-free. However, in a noisy industrial environment, it could be challenging and therefore the application of two hand gesture-based technologies is also supported, one using point-cloud data registered by depth cameras and the other using a special interpreter glove that measures the relative displacement of the hand and fingers.  Figure 1: Schematic architecture of the HMIC implementation and its immediate environment in the production system. Implementation and use case A complete server–client-based solution for the human–machine interface system was implemented in accordance with the aforementioned requirements and technologies. The system is named Human Machine Interface Controller (HMIC). Figure 1 shows its major structure (backend/frontend design) and its connections to other elements of the ecosystem. The implemented HMIC system was successfully demonstrated in the laboratory simulation of an automotive assembly use case, where 29 parts were assembled in 19 tasks (see Figure 2). The research project is now in its closing phase, where the focus is on the development of demonstrator case studies and the evaluation of the perceived work experience with the use of the generated content and the multi-modal content delivery system.   Figure 2: Demonstration of an automotive assembly case study using the HMIC system. The devices available for the user are a large touchscreen, a smartphone and an AR-glass.  Acknowledgement: This research has been supported by the EU H2020 Grant SYMBIO-TIC No. 637107 and by the GINOP-2.3.2-15-2016-00002 grant on an “Industry 4.0 research and innovation center of excellence”. References: [1] SYMBIO-TIC EU H2020 Project, http://www.symbio-tic.eu [2] Kardos, Cs.; Kemény, Zs.; Kovács, A.; Pataki, B.E.; Váncza, J.: Context-dependent multimodal communication in human–robot collaboration. 51st CIRP International Conference on Manufacturing Systems, 2018. [3] Kardos, Cs.; Kovács, A.; Váncza, J.: Decomposition approach to optimal feature-based assembly planning. CIRP Annals – Manufacturing Technology, 66(1):417-420, 2017. [4] Kardos, Cs.; Kovács, A.; Pataki, B.E.; Váncza, J.: Generating human work instructions from assembly plans. 2nd ICAPS Workshop on User Interfaces and Scheduling and Planning (UISP2018), Delft, The Netherlands, 2018. Please contact: Csaba Kardos MTA SZTAKI: Institute for Computer Science and Control, Hungarian Academy of Sciences + 36 1 279 6189 csaba.kardos@sztaki.mta.hu 

Multi-modal interfaces for human–robot communication in collaborative assembly

SZTAKI Publication Repository

ERCIM NEWSNumber 114 July 2018www.ercim.euSpecial themeHuman-Robot Interaction Also in this issue:Research and Innovation: Corpus Conversion Service: A Machine Learning Platform to Ingest Documents at ScaleERCIM NEWS 114   July 2018 15The main goal of the SYMBIO-TICH2020 project is to provide a safe,dynamic, intuitive and cost effectiveworking environment, hosting immer-sive and symbiotic collaborationbetween human workers and robots[L1]. In such a dynamic environment, akey to boosting the efficiency of humanworkers is supporting them with con-text-dependent work instructions, deliv-ered via communication modalities thatsuit the actual context. Workers, in turn,should be able to control the robot orother components of the productionsystem by using the most convenientmodality, thus lifting the limitations oftraditional interfaces such as push but-tons installed at fixed locations. As partof the SYMIBIO-TIC project, we aredeveloping a system that addressesthese needs.Context-awareness in human-robotcollaborationTo harness the flexibility of an HRCproduction environment, it is essentialthat the worker assistance systemdelivers information to the humanworker that suits the actual context ofproduction. In order to gather the infor-mation describing the context, datarelated to both the worker (individualproperties, location, activity) and to theprocess under execution is required.This information is provided to theworker assistance system by three con-nected systems, which together formthe HRC ecosystem, namely (1) theworkcell-level task execution and con-trol (unit controller, UC), (2) theshopfloor-level scheduling (cockpit),and (3) the mobile worker identification(MWI) systems [1].The process execution context isdefined by the state of the task execu-tion in the UC. The identification andlocation of the worker by the MWI isessential in order to trigger the workerassistance system and to properly utilisethe devices around the worker. Actionsof the worker have to be captured eitherdirectly by devices available to theworker, or the sensors deployed in theMulti-Modal Interfaces for Human–Robot Communication in Collaborative Assemblyby Gergely Horváth, Csaba Kardos, Zsolt Kemény, András Kovács, Balázs E. Pataki and József Váncza (MTA SZTAKI)Human–Robot Collaboration (HRC) in production—especially in assembly— offers, on one hand, flexibility and asolution for maintaining competitiveness. On the other hand, there are still numerous challenges that have to beanswered to allow the realization of HRC. Beyond the essential problems of safety, the efficient sharing of workand workspace between human and robot requires new interfaces for communication as well. As a part of theSYMBIO-TIC H2020 project, a dynamic context-aware and bi-directional, multi-modal communication system isintroduced and implemented for supporting human operators in collaborative assembly. Figure 1. Schematic architecture of the HMIC implementation and its immediate environment in the production system.ERCIM NEWS 114   July 201816Special Theme: Human-Robot Interactionworkcell, registering the worker ’sactivity context. The properties of theworker, such as skills or preferences ofinstruction delivery, define the finalformat of the instructions delivered.Automatically generated workinstructionsThe complexity of managing workinstructions in production environmentscharacterised by shortening product lifecycles and increasing product variety, aswell as the requirement to fully exploitthe available context data in customisedinstructions, calls for the automatedgeneration of human work instructions.A method for this, relying on a feature-based representation of the assemblyprocess [2], computer-aided processplanning (CAPP) techniques, and ahierarchical template for the instruc-tions, has been proposed in [3]. Themethod generates textual work instruc-tions (with potential audio interpreta-tion using text-to-speech tools) andX3D animations of the process tailoredto the skill level and the language pref-erences of the individual worker. Thepresentation of the instruction can befurther customised in real time by theinstruction delivery system: e.g., theselection of the modality and the device,as well as the font size and the soundvolume, can be adjusted according tothe current context.Multi-modal communicationTraditionally, worker assistance is pro-vided by visual media, mostly in theform of text or images. The currentlyprevailing digital assistance systemshence focus on delivering visualinstructions to the workers. However, ina HRC setting, it is also necessary toprovide bi-directional interfaces thatallow the workers to control the robotsand other equipment participating in theproduction process.The worker assistance system that wehave developed is designed to delivervarious forms of visual work instruc-tions, such as text, images, static andanimated 3D content and videos. Audioinstructions are also supported: by usingtext-to-speech software, the textualinstructions can be transformed as well.Instruction delivery is implemented asan HTML5 webpage, which supportsembedding multi-media content andalso allows multiple devices to be usedfor both visual and audio content, suchas smartphones, AR-glasses, computerscreens, or tablets.Our web-based solution for input inter-faces provide the classic button-likeinput channels, which are still required inmost industrial scenarios. Potentiallypromising contactless technologies arealso integrated into the system.Interpreting audio commands showsgreat potential as it is not only contact-less, but also hands-free. However, in anoisy industrial environment, it could bechallenging and therefore the applicationof two hand gesture-based technologiesis also supported, one using point-clouddata registered by depth cameras and theother using a special interpreter glovethat measures the relative displacementof the hand and fingers.Implementation and use caseA complete server–client-based solu-tion for the human–machine interfacesystem was implemented in accordancewith the aforementioned requirementsand technologies. The system is namedHuman Machine Interface Controller(HMIC). Figure 1 shows its major struc-ture (backend/frontend design) and itsconnections to other elements of theecosystem. The implemented HMICsystem was successfully demonstratedin the laboratory simulation of an auto-motive assembly use case, where 29parts were assembled in 19 tasks (seeFigure 2). The research project is nowin its closing phase, where the focus ison the development of demonstratorcase studies and the evaluation of theperceived work experience with the useof the generated content and the multi-modal content delivery system.This research has been supported by theEU H2020 Grant SYMBIO-TIC No.637107 and by the GINOP-2.3.2-15-2016-00002 grant on an “Industry 4.0research and innovation center of excel-lence”.Link:[L1] http://www.symbio-tic.euReferences:[1] Cs. Kardos, et al.: “Context-dependent multimodalcommunication in human–robotcollaboration”, 51st CIRPInternational Conference onManufacturing Systems, 2018.[2] Cs. Kardos, A. Kovács, J. Váncza:“Decomposition approach tooptimal feature-based assemblyplanning”, CIRP Annals –Manufacturing Technology,66(1):417-420, 2017.[3] Cs. Kardos, A. Kovács, A.; B.E.Pataki, J. Váncza: “Generatinghuman work instructions fromassembly plans”, 2nd ICAPSWorkshop on User Interfaces andScheduling and Planning(UISP2018), 2018.Please contact:Csaba KardosMTA SZTAKI: Institute for ComputerScience and Control, HungarianAcademy of Sciences+ 36 1 279 6189csaba.kardos@sztaki.mta.huFigure 2. Demonstration of an automotive assembly case study using the HMIC system. Thedevices available for the user are a large touchscreen, a smartphone and an AR-glass. 

Multi-Modal Interfaces for Human–Robot Communication in Collaborative Assembly

http://eprints.sztaki.hu/9619/1/Horvath_15_3394732_ny.pdf

Multi-Modal Interfaces for Human–Robot Communication in Collaborative Assembly

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library

SZTAKI Publication Repository