Aganian, Dustin

Eisenbach, Markus

Groß, Horst-Michael

Köhler, Mona

Schröter, Christof

Stephan, Benedict

German

Digitale Bibliothek Thüringen

TU Ilmenau | Universitätsbibliothek | ilmedia, 2022 http://www.tu-ilmenau.de/ilmedia Eisenbach, Markus; Aganian, Dustin; Köhler, Mona; Stephan, Benedict; Schröter, Christof; Groß, Horst-Michael This publication was produced within the framework of the event: IEEE International Conference on Automation Science and Engineering ; 17 (Lyon, France) : 2021.08.23-27 TuBT7 Special Session: Robotic Control and Robotization of Tasks within Industry 4.0 Date published: 2022-02-09 Visual Scene Understanding for Enabling Situation-Aware Cobots DOI: 10.22032/dbt.51471 URN:  urn:nbn:de:gbv:ilm1-2022200040 Visual Scene Understanding for Enabling Situation-Aware CobotsMarkus Eisenbach, Dustin Aganian, Mona Köhler, Benedict Stephan,Christof Schröter and Horst-Michael GrossAbstract— Although in the course of Industry 4.0, a highdegree of automation is the objective, not every process canbe fully automated – especially in versatile manufacturing. Inthese applications, collaborative robots (cobots) as helpers area promising direction. We analyze the collaborative assemblyscenario and conclude that visual scene understanding is aprerequisite to enable autonomous decisions by cobots. Weidentify the open challenges in these visual recognition tasksand propose promising new ideas on how to overcome them.I. INTRODUCTIONFor many small and medium-sized enterprises (SMEs),large and highly specialized production lines are not flexibleenough to justify their high investment. Instead, these SMEscan generate their profits with versatile assembly in smallquantities [1]. In this scenario, full automation is rarelyachieved and some complex assembly sequences can beexecuted faster and cheaper by a worker. To optimize theprocess in the course of Industry 4.0, robots can executesome simple assembly and transport tasks, which may re-quire close interaction with humans. This can speed up theassembly process, not through high automation, but throughteamwork and parallelization.However, robots and humans usually work in separateareas. If a human gets close to a robot, the robot stopsimmediately to avoid injury. Furthermore, robots only per-form tasks in which every production step and every objectinvolved is known, which is a severe limitation in flexibleproduction. Therefore, the goal of the next stage in Industry4.0 is the cooperation between humans and situation-awarecollaborative robots (cobots) [2]. To enable situation-awaredecisions of the cobot, visual scene understanding plays akey role, which is the focus of our paper.II. COLLABORATIVE ASSEMBLY SCENARIOTo further define the context in which our cobot has towork, in the following, we present our collaborative assemblyscenario. Our cobot has access to multiple cameras observingthe workspace. After observing the assembly process a fewtimes, the cobot should be able to anticipate when tools orworkpieces are needed (see Fig. 1). The cobot should act asThis work has received funding from the Carl Zeiss Foundation as partof the project Engineering for Smart Manufacturing (E4SM) – Engineeringof machine-learning-based assistance systems for data-intensive industrialscenarios under grant agreement no. P2017-01-005C. Schröter is with MetraLabs GmbH, 98693 Ilmenau, Germanywww.metralabs.comM. Eisenbach, D. Aganian, M. Köhler, B. Stephan and H.-M. Gross arewith Neuroinformatics and Cognitive Robotics Lab, Ilmenau University ofTechnology, 98693 Ilmenau, Germany www.tu-ilmenau.de/neurobIn the next assembly step, the worker needs to attach the foot with screws.Therefore, I should hand over the screws and hold the piece to be attached.Fig. 1. The cobot is assisting the worker at our prototypical workplacein our vision lab. To be able to autonomously plan helpful actions, thecobot must understand the assembly process, and thus must analyze itsvisual input. This includes detecting objects, recognizing human actions,estimating 3D shapes, and predicting adequate grasp poses.a helper, performing transport tasks between warehouse andworkplace and handing over tools and objects as needed.Obviously, for this scenario, the cobot must master manytasks, such as navigation and actuator control. But besidesthese complex robotic tasks, first of all, the cobot mustrecognize objects and people in the scene and understandthe assembly process. The cobot must know what the currentassembly state is and what it can do to help the worker.III. VISUAL SCENE UNDERSTANDINGRecognizing the current assembly state is especially chal-lenging because it requires robust object detection and humanaction recognition. This should also be achieved even ifcertain objects or human actions are very similar or notpresent in training data. But scene understanding for manip-ulation is also extremely difficult, as it requires adequate 3Dshape and grasp pose estimation combined with uncertaintyestimation to avoid accidents. Therefore, in order to makethe robot aware of the situation and let it act as a helper,initially, scene analysis should be the main focus and allsubtasks should be addressed by powerful machine learningapproaches. When applying state-of-the-art approaches forvisual scene understanding, certain shortcomings must beaddressed, as described below.A. Object DetectionAs occurring objects vary, extensive scenario-specific la-beling of training data is impractical. A promising directionin our scenario is few-shot object detection [3]. It leverages2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)August 23-27, 2021. Lyon, FranceCopyright ©2021 Authorsknowledge from training on base categories with abundanttraining data in order to detect objects from novel categorieswith only a few labeled instances per category. Its applicationin an industrial setting has yet to be implemented.As another way of recognizing objects, which has not beenexplored yet, we propose to apply a class-agnostic objectdetector [4], which aims to detect objects irrespective of theirsemantic class. The resulting detections can then be fed intoa module for re-identifying previously seen objects.B. Human Action RecognitionTypically, in action detection and recognition, actions areclassified only on already known classes. Although there areseveral hundred different classes present in current datasets,e.g. [5], only few of them are relevant for industrial assem-bling. Furthermore, the few available assembly datasets, suchas [6], are rather small.However, in advance it is unclear which actions a personwill perform when building a new workpiece and post-labeling and post-training of the performed actions is im-practical. Therefore, as a novel idea, we formulate this actionrecognition as a re-identification problem. By recognizing ac-tions from previous assembly processes, the current assemblystate can be identified.To enable action re-identification, we prefer to use skele-tons rather than RGB images as they generalize better fordifferent persons and environments. Since some assemblyactions can only be distinguished by the movement of thehands, we extend the full body skeleton with estimatedhand skeletons. Moreover, tables or the workpiece itselflead to occlusion of skeleton joints. Therefore, we makeuse of multiple cameras around the workplace to predictcomplementary skeletons.C. 3D Shape EstimationModel-based methods for grasp pose estimation expecta 3D shape description. However, in general training datafor 3D shape estimation [7] are synthetic and often ren-dered without textures and background. Even if textures andbackground were present, widely used methods generalizebadly to new object categories [8]. Therefore, we propose toestimate uncertainties [9] and adapt these predictors by post-training on the most uncertain objects in order to specializeto scenario-specific objects.D. Grasp Pose EstimationGrasp Pose Estimation either expects 3D shapes of theobject (see Sec. III-C) or processes depth images directly.However, for the latter methods, common datasets, suchas [10], contain images with only few labeled grasp poses.Yet, providing an exhaustive amount of labeled grasp posesis impossible due to an infinite amount of possible graspposes. This may result in possible contradictions in thedataset where a similar input is labeled with different graspsposes. A neural network trained on these contradictionswould average these poses resulting in possibly unfeasible orunstable grasps poses. Current work [11] only partially solvesthis problem. We propose a more promising way to resolveexisting contradictions by estimating the aleatoric uncertainty(a.k.a. data uncertainty). This way, we can either rejectestimates with high uncertainty or incorporate the uncertainestimates into the training process of the self-learning cobot.E. Uncertainty EstimationTo avoid accidents, uncertainties should be estimated [9]and handled for all visual scene understanding tasks. Thisincludes uncertainties in the detected positions of objects andthe worker, uncertainties in features for re-identification ofassembly actions, uncertainties in 3D object shape estimates,and uncertainties in estimations of grasp poses. Unfortu-nately, methods for robust uncertainty estimation in deeplearning models are still in their infancy, but are essentialfor a practical implementation of safe cobots. Therefore,transferring uncertainty estimation for classification tasks todetection, grasp pose estimation, etc., will also be in thefocus of our future work.IV. CONCLUSIONIn the context of Industry 4.0, cobots as helpers becomemore relevant, especially in versatile manufacturing. Sincethe cobot should act as an autonomous helper, we proposeto focus on visual scene understanding as an important firststep. We identified several challenges when applying currentstate-of-the-art methods to an industrial setting. To overcomethese challenges, we propose promising new directions:• As occurring objects vary in versatile manufacturing,we propose few-shot learning, which allows detectionwith little training data.• To cope with new objects and human actions, whichare unknown during training, we propose to formulateobject detection and human action recognition as a re-identification problem.• For a more generalizing action recognition to varyingpersons and environments we propose to use multi-viewfull body and hand skeletons instead of image data.• To combat generalization deficits in 3D shape estima-tion and contradictions when training grasp pose detec-tion, we propose to estimate and handle uncertainties.REFERENCES[1] A. Radziwon et al., “The smart factory: Exploring adaptive and flexiblemanufacturing solutions,” Procedia Engineering, 2014.[2] M. Strehlitz, “Tempolimit für Roboter,” VDE dialog, 2018.[3] Q. Fan et al., “Few-shot object detection with attention-RPN andmulti-relation detector,” in CVPR, 2020.[4] A. Jaiswal et al., “Class-agnostic object detection,” in WACV, 2021.[5] L. Smaira et al., “A short note on the kinetics-700-2020 human actiondataset,” arXiv, 2020.[6] Y. Ben-Shabat et al., “The IKEA ASM dataset: Understanding peopleassembling furniture through actions, objects and pose,” in WACV,2021.[7] N. Wang et al., “Pixel2Mesh: Generating 3D mesh models from singleRGB images,” in ECCV, 2018.[8] M. Tatarchenko et al., “What do single-view 3D reconstruction net-works learn?” in CVPR, 2019.[9] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation:Representing model uncertainty in deep learning,” in ICML, 2016.[10] “Cornell dataset,” http://pr.cs.cornell.edu/grasping/rectdata/data.php.[11] D. Morrison et al., “Closing the loop for robotic grasping: A real-time,generative grasp synthesis approach,” arXiv, 2018.

Visual Scene Understanding for Enabling Situation-Aware Cobots

https://www.db-thueringen.de/servlets/MCRFileNodeServlet/dbt_derivate_00055707/ilm1-2022200040.pdf

Visual Scene Understanding for Enabling Situation-Aware Cobots

Abstract

Similar works

Full text

Available Versions

Digitale Bibliothek Thüringen