Augmented Virtuality Data Annotation and Human-in-the-Loop Refinement for RGBD Data in Industrial Bin-Picking Scenarios

Beyond conventional automated tasks, autonomous robot capabilities aside to human cognitive skills are gaining importance. This comprises goods commissioning and material supply in intralogistics as well as material feeding and assembly operations in production. Deep learning-based computer vision is considered as enabler for autonomy. Currently, the effort to generate specific datasets is challenging. Adaptation of new components often also results in downtimes. The objective of this paper is to propose an augmented virtuality (AV) based RGBD data annotation and refinement method. The approach reduces required effort in initial dataset generation to enable prior system commissioning and enables dataset quality improvement up to operational readiness during ramp-up. In addition, remote fault intervention through a teleoperation interface is provided to increase operational system availability. Several components within a real-world experimental bin-picking setup serve for evaluation. The results are quantified by comparison to established annotation methods and through known evaluation metrics for pose estimation in bin-picking scenarios. The results enable to derive accurate and more time-efficient data annotation for different algorithms. The AV approach shows a noticeable reduction in required effort and timespan for annotation as well as dataset refinement