3D Semantic Scene Reconstruction from a Single Viewport

We introduce a novel method for semantic volumetric reconstructions from a single RGB image. To overcome the problem of semantically reconstructing regions in 3D that are occluded in the 2D image, we propose to combine both in an implicit encoding. By relying on a headless autoencoder, we are able to encode semantic categories and implicit TSDF values into a compressed latent representation. A second network then uses these as a reconstruction target and learns to convert color images into these latent representations, which get decoded after inference. Additionally, we introduce a novel loss-shaping technique for this implicit representation. In our experiments on the realistic benchmark Replica dataset, we achieve a full reconstruction of a scene, which is visually and in terms of quantitative measures better than current methods while only using synthetic data during training. On top of that, we evaluate our approach on color images recorded in the wild

Learning to Localize in New Environments from Synthetic Training Data

Most existing approaches for visual localization either need a detailed 3D model of the environment or, in the case of learning-based methods, must be retrained for each new scene. This can either be very expensive or simply impossible for large, unknown environments, for example in search-and-rescue scenarios. Although there are learning-based approaches that operate scene-agnostically, the generalization capability of these methods is still outperformed by classical approaches. In this paper, we present an approach that can generalize to new scenes by applying specific changes to the model architecture, including an extended regression part, the use of hierarchical correlation layers, and the exploitation of scale and uncertainty information. Our approach outperforms the 5-point algorithm using SIFT features on equally big images and additionally surpasses all previous learning-based approaches that were trained on different data. It is also superior to most of the approaches that were specifically trained on the respective scenes. We also evaluate our approach in a scenario where only very few reference images are available, showing that under such more realistic conditions our learning-based approach considerably exceeds both existing learning-based and classical methods

RECALL: Rehearsal-free Continual Learning for Object Classification

Convolutional neural networks show remarkable results in classification but struggle with learning new things on the fly. We present a novel rehearsal-free approach, where a deep neural network is continually learning new unseen object categories without saving any data of prior sequences. Our approach is called RECALL, as the network recalls categories by calculating logits for old categories before training new ones. These are then used during training to avoid changing the old categories. For each new sequence, a new head is added to accommodate the new categories. To mitigate forgetting, we present a regularization strategy where we replace the classification with a regression. Moreover, for the known categories, we propose a Mahalanobis loss that includes the variances to account for the changing densities between known and unknown categories. Finally, we present a novel dataset for continual learning, especially suited for object recognition on a mobile robot (HOWS-CL-25), including 150,795 synthetic images of 25 household object categories. Our approach RECALL outperforms the current state of the art on CORe50 and iCIFAR-100 and reaches the best performance on HOWS-CL-25

BlenderProc2: A Procedural Pipeline for Photorealistic Rendering

BlenderProc2 is a procedural pipeline that can render realistic images for the training of neural networks. Our pipeline can be employed in various use cases, including segmentation, depth, normal and pose estimation, and many others. A key feature of our Blender extension is the simple-to-use python API, designed to be easily extendable. Furthermore, many public datasets, such as 3D FRONT (Fu et al., 2021) or Shapenet (Chang et al., 2015), are already supported, making it easier to clutter synthetic scenes with additional objects

BlenderProc: Reducing the Reality Gap with Photorealistic Rendering

BlenderProc is an open-source and modular pipeline for rendering photorealistic images of procedurally generated 3D scenes which can be used for training data-hungry deep learning models. The presented results on the tasks of instance segmentation and surface normal estimation suggest that our photorealistic training images reduce the gap between the synthetic training and real test domains, compared to less realistic training images combined with domain randomization. BlenderProc can be used to train models for various computer vision tasks such as semantic segmentation or estimation of depth, optical flow, and object pose. By offering standard modules for parameterizing and sampling materials, objects, cameras and lights, BlenderProc can simulate various real-world scenarios and provide means to systematically investigate the essential factors for sim2real transfer

Extending the Knowledge Driven Approach for Scalable Autonomy Teleoperation of a Robotic Avatar

Crewed missions to celestial bodies such as Moon and Mars are in the focus of an increasing number of space agencies. Precautions to ensure a safe landing of the crew on the extraterrestrial surface, as well as reliable infrastructure on the remote location, for bringing the crew back home are key considerations for mission planning. The European Space Agency (ESA) identified in its Terrae Novae 2030+ roadmap, that robots are needed as precursors and scouts to ensure the success of such missions. An important role these robots will play, is the support of the astronaut crew in orbit to carry out scientific work, and ultimately ensuring nominal operation of the support infrastructure for astronauts on the surface. The METERON SUPVIS Justin ISS experiments demonstrated that supervised autonomy robot command can be used for executing inspection, maintenance and installation tasks using a robotic co-worker on the planetary surface. The knowledge driven approach utilized in the experiments only reached its limits when situations arise that were not anticipated by the mission design. In deep space scenarios, the astronauts must be able to overcome these limitations. An approach towards more direct command of a robot was demonstrated in the METERON ANALOG-1 ISS experiment. In this technical demonstration, an astronaut used haptic telepresence to command a robotic avatar on the surface to execute sampling tasks. In this work, we propose a system that combines supervised autonomy and telepresence by extending the knowledge driven approach. The knowledge management is based on organizing the prior knowledge of the robot in an object-centered context. Action Templates are used to define the knowledge on the handling of the objects on a symbolic and geometric level. This robot-agnostic system can be used for supervisory command of any robotic coworker. By integrating the robot itself as an object into the object-centered domain, robot-specific skills and (tele-)operation modes can be injected into the existing knowledge management system by formulating respective Action Templates. In order to efficiently use advanced teleoperation modes, such as haptic telepresence, a variety of input devices are integrated into the proposed system. This work shows how the integration of these devices is realized in a way that is agnostic to the input devices and operation modes. The proposed system is evaluated in the Surface Avatar ISS experiment. This work shows how the system is integrated into a Robot Command Terminal featuring a 3-Degree-of-Freedom Joystick and a 7-Degree-of-Freedom haptic input device in the Columbus module of the ISS. In the preliminary experiment sessions of Surface Avatar, two astronauts on orbit took command of the humanoid service robot Rollin' Justin in Germany. This work presents and discusses the results of these ISS-to-ground sessions and derives requirements for extending the scalable autonomy system for the use with a heterogeneous robotic team

Introduction to Surface Avatar: the First Heterogeneous Robotic Team to be Commanded with Scalable Autonomy from the ISS

Robotics is vital to the continued development toward Lunar and Martian exploration, in-situ resource utilization, and surface infrastructure construction. Large-scale extra-terrestrial missions will require teams of robots with different, complementary capabilities, together with a powerful, intuitive user interface for effective commanding. We introduce Surface Avatar, the newest ISS-to-Earth telerobotic experiment series, to be conducted in 2022-2024. Spearheaded by DLR, together with ESA, Surface Avatar builds on expertise on commanding robots with different levels of autonomy from our past telerobotic experiments: Kontur-2, Haptics, Interact, SUPVIS Justin, and Analog-1. A team of four heterogeneous robots in a multi-site analog environment at DLR are at the command of a crew member on the ISS. The team has a humanoid robot for dexterous object handling, construction and maintenance; a rover for long traverses and sample acquisition; a quadrupedal robot for scouting and exploring difficult terrains; and a lander with robotic arm for component delivery and sample stowage. The crew's command terminal is multimodal, with an intuitive graphical user interface, 3-DOF joystick, and 7-DOF input device with force-feedback. The autonomy of any robot can be scaled up and down depending on the task and the astronaut's preference: acting as an avatar of the crew in haptically-coupled telepresence, or receiving task-level commands like an intelligent co-worker. Through crew performing collaborative tasks in exploration and construction scenarios, we hope to gain insight into how to optimally command robots in a future space mission. This paper presents findings from the first preliminary session in June 2022, and discusses the way forward in the planned experiment sessions

An efficient probabilistic online classification approach for object recognition with random forests

Online learning on big data sets is still an open problem in the classification of images. Many problems in the real world don't have all data available in the beginning of the training. Therefore it is necessary that the approach is able to integrate new incoming datapoints. Random Forest have been proven to be good in online learning. However the existing approaches do only generate very few trees, which only have a height of five. To overcome this shortcoming this thesis presents several methods to improve the generation of Decision trees, which leads to an algorithm, which can train thousands of tree with a sufficient height. Furthermore the Random Forest were then used in combination with an online sparse Gaussian Process to classify the outliners. These falsely classified points weren't classified correctly by the Random Forest in the first place. This whole approach was then optimized and tested on different datasets. The far most important result was that the presented online approach always yields better results than the offline approach, which is a remarkable result for an online learning approach. Furthermore we outperformed the result from Saffari et al. on the USPS dataset

An exemplary application of decision theory in robotics: autonomous calibration of depth cameras with non-overlapping fields of view on a mobile platform

This thesis is about the creation of a decider, which is able to evaluate decision trees and utility tables. The decider enables the robot to decide by its own, which action should be performed in the actual situation. Afterwards the decider is used for the calibration process of time of flight cameras with non-overlapping field of views on a mobile platform. A framework is the main objective of this thesis, which can be utilized for known problems as Localization and Exploration in mobile robotics. Therefore, clear interfaces will be designed, to ensure that the framework will be used in future works. After the implementation of the decider a decision tree is designed containing the calibration’s process. This tree must have clearly specified actions and states, which can be observed by the robot itself. At the end the combination of the decision process with the calibration will be evaluated