Visuo-tactile pose tracking method for in-hand robot manipulation tasks of quotidian objects

International audienceAfter more than three decades of research in robot manipulation problems, we observed a considerable level of maturity in different related problems. Many high-performant objects pose tracking exists, one of the main problems for these methods is the robustness again occlusion during in-hand manipulation. This work presents a new multimodal perception approach in order to estimate the pose of an object during an in-hand manipulation. Here, we propose a novel learning-based approach to recover the pose of an object in hand by using a regression method. Particularly, we fuse the visual-based tactile information and depth visual information in order to overpass occlusion problems commonly presented during robot manipulation tasks. Our method is trained and evaluated using simulation. We compare the proposed method against different state-of-the-art approaches to show its robustness in hard scenarios. The recovered results show a reliable increment in performance, while they are obtained using a benchmark in order to obtain replicable and comparable results

Multimodal Neural Radiance Field for In-Hand Robot Manipulation Tasks of Transparent Objects

International audienceIn recent years, there has been significant progress in robot manipulation research, but challenges persist when it comes to tracking transparent objects using RGB-D sensors. To address this, polarimetric imaging technology has gained popularity due to cost-effective sensors becoming more accessible. This technology, which analyzes the polarization properties of light, offers valuable insights into object properties, aids in material differentiation, depth estimation, and precise pose estimation, even in challenging lighting conditions. A novel multimodal perception approach is introduced in this study, utilizing a learning-based neural radiance field method based on multimodal data to overcome the limitations of RGB-D imaging when dealing with transparent objects during in-hand manipulation (Kerr et al.