Real-time planar segmentation of depth images: from three-dimensional edges to segmented planes

Abstract. Real-time execution of processing algorithms for handling depth images in a three-dimensional (3-D) data framework is a major challenge. More specifically, considering depth images as point clouds and performing planar segmentation requires heavy computation, because available planar segmentation algorithms are mostly based on surface normals and/or curvatures, and, consequently, do not provide real-time performance. Aiming at the reconstruction of indoor environments, the spaces mainly consist of planar surfaces, so that a possible 3-D application would strongly benefit from a real-time algorithm. We introduce a real-time planar segmentation method for depth images avoiding any surface normal calculation. First, we detect 3-D edges in a depth image and generate line segments between the identified edges. Second, we fuse all the points on each pair of intersecting line segments into a plane candidate. Third and finally, we implement a validation phase to select planes from the candidates. Furthermore, various enhancements are applied to improve the segmentation quality. The GPU implementation of the proposed algorithm segments depth images into planes at the rate of 58 fps. Our pipeline-interleaving technique increases this rate up to 100 fps. With this throughput rate improvement, the application benefit of our algorithm may be further exploited in terms of quality and enhancing the localization

Camera-to-model back-raycasting for extraction of RGB-D images from pointclouds

Conventional raycasting methods extract 2D-images from pointclouds in two main steps. The pointcloud is voxelized and then, rays are casted from a virtual-camera center towards the model. The value for each pixel in the resulting image is calculated based on the closest non-empty voxel intersected with the corresponding ray. Both voxelizing and such raycasting limit the quality (resolution) of the extracted image and impose high memory demands. In this paper, we propose an alternative backraycasting method, where rays are casted from the model towards the virtual-camera center and intersecting an image plane. This does not require any voxel grid to be generated. Moreover, this method allows to obtain images with any required resolution with all the points involved. Besides this, a neighbours-consistency technique is introduced to enhance the resulting image quality. The proposed method has been evaluated based on several criteria and for various resolutions. Evaluation results show that the proposed method compared to the conventional approach executes upto 49 times faster and improves PSNR and SSIM metrics for the resulting images by 26% and 12%; respectively. This improvement is beneficial for such domains as feature matching, edge detection, OCR and calibration. To enable researchers generating the same results and extend this work, the dataset and implementation codes are publicly available [1]

R\u3csup\u3e3\u3c/sup\u3eP:real-time RGB-D registration pipeline

\u3cp\u3eApplications based on colored 3-D data sequences suffer from lack of efficient algorithms for transformation estimation and key points extraction to perform accurate registration and sensor localization either in the 2-D or 3-D domain. Therefore, we propose a real-time RGB-D registration pipeline, named R\u3csup\u3e3\u3c/sup\u3eP, presented in processing layers. In this paper, we present an evaluation of several algorithm combinations for each layer, to optimize the registration and sensor localization for specific applications. The resulting dynamic reconfigurability of R\u3csup\u3e3\u3c/sup\u3eP makes it suitable as a front-end system for any SLAM reconstruction algorithm. Evaluation results on several public datasets reveal that R\u3csup\u3e3\u3c/sup\u3eP delivers real-time registration with 59 fps and high accuracy with the relative pose error (for a time span of 40 frames) for rotation and translation of 0.5� and 8 mm, respectively. All the heterogeneous dataset and implementations are publicly available under an open-source license [21].\u3c/p\u3

Camera-to-model back-raycasting for extraction of RGB-D images from pointclouds

\u3cp\u3eConventional raycasting methods extract 2D-images from pointclouds in two main steps. The pointcloud is voxelized and then, rays are casted from a virtual-camera center towards the model. The value for each pixel in the resulting image is calculated based on the closest non-empty voxel intersected with the corresponding ray. Both voxelizing and such raycasting limit the quality (resolution) of the extracted image and impose high memory demands. In this paper, we propose an alternative backraycasting method, where rays are casted from the model towards the virtual-camera center and intersecting an image plane. This does not require any voxel grid to be generated. Moreover, this method allows to obtain images with any required resolution with all the points involved. Besides this, a neighbours-consistency technique is introduced to enhance the resulting image quality. The proposed method has been evaluated based on several criteria and for various resolutions. Evaluation results show that the proposed method compared to the conventional approach executes upto 49 times faster and improves PSNR and SSIM metrics for the resulting images by 26% and 12%; respectively. This improvement is beneficial for such domains as feature matching, edge detection, OCR and calibration. To enable researchers generating the same results and extend this work, the dataset and implementation codes are publicly available [1].\u3c/p\u3

Real-time planar segmentation of depth images:from three-dimensional edges to segmented planes

Abstract. Real-time execution of processing algorithms for handling depth images in a three-dimensional (3-D) data framework is a major challenge. More specifically, considering depth images as point clouds and performing planar segmentation requires heavy computation, because available planar segmentation algorithms are mostly based on surface normals and/or curvatures, and, consequently, do not provide real-time performance. Aiming at the reconstruction of indoor environments, the spaces mainly consist of planar surfaces, so that a possible 3-D application would strongly benefit from a real-time algorithm. We introduce a real-time planar segmentation method for depth images avoiding any surface normal calculation. First, we detect 3-D edges in a depth image and generate line segments between the identified edges. Second, we fuse all the points on each pair of intersecting line segments into a plane candidate. Third and finally, we implement a validation phase to select planes from the candidates. Furthermore, various enhancements are applied to improve the segmentation quality. The GPU implementation of the proposed algorithm segments depth images into planes at the rate of 58 fps. Our pipeline-interleaving technique increases this rate up to 100 fps. With this throughput rate improvement, the application benefit of our algorithm may be further exploited in terms of quality and enhancing the localization

Solidarity filter for noise reduction of 3D edges in depth images

3D applications processing depth images significantly benefit from 3D-edge extraction techniques. Intrinsic sensor noise in depth images is largely inherited to the extracted 3D edges. Conventional denoising algorithms remove some of this noise, but also weaken narrow edges, amplify noisy pixels and introduce false edges. We therefore propose a novel solidarity filter for noise removal in 3D edge images without artefacts such as false edges. The proposed filter is defining neighbouring pixels with similar properties and connecting those into larger segments beyond the size of a conventional filter aperture. The experimental results show that the solidarity filter outperforms the median and morphological close filters with 42% and 69% higher PSNR, respectively. In terms of the mean SSIM metric, the solidarity filter provides results that are 11% and 21% closer to the ground truth than the corresponding results obtained by the median and close filters, respectively