3D Scene Geometry Estimation from 360∘^\circ Imagery: A Survey

This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under the omnidirectional optics. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360$^\circ$, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this paper by pointing out current and future trends.Comment: Published in ACM Computing Survey

Block world reconstruction from spherical stereo image pairs

We propose a block-based scene reconstruction method using multiple stereo pairs of spherical images. We assume that the urban scene consists of axis-aligned planar structures (Manhattan world). Captured spherical stereo images are converted into six central-point perspective images by cubic projection and fa____c cade alignment. Depth information is recovered by stereo matching between images. Semantic regions are segmented based on colour, edge and normal information. Independent 3D rectangular planes are constructed by fitting planes aligned with the principal axes of the segmented 3D points. Finally cuboid-based scene structure is recovered from multiple viewpoints by merging and refining planes based on connectivity and visibility. The reconstructed model efficiently shows the structure of the scene with a small amount of data

Block world reconstruction from spherical stereo image pairs

We propose a block-based scene reconstruction method using multiple stereo pairs of spherical images. We assume that the urban scene consists of axis-aligned planar structures (Manhattan world). Captured spherical stereo images are converted into six central-point perspective images by cubic projection and façade alignment. Depth information is recovered by stereo matching between images. Semantic regions are segmented based on colour, edge and normal information. Independent 3D rectangular planes are constructed by fitting planes aligned with the principal axes of the segmented 3D points. Finally cuboid-based scene structure is recovered from multiple viewpoints by merging and refining planes based on connectivity and visibility. The reconstructed model efficiently shows the structure of the scene with a small amount of data

Virtual 3D reconstruction of complex urban environments

[ES] Este trabajo presenta una metodología para la generación de modelos tridimensionales de entornos urbanos. Se utiliza una plataforma terrestre multi-sensores compuesta por un LIDAR, una cámara esférica, GPS y otros sistemas inerciales. Los datos de los sensores están sincronizados con el sistema de navegación y georrefenciados. La metodología de digitalizaciónn se centra en 3 procesos principales. (1) La reconstrucción tridimensional, en el cual se elimina el ruido en los datos 3D y se disminuye la distorsión en las imágenes. Posteriormente se construye una imagen panorámica. (2) La texturización, se describe a detalle el algoritmo para asegurar la menor incertidumbre en el proceso de extracción de color. (3) La generación de mallas, se describe el proceso de mallado basado en octree’s, desde la generación de la semilla, el teselado, así como la eliminación de huecos en las mallas. Por último, se realiza una evaluación cuantitativa de la propuesta y se compara con otros enfoques existen[EN] This paper presents a methodology for the generation of three-dimensional models of urban environments. A multi-sensor terrestrial platform composed of a LIDAR, a spherical camera, GPS and IMU systems is used. The data of the sensors are synchronized with the navigation system and georeferenced. The digitalization methodology is focused on 3 main processes. (1) The three-dimensional reconstruction, in which the noise in the 3D data is eliminated and the distortion in the images is reduced. Later, a panoramic image is built. (2) Texturing, the algorithm is described in detail to ensure the least uncertainty in this color extraction process. (3) Mesh generation, the meshing process based on octree’s is described, from the generation of the seed, the tessellation, as well as the elimination of gaps in the meshes. Finally, a quantitative evaluation of our proposal is made and compared with other existing approaches in the state-of-the-art. The results obtained are discussed in detail.García-Moreno, A.; González-Barbosa, J. (2020). Reconstrucción virtual tridimensional de entornos urbanos complejos. Revista Iberoamericana de Automática e Informática industrial. 17(1):22-33. https://doi.org/10.4995/riai.2019.11203OJS2233171Bernard O Abayowa, Alper Yilmaz, and Russell C Hardie. Automatic registration of optical aerial imagery to a lidar point cloud for generation of city models. ISPRS Journal of Photogrammetry and Remote Sensing, 106:68-81, 2015. https://doi.org/10.1016/j.isprsjprs.2015.05.006Gerardo Atanacio-Jiménez, José-Joel González-Barbosa, Juan B Hurtado-Ramos, Francisco J Ornelas-Rodríguez, Hugo Jiménez-Hernández, Teresa García-Ramirez, and Ricardo González-Barbosa. Lidar velodyne hdl-64e calibration using pattern planes. International Journal on Advanced Robotics Systems, 8(5):70-82, 2011. https://doi.org/10.5772/50900Matthew Brown, Richard Szeliski, and Simon Winder. Multi-image matching using multi-scale oriented patches. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 510-517. IEEE, 2005.Jonathan C Carr, Richard K Beatson, Jon B Cherrie, Tim J Mitchell, W Richard Fright, Bruce C McCallum, and Tim R Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 67-76. ACM, 2001.Ke Chen, Weisheng Lu, Fan Xue, Pingbo Tang, and Ling Hin Li. Automatic building information model reconstruction in high-density urban areas: Augmenting multi-source data with architectural knowledge. Automation in Construction, 93:22-34, 2018. https://doi.org/10.1016/j.autcon.2018.05.009Tamal K Dey and Samrat Goswami. Provable surface reconstruction from noisy samples. In Proceedings of the twentieth annual symposium on Computational geometry, pages 330-339. ACM, 2004.Luca Di Angelo, Paolo Di Stefano, and Luigi Giaccari. A new mesh-growing algorithm for fast surface reconstruction. Computer-Aided Design, 43(6): 639-650, 2011. https://doi.org/10.1016/j.cad.2011.02.012Julie Digne. An analysis and implementation of a parallel ball pivoting algorithm. Image Processing On Line, 4:149-168, 2014. https://doi.org/10.5201/ipol.2014.81Damien Garcia. Robust smoothing of gridded data in one and higher dimensions with missing values. Computational statistics & data analysis, 54(4):1167-1178, 2010. https://doi.org/10.1016/j.csda.2009.09.020Angel-Iván García-Moreno, José-Joel Gonzalez-Barbosa, Francisco-Javier Ornelas-Rodriguez, Juan B Hurtado-Ramos, and Marco-Neri Primo-Fuentes. Lidar and panoramic camera extrinsic calibration approach using a pattern plane. In Pattern Recognition. Springer, 2013. https://doi.org/10.1007/978-3-642-38989-4_11Angel-Iván García-Moreno, Denis-Eduardo Hernandez-García, José-Joel Gonzalez-Barbosa, Alfonso Ramírez-Pedraza, Juan B Hurtado-Ramos, and Francisco-Javier Ornelas-Rodriguez. Error propagation and uncertainty analysis between 3d laser scanner and camera. Robotics and Autonomous Systems, 62(6):782-793, 2014. https://doi.org/10.1016/j.robot.2014.02.004Angel-Iván García-Moreno, José-Joel González-Barbosa, Alfonso Ramírez-Pedraza, Juan B Hurtado-Ramos, and Francisco-Javier Ornelas-Rodriguez. Accurate evaluation of sensitivity for calibration between a lidar and a panoramic camera used for remote sensing. Journal of Applied Remote Sensing, 10(2):024002-024002, 2016. https://doi.org/10.1117/1.JRS.10.024002Jianwei Guo, Dong-Ming Yan, Li Chen, Xiaopeng Zhang, Oliver Deussen, and Peter Wonka. Tetrahedral meshing via maximal poisson-disk sampling. Computer Aided Geometric Design, 43:186-199, 2016. https://doi.org/10.1016/j.cagd.2016.02.004Rostam Affendi Hamzah, A Fauzan Kadmin, M Saad Hamid, S Fakhar A Ghani, and Haidi Ibrahim. Improvement of stereo matching algorithm for 3d surface reconstruction. Signal Processing: Image Communication, 65:165-172, 2018. https://doi.org/10.1016/j.image.2018.04.001Chris Harris. Geometry from visual motion. In Active vision, pages 263-284. MIT Press, 1993.C. Hatger and C. Brenner. Extraction of road geometry parameters from laser scanning and existing databases. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 34(3/W13):225-230, 2003.Dorota Iwaszczuk and Uwe Stilla. Camera pose refinement by matching uncertain 3d building models with thermal infrared image sequences for high quality texture extraction. ISPRS Journal of Photogrammetry and Remote Sensing, 132:33-47, 2017. https://doi.org/10.1016/j.isprsjprs.2017.08.006Hansung Kim and Adrian Hilton. Block world reconstruction from spherical stereo image pairs. Computer Vision and Image Understanding, 139:104-121, 2015. https://doi.org/10.1016/j.cviu.2015.04.001Eyal Kushilevitz, Rafail Ostrovsky, and Yuval Rabani. Efficient search for approximate nearest neighbor in high dimensional spaces. SIAM Journal on Computing, 30(2):457-474, 2000. https://doi.org/10.1137/S0097539798347177Maxime Lhuillier. Surface reconstruction from a sparse point cloud by enforcing visibility consistency and topology constraints. Computer Vision and Image Understanding, 175:52-71, 2018. https://doi.org/10.1016/j.cviu.2018.09.007Lingyun Liu and Ioannis Stamos. A systematic approach for 2d-image to 3drange registration in urban environments. Computer Vision and Image Understanding, 116(1):25-37, 2012. https://doi.org/10.1016/j.cviu.2011.07.009Jules Morel, Alexandra Bac, and Cédric Véga. Surface reconstruction of incomplete datasets: A novel poisson surface approach based on csrbf. Computers & Graphics, 74:44-55, 2018. https://doi.org/10.1016/j.cag.2018.05.004Gaurav Pandey, James R McBride, and Ryan M Eustice. Ford campus vision and lidar data set. The International Journal of Robotics Research, 30(13): 1543-1552, 2011. https://doi.org/10.1177/0278364911400640Gaurav Pandey, James R McBride, Silvio Savarese, and Ryan M Eustice. Automatic extrinsic calibration of vision and lidar by maximizing mutual information. Journal of Field Robotics, 2014. https://doi.org/10.1002/rob.21542Yun Shi, Shunping Ji, Xiaowei Shao, Peng Yang, Wenbin Wu, Zhongchao Shi, and Ryosuke Shibasaki. Fusion of a panoramic camera and 2d laser scanner data for constrained bundle adjustment in gps-denied environments. Image and Vision Computing, 40:28-37, 2015. https://doi.org/10.1016/j.imavis.2015.06.002Miao Wang and Yi-Hsing Tseng. Automatic segmentation of lidar data into coplanar point clusters using an octree-based split-and-merge algorithm. Photogrammetric Engineering & Remote Sensing, 76(4):407-420, 2010. https://doi.org/10.14358/PERS.76.4.407Ruisheng Wang, Jeff Bach, Jane Macfarlane, and Frank P Ferrie. A new upsampling method for mobile lidar data. In Applications of Computer Vision (WACV), 2012 IEEE Workshop on, pages 17-24. IEEE, 2012. https://doi.org/10.1109/WACV.2012.6162998Bin Wu, Bailang Yu, Qiusheng Wu, Shenjun Yao, Feng Zhao, Weiqing Mao, and Jianping Wu. A graph-based approach for 3d building model reconstruction from airborne lidar point clouds. Remote Sensing, 9(1):92, 2017. https://doi.org/10.3390/rs9010092Lin Yang, Yehua Sheng, and Bo Wang. 3d reconstruction of building façade with fused data of terrestrial lidar data and optical image. Optik-International Journal for Light and Electron Optics, 127(4):2165-2168, 2016. https://doi.org/10.1016/j.ijleo.2015.11.147Michael Ying Yang, Yanpeng Cao, and John McDonald. Fusion of camera images and laser scans for wide baseline 3d scene alignment in urban environments. ISPRS Journal of Photogrammetry and Remote Sensing, 66(6): S52-S61, 2011. https://doi.org/10.1016/j.isprsjprs.2011.09.004Cheng Yi, Yuan Zhang, Qiaoyun Wu, Yabin Xu, Oussama Remil, Mingqiang Wei, and JunWang. Urban building reconstruction from raw lidar point data. Computer-Aided Design, 93:1-14, 2017. https://doi.org/10.1016/j.cad.2017.07.005Fanyang Zeng and Ruofei Zhong. The algorithm to generate color point-cloud with the registration between panoramic image and laser point-cloud. In IOP Conference Series: Earth and Environmental Science, volume 17, page 012160. IOP Publishing, 2014. https://doi.org/10.1088/1755-1315/17/1/012160SM Iman Zolanvari, Debra F Laefer, and Atteyeh S Natanzi. Three-dimensional building fac¸ade segmentation and opening area detection from point clouds. ISPRS journal of photogrammetry and remote sensing, 143:134-149, 2018. https://doi.org/10.1016/j.isprsjprs.2018.04.00