Intelligent Robotic Perception Systems

Robotic perception is related to many applications in robotics where sensory data and artificial intelligence/machine learning (AI/ML) techniques are involved. Examples of such applications are object detection, environment representation, scene understanding, human/pedestrian detection, activity recognition, semantic place classification, object modeling, among others. Robotic perception, in the scope of this chapter, encompasses the ML algorithms and techniques that empower robots to learn from sensory data and, based on learned models, to react and take decisions accordingly. The recent developments in machine learning, namely deep-learning approaches, are evident and, consequently, robotic perception systems are evolving in a way that new applications and tasks are becoming a reality. Recent advances in human-robot interaction, complex robotic tasks, intelligent reasoning, and decision-making are, at some extent, the results of the notorious evolution and success of ML algorithms. This chapter will cover recent and emerging topics and use-cases related to intelligent perception systems in robotics

Sensing of complex buildings and reconstruction into photo-realistic 3D models

The 3D reconstruction of indoor and outdoor environments has received an interest only recently, as companies began to recognize that using reconstructed models is a way to generate revenue through location-based services and advertisements. A great amount of research has been done in the field of 3D reconstruction, and one of the latest and most promising applications is Kinect Fusion, which was developed by Microsoft Research. Its strong points are the real-time intuitive 3D reconstruction, interactive frame rate, the level of detail in the models, and the availability of the hardware and software for researchers and enthusiasts. A representative effort towards 3D reconstruction is the Point Cloud Library (PCL). PCL is a large scale, open project for 2D/3D image and point cloud processing. On December 2011, PCL made available an implementation of Kinect Fusion, namely KinFu. KinFu emulates the functionality provided in Kinect Fusion. However, both implementations have two major limitations: 1. The real-time reconstruction takes place only within a cube with a size of 3 meters per axis. The cube's position is fixed at the start of execution, and any object outside of this cube is not integrated into the reconstructed model. Therefore the volume that can be scanned is always limited by the size of the cube. It is possible to manually align many small-size cubes into a single large model, however this is a time-consuming and difficult task, especially when the meshes have complex topologies and high polygon count, as is the case with the meshes obtained from KinFu. 2. The output mesh does not have any color textures. There are some at-tempts to add color in the output point cloud; however, the resulting effect is not photo-realistic. Applying photo-realistic textures to a model can enhance the user experience, even when the model has a simple topology. The main goal of this project is to design and implement a system that captures large indoor environments and generates 3D photo-realistic large indoor models in real time. This report describes an extended version of the KinFu system. The extensions overcome the scalability and texture reconstruction limitations using commodity hardware and open-source software. The complete hardware setup used in this project is worth €2,000, which is comparable to the cost of a single professional laser scanner. The software is released under BSD license, which makes it completely free to use and commercialize. The system has been integrated into the open-source PCL project. The immediate benefits are three-fold: the system becomes a potential industry standard, it is maintained and extended by many developers around the world with no addition-al cost to the VCA group, and it can reduce the application development time by reusing numerous state-of-the-art algorithms

3D Reconstruction of Civil Infrastructures from UAV Lidar point clouds

Na atualidade, infraestruturas para transporte, comunicação, energia, produção industrial e fim social apresentam-se como pilares da sociedade, sendo imprescindíveis para o seu bom funcionamento. Aliada a esta grande importância dentro da sociedade, existe necessidade de garantir a segurança e durabilidade destes ativos. Assim, técnicas confiáveis devem ser utilizadas para avaliar o seu estado. Com o avanço tecnológico e o desenvolvimento de novos métodos de aquisição de dados, algumas tarefas relacionadas com a construção civil, atualmente realizadas por seres humanos, como a inspeção e o controlo de qualidade, tornam-se ineficientes dado o seu perigo e custo. Neste contexto, a reconstrução 3D de infraestruturas surge como uma possível solução, apresentando-se como um primeiro passo para a monitorização e acompanhamento de infraestruturas, bem como uma ferramenta para processos de inspeção semi ou completamente automatizados. Para o desenvolvimento desta tese, recorreu-se a um sensor Lidar acoplado a um UAV. Com este equipamento, tornou-se possível sobrevoar de forma autónoma infraestruturas reais, extraindo dados de todas as suas superfícies, independentemente das dificuldades que poderiam surgir para alcançar tais regiões a partir do solo. Os dados são extraídos na forma de nuvens de pontos com respetivas intensidades, filtrados, e utilizados em algoritmos de reconstrução e texturização, culminando numa representação virtual e tridimensional da infraestrutura alvo. Com estas representações torna-se possível avaliar a evolução da infraestrutura aquando da sua construção ou reparação, bem como permite avaliar a evolução temporal de determinados defeitos presentes na construção, bastando, para isso, comparar modelos relativos ao mesmo cenário obtidos a partir de dados extraídos em diferentes ocasiões. Esta abordagem permite que o processo de monitorização de infraestruturas possa ser realizado de forma mais eficiente, com menores custos e garantindo a segurança dos trabalhadores.Nowadays, infrastructures for transportation, communication, energy, industrial production and social purpose are presented as pillars of society, being essential for its proper functioning. Coupled with this great importance within society, there is a need to ensure the safety and durability of these assets. Thus, reliable techniques should be used to assess their condition. With technological advances and the development of new methods of data acquisition, some tasks related to civil construction, currently performed by human beings, such as inspection and quality control, become inefficient due to their danger and cost. In this context, 3D reconstruction of infrastructures appears as a possible solution, presenting itself as a first step for the monitoring of infrastructures, as well as a tool for semi or completely automated inspection processes. For the development of this thesis, a Lidar sensor coupled to a UAV was used. With this equipment, it became possible to autonomously fly over real infrastructures, extracting data from all its surfaces, regardless of the difficulties that could arise to reach such regions from the ground. The data is extracted in the form of point clouds with respective intensities, filtered, and used in reconstruction and texturing algorithms, culminating in a virtual and three-dimensional representation of the target infrastructure. With these representations, it is possible to evaluate the evolution of the infrastructure during its construction or repair, as well as to evaluate the temporal evolution of certain defects present in the construction, by comparing models for the same scenario obtained from data extracted on different occasions. This approach allows the process of monitoring infrastructures to be carried out more efficiently, with lower costs and ensuring the safety of workers

Large-Scale Textured 3D Scene Reconstruction

Die Erstellung dreidimensionaler Umgebungsmodelle ist eine fundamentale Aufgabe im Bereich des maschinellen Sehens. Rekonstruktionen sind für eine Reihe von Anwendungen von Nutzen, wie bei der Vermessung, dem Erhalt von Kulturgütern oder der Erstellung virtueller Welten in der Unterhaltungsindustrie. Im Bereich des automatischen Fahrens helfen sie bei der Bewältigung einer Vielzahl an Herausforderungen. Dazu gehören Lokalisierung, das Annotieren großer Datensätze oder die vollautomatische Erstellung von Simulationsszenarien. Die Herausforderung bei der 3D Rekonstruktion ist die gemeinsame Schätzung von Sensorposen und einem Umgebunsmodell. Redundante und potenziell fehlerbehaftete Messungen verschiedener Sensoren müssen in eine gemeinsame Repräsentation der Welt integriert werden, um ein metrisch und photometrisch korrektes Modell zu erhalten. Gleichzeitig muss die Methode effizient Ressourcen nutzen, um Laufzeiten zu erreichen, welche die praktische Nutzung ermöglichen. In dieser Arbeit stellen wir ein Verfahren zur Rekonstruktion vor, das fähig ist, photorealistische 3D Rekonstruktionen großer Areale zu erstellen, die sich über mehrere Kilometer erstrecken. Entfernungsmessungen aus Laserscannern und Stereokamerasystemen werden zusammen mit Hilfe eines volumetrischen Rekonstruktionsverfahrens fusioniert. Ringschlüsse werden erkannt und als zusätzliche Bedingungen eingebracht, um eine global konsistente Karte zu erhalten. Das resultierende Gitternetz wird aus Kamerabildern texturiert, wobei die einzelnen Beobachtungen mit ihrer Güte gewichtet werden. Für eine nahtlose Erscheinung werden die unbekannten Belichtungszeiten und Parameter des optischen Systems mitgeschätzt und die Bilder entsprechend korrigiert. Wir evaluieren unsere Methode auf synthetischen Daten, realen Sensordaten unseres Versuchsfahrzeugs und öffentlich verfügbaren Datensätzen. Wir zeigen qualitative Ergebnisse großer innerstädtischer Bereiche, sowie quantitative Auswertungen der Fahrzeugtrajektorie und der Rekonstruktionsqualität. Zuletzt präsentieren wir mehrere Anwendungen und zeigen somit den Nutzen unserer Methode für Anwendungen im Bereich des automatischen Fahrens

AdaSplats: Adaptive Splatting of Point Clouds for Accurate 3D Modeling and Real-time High-Fidelity LiDAR Simulation

LiDAR sensors provide rich 3D information about their surrounding and are becoming increasingly important for autonomous vehicles tasks, such as semantic segmentation, object detection, and tracking. Simulating a LiDAR sensor accelerates the testing, validation, and deployment of autonomous vehicles, while reducing the cost and eliminating the risks of testing in real-world scenarios. We address the problem of high-fidelity LiDAR simulation and present a pipeline that leverages real-world point clouds acquired by mobile mapping systems. Point-based geometry representations, more specifically splats, have proven their ability to accurately model the underlying surface in very large point clouds. We introduce an adaptive splats generation method that accurately models the underlying 3D geometry, especially for thin structures. Moreover, we introduce a physics-based, faster-than-real-time LiDAR simulator, in the splatted model, leveraging the GPU parallel architecture with an acceleration structure, while focusing on efficiently handling large point clouds. We test our LiDAR simulation in real-world conditions, showing qualitative and quantitative results compared to basic splatting and meshing techniques, demonstrating the interest of our modeling technique.Comment: 28 pages, 11 figures, 6 table

Automatic Dense 3D Scene Mapping from Non-overlapping Passive Visual Sensors for Future Autonomous Systems

The ever increasing demand for higher levels of autonomy for robots and vehicles means there is an ever greater need for such systems to be aware of their surroundings. Whilst solutions already exist for creating 3D scene maps, many are based on active scanning devices such as laser scanners and depth cameras that are either expensive, unwieldy, or do not function well under certain environmental conditions. As a result passive cameras are a favoured sensor due their low cost, small size, and ability to work in a range of lighting conditions. In this work we address some of the remaining research challenges within the problem of 3D mapping around a moving platform. We utilise prior work in dense stereo imaging, Stereo Visual Odometry (SVO) and extend Structure from Motion (SfM) to create a pipeline optimised for on vehicle sensing. Using forward facing stereo cameras, we use state of the art SVO and dense stereo techniques to map the scene in front of the vehicle. With significant amounts of prior research in dense stereo, we addressed the issue of selecting an appropriate method by creating a novel evaluation technique. Visual 3D mapping of dynamic scenes from a moving platform result in duplicated scene objects. We extend the prior work on mapping by introducing a generalized dynamic object removal process. Unlike other approaches that rely on computationally expensive segmentation or detection, our method utilises existing data from the mapping stage and the findings from our dense stereo evaluation. We introduce a new SfM approach that exploits our platform motion to create a novel dense mapping process that exceeds the 3D data generation rate of state of the art alternatives. Finally, we combine dense stereo, SVO, and our SfM approach to automatically align point clouds from non-overlapping views to create a rotational and scale consistent global 3D model

Automatic Reconstruction of Textured 3D Models

Three dimensional modeling and visualization of environments is an increasingly important problem. This work addresses the problem of automatic 3D reconstruction and we present a system for unsupervised reconstruction of textured 3D models in the context of modeling indoor environments. We present solutions to all aspects of the modeling process and an integrated system for the automatic creation of large scale 3D models