Wireless realtime motion tracking system using localised orientation estimation

A realtime wireless motion tracking system is developed. The system is capable of tracking the orientations of multiple wireless sensors, using a semi-distributed implementation to reduce network bandwidth and latency, to produce real-time animation of rigid body models, such as the human skeleton. The system has been demonstrated to be capable of full-body posture tracking of a human subject using fifteen devices communicating with a basestation over a single, low bandwidth, radio channel. The thesis covers the theory, design, and implementation of the tracking platform, the evaluation of the platform’s performance, and presents a summary of possible future applications

Vision-Inertial SLAM using Natural Features in Outdoor Environments

Simultaneous Localization and Mapping (SLAM) is a recursive probabilistic inferencing process used for robot navigation when Global Positioning Systems (GPS) are unavailable. SLAM operates by building a map of the robot environment, while concurrently localizing the robot within this map. The ultimate goal of SLAM is to operate anywhere using the environment's natural features as landmarks. Such a goal is difficult to achieve for several reasons. Firstly, different environments contain different types of natural features, each exhibiting large variance in its shape and appearance. Secondly, objects look differently from different viewpoints and it is therefore difficult to always recognize them. Thirdly, in most outdoor environments it is not possible to predict the motion of a vehicle using wheel encoders because of errors caused by slippage. Finally, the design of a SLAM system to operate in a large-scale outdoor setting is in itself a challenge. The above issues are addressed as follows. Firstly, a camera is used to recognize the environmental context (e. g. , indoor office, outdoor park) by analyzing the holistic spectral content of images of the robot's surroundings. A type of feature (e. g. , trees for a park) is then chosen for SLAM that is likely observable in the recognized setting. A novel tree detection system is introduced, which is based on perceptually organizing the content of images into quasi-vertical structures and marking those structures that intersect ground level as tree trunks. Secondly, a new tree recognition system is proposed, which is based on extracting Scale Invariant Feature Transform (SIFT) features on each tree trunk region and matching trees in feature space. Thirdly, dead-reckoning is performed via an Inertial Navigation System (INS), bounded by non-holonomic constraints. INS are insensitive to slippage and varying ground conditions. Finally, the developed Computer Vision and Inertial systems are integrated within the framework of an Extended Kalman Filter into a working Vision-INS SLAM system, named VisSLAM. VisSLAM is tested on data collected during a real test run in an outdoor unstructured environment. Three test scenarios are proposed, ranging from semi-automatic detection, recognition, and initialization to a fully automated SLAM system. The first two scenarios are used to verify the presented inertial and Computer Vision algorithms in the context of localization, where results indicate accurate vehicle pose estimation for the majority of its journey. The final scenario evaluates the application of the proposed systems for SLAM, where results indicate successful operation for a long portion of the vehicle journey. Although the scope of this thesis is to operate in an outdoor park setting using tree trunks as landmarks, the developed techniques lend themselves to other environments using different natural objects as landmarks

Maritime Augmented Reality mit a prioriWissen aus Seekarten

The main objective of this thesis is to provide a concept to augment mar- itime sea chart information into the camera view of the user. The benefit is the simpler navigation due to the offered 3D information and the overlay onto the real 3D environment. In the maritime context special conditions hold. The sensor technologies have to be reliable in the environment of a ship’s ferrous construction. The aug- mentation of the objects has to be very precise due to the far distances of observable objects on the sea surface. Furthermore, the approach has to be reliable due to the wide range of light conditions. For a practical solution, the system has to be mobile, light-weight and with a real-time performance. To achieve this goal, the requirements are set, the possible measurement units and the data base structure are presented. First, the requirements are analyzed and a suitable system is designed. By the combination of proper sensor techniques, the local position and orienta- tion of the user can be estimated. To verify the concept, several prototypes with exchangeable units have been evaluated. This first concept is based on a marker-based approach which leads to some drawbacks. To overcome the drawbacks, the second aspect is the improvement of the sys- tem and the analysis of markerless approaches. One possible strategy will be presented. The approach uses the statistical technique of Bayesian networks to vote for single objects in the environment. By this procedure it will be shown, that due to the a priori information the underlying sea chart system has the most benefit. The analysis of the markerless approach shows, that the sea charts structure has to be adapted to the new requirements of interactive 3D augmentation scenes. After the analysis of the chart data concept, an approach for the optimization of the charts by building up an object-to-object topology within the charts data and the Bayesian object detection approach is presented. Finally, several evaluations show the performance of the imple- mented evaluation application.Diese Arbeit stellt ein Konzept zur Verfügung, um Seekarteninformationen in eine Kamera so einzublenden, dass die Informationen lagerichtig im Sichtfeld des Benutzers erscheinen. Der Mehrwert ist eine einfachere Navigation durch die Nutzung von 3D-Symbolen in der realen Umgebung. Im maritimen Umfeld gelten besondere Anforderungen an die Aufgabenstellung. Die genutzten Sensoren müssen in der Lage sein, robuste Daten in Anwesenheit der eisenhaltigen Materialien auf dem Schiff zu liefern. Die Augmentierung muss hoch genau berechnet werden, da die beobachtbaren Objekte zum Teil sehr weit entfernt auf der Meeresoberfläche verteilt sind. Weiterhin gelten die Bedingungen einer Außenumgebung, wie variierende Wetter- und Lichtbedingungen. Um eine praktikable Anwendung gewährleisten zu können, ist ein mobiles, leicht-gewichtiges und echtzeitfähiges System zu entwickeln. In dieser Arbeit werden die Anforderungen gesetzt und Konzepte für die Hardware- und Softwarelösungen beschrieben. Im ersten Teil werden die Anforderungen analysiert und ein geeignetes Hardwaresystem entwickelt. Durch die passende Kombination von Sensortechnologien kann damit die lokale Position und Orientierung des Benutzers berechnet werden. Um das Konzept zu evaluieren sind verschiedene modulare Hardware- und Softwarekonzepte als Prototypen umgesetzt worden. Das erste Softwarekonzept befasst sich mit einem markerbasierten Erkennungsalgorithmus, der in der Evaluation einige Nachteile zeigt. Dementsprechende Verbesserungen wurden in einem zweiten Softwarekonzept durch einen markerlosen Ansatz umgesetzt. Dieser Lösungsansatz nutzt Bayes'sche Netzwerke zur Erkennung einzelner Objekte in der Umgebung. Damit kann gezeigt werden, dass mit der Hilfe von a priori Informationen die dem System zugrunde liegenden Seekarten sehr gut zu diesem Zweck genutzt werden können. Die Analyse des Systemkonzeptes zeigt des weiteren, dass die Datenstruktur der Seekarten für die Anforderungen einer interaktiven, benutzergeführten 3D- Augmentierungsszene angepasst werden müssen. Nach der ausführlichen Analyse des Seekarten-Datenkonzeptes wird ein Lösungsansatz zur Optimierung der internen Seekartenstruktur aufgezeigt. Dies wird mit der Erstellung einer Objekt-zu-Objekt-Topologie in der Datenstruktur und der Verbindung zum Bayes'schen Objekterkennungsalgorithmus umgesetzt. Anschließend zeigen Evaluationen die Fähigkeiten des endgültigen Systems

Low-Cost Sensors and Biological Signals

Many sensors are currently available at prices lower than USD 100 and cover a wide range of biological signals: motion, muscle activity, heart rate, etc. Such low-cost sensors have metrological features allowing them to be used in everyday life and clinical applications, where gold-standard material is both too expensive and time-consuming to be used. The selected papers present current applications of low-cost sensors in domains such as physiotherapy, rehabilitation, and affective technologies. The results cover various aspects of low-cost sensor technology from hardware design to software optimization

Real-time spatial modeling to detect and track resources on construction sites

For more than 10 years the U.S. construction industry has experienced over 1,000 fatalities annually. Many fatalities may have been prevented had the individuals and equipment involved been more aware of and alert to the physical state of the environment around them. Awareness may be improved by automatic 3D (three-dimensional) sensing and modeling of the job site environment in real-time. Existing 3D modeling approaches based on range scanning techniques are capable of modeling static objects only, and thus cannot model in real-time dynamic objects in an environment comprised of moving humans, equipment, and materials. Emerging prototype 3D video range cameras offer another alternative by facilitating affordable, wide field of view, automated static and dynamic object detection and tracking at frame rates better than 1Hz (real-time). This dissertation presents an imperical work and methodology to rapidly create a spatial model of construction sites and in particular to detect, model, and track the position, dimension, direction, and velocity of static and moving project resources in real-time, based on range data obtained from a three-dimensional video range camera in a static or moving position. Existing construction site 3D modeling approaches based on optical range sensing technologies (laser scanners, rangefinders, etc.) and 3D modeling approaches (dense, sparse, etc.) that offered potential solutions for this research are reviewed. The choice of an emerging sensing tool and preliminary experiments with this prototype sensing technology are discussed. These findings led to the development of a range data processing algorithm based on three-dimensional occupancy grids which is demonstrated in detail. Testing and validation of the proposed algorithms have been conducted to quantify the performance of sensor and algorithm through extensive experimentation involving static and moving objects. Experiments in indoor laboratory and outdoor construction environments have been conducted with construction resources such as humans, equipment, materials, or structures to verify the accuracy of the occupancy grid modeling approach. Results show that modeling objects and measuring their position, dimension, direction, and speed had an accuracy level compatible to the requirements of active safety features for construction. Results demonstrate that video rate 3D data acquisition and analysis of construction environments can support effective detection, tracking, and convex hull modeling of objects. Exploiting rapidly generated three-dimensional models for improved visualization, communications, and process control has inherent value, broad application, and potential impact, e.g. as-built vs. as-planned comparison, condition assessment, maintenance, operations, and construction activities control. In combination with effective management practices, this sensing approach has the potential to assist equipment operators to avoid incidents that result in reduce human injury, death, or collateral damage on construction sites.Civil, Architectural, and Environmental Engineerin

Adaptive Vision Based Scene Registration for Outdoor Augmented Reality

Augmented Reality (AR) involves adding virtual content into real scenes. Scenes are viewed using a Head-Mounted Display or other display type. In order to place content into the user's view of a scene, the user's position and orientation relative to the scene, commonly referred to as their pose, must be determined accurately. This allows the objects to be placed in the correct positions and to remain there when the user moves or the scene changes. It is achieved by tracking the user in relation to their environment using a variety of technology. One technology which has proven to provide accurate results is computer vision. Computer vision involves a computer analysing images and achieving an understanding of them. This may be locating objects such as faces in the images, or in the case of AR, determining the pose of the user. One of the ultimate goals of AR systems is to be capable of operating under any condition. For example, a computer vision system must be robust under a range of different scene types, and under unpredictable environmental conditions due to variable illumination and weather. The majority of existing literature tests algorithms under the assumption of ideal or 'normal' imaging conditions. To ensure robustness under as many circumstances as possible it is also important to evaluate the systems under adverse conditions. This thesis seeks to analyse the effects that variable illumination has on computer vision algorithms. To enable this analysis, test data is required to isolate weather and illumination effects, without other factors such as changes in viewpoint that would bias the results. A new dataset is presented which also allows controlled viewpoint differences in the presence of weather and illumination changes. This is achieved by capturing video from a camera undergoing a repeatable motion sequence. Ground truth data is stored per frame allowing images from the same position under differing environmental conditions, to be easily extracted from the videos. An in depth analysis of six detection algorithms and five matching techniques demonstrates the impact that non-uniform illumination changes can have on vision algorithms. Specifically, shadows can degrade performance and reduce confidence in the system, decrease reliability, or even completely prevent successful operation. An investigation into approaches to improve performance yields techniques that can help reduce the impact of shadows. A novel algorithm is presented that merges reference data captured at different times, resulting in reference data with minimal shadow effects. This can significantly improve performance and reliability when operating on images containing shadow effects. These advances improve the robustness of computer vision systems and extend the range of conditions in which they can operate. This can increase the usefulness of the algorithms and the AR systems that employ them

Enabling Multi-LiDAR Sensing in GNSS-Denied Environments: SLAM Dataset, Benchmark, and UAV Tracking with LiDAR-as-a-camera

The rise of Light Detection and Ranging (LiDAR) sensors has profoundly impacted industries ranging from automotive to urban planning. As these sensors become increasingly affordable and compact, their applications are diversifying, driving precision, and innovation. This thesis delves into LiDAR's advancements in autonomous robotic systems, with a focus on its role in simultaneous localization and mapping (SLAM) methodologies and LiDAR as a camera-based tracking for Unmanned Aerial Vehicles (UAV). Our contributions span two primary domains: the Multi-Modal LiDAR SLAM Benchmark, and the LiDAR-as-a-camera UAV Tracking. In the former, we have expanded our previous multi-modal LiDAR dataset by adding more data sequences from various scenarios. In contrast to the previous dataset, we employ different ground truth-generating approaches. We propose a new multi-modal multi-lidar SLAM-assisted and ICP-based sensor fusion method for generating ground truth maps. Additionally, we also supplement our data with new open road sequences with GNSS-RTK. This enriched dataset, supported by high-resolution LiDAR, provides detailed insights through an evaluation of ten configurations, pairing diverse LiDAR sensors with state-of-the-art SLAM algorithms. In the latter contribution, we leverage a custom YOLOv5 model trained on panoramic low-resolution images from LiDAR reflectivity (LiDAR-as-a-camera) to detect UAVs, demonstrating the superiority of this approach over point cloud or image-only methods. Additionally, we evaluated the real-time performance of our approach on the Nvidia Jetson Nano, a popular mobile computing platform. Overall, our research underscores the transformative potential of integrating advanced LiDAR sensors with autonomous robotics. By bridging the gaps between different technological approaches, we pave the way for more versatile and efficient applications in the future

Influence of complex environments on LiDAR-Based robot navigation

La navigation sécuritaire et efficace des robots mobiles repose grandement sur l’utilisation des capteurs embarqués. L’un des capteurs qui est de plus en plus utilisé pour cette tâche est le Light Detection And Ranging (LiDAR). Bien que les recherches récentes montrent une amélioration des performances de navigation basée sur les LiDARs, faire face à des environnements non structurés complexes ou des conditions météorologiques difficiles reste problématique. Dans ce mémoire, nous présentons une analyse de l’influence de telles conditions sur la navigation basée sur les LiDARs. Notre première contribution est d’évaluer comment les LiDARs sont affectés par les flocons de neige durant les tempêtes de neige. Pour ce faire, nous créons un nouvel ensemble de données en faisant l’acquisition de données durant six précipitations de neige. Une analyse statistique de ces ensembles de données, nous caractérisons la sensibilité de chaque capteur et montrons que les mesures de capteurs peuvent être modélisées de manière probabilistique. Nous montrons aussi que les précipitations de neige ont peu d’influence au-delà de 10 m. Notre seconde contribution est d’évaluer l’impact de structures tridimensionnelles complexes présentes en forêt sur les performances d’un algorithme de reconnaissance d’endroits. Nous avons acquis des données dans un environnement extérieur structuré et en forêt, ce qui permet d’évaluer l’influence de ces derniers sur les performances de reconnaissance d’endroits. Notre hypothèse est que, plus deux balayages laser sont proches l’un de l’autre, plus la croyance que ceux-ci proviennent du même endroit sera élevée, mais modulé par le niveau de complexité de l’environnement. Nos expériences confirment que la forêt, avec ses réseaux de branches compliqués et son feuillage, produit plus de données aberrantes et induit une chute plus rapide des performances de reconnaissance en fonction de la distance. Notre conclusion finale est que, les environnements complexes étudiés influencent négativement les performances de navigation basée sur les LiDARs, ce qui devrait être considéré pour développer des algorithmes de navigation robustes.To ensure safe and efficient navigation, mobile robots heavily rely on their ability to use on-board sensors. One such sensor, increasingly used for robot navigation, is the Light Detection And Ranging (LiDAR). Although recent research showed improvement in LiDAR-based navigation, dealing with complex unstructured environments or difficult weather conditions remains problematic. In this thesis, we present an analysis of the influence of such challenging conditions on LiDAR-based navigation. Our first contribution is to evaluate how LiDARs are affected by snowflakes during snowstorms. To this end, we create a novel dataset by acquiring data during six snowfalls using four sensors simultaneously. Based on statistical analysis of this dataset, we characterized the sensitivity of each device and showed that sensor measurements can be modelled in a probabilistic manner. We also showed that falling snow has little impact beyond a range of 10 m. Our second contribution is to evaluate the impact of complex of three-dimensional structures, present in forests, on the performance of a LiDAR-based place recognition algorithm. We acquired data in structured outdoor environment and in forest, which allowed evaluating the impact of the environment on the place recognition performance. Our hypothesis was that the closer two scans are acquired from each other, the higher the belief that the scans originate from the same place will be, but modulated by the level of complexity of the environments. Our experiments confirmed that forests, with their intricate network of branches and foliage, produce more outliers and induce recognition performance to decrease more quickly with distance when compared with structured outdoor environment. Our conclusion is that falling snow conditions and forest environments negatively impact LiDAR-based navigation performance, which should be considered to develop robust navigation algorithms

Sistemas automáticos de informação e segurança para apoio na condução de veículos

Doutoramento em Engenharia MecânicaO objeto principal desta tese é o estudo de algoritmos de processamento e representação automáticos de dados, em particular de informação obtida por sensores montados a bordo de veículos (2D e 3D), com aplicação em contexto de sistemas de apoio à condução. O trabalho foca alguns dos problemas que, quer os sistemas de condução automática (AD), quer os sistemas avançados de apoio à condução (ADAS), enfrentam hoje em dia. O documento é composto por duas partes. A primeira descreve o projeto, construção e desenvolvimento de três protótipos robóticos, incluindo pormenores associados aos sensores montados a bordo dos robôs, algoritmos e arquitecturas de software. Estes robôs foram utilizados como plataformas de ensaios para testar e validar as técnicas propostas. Para além disso, participaram em várias competições de condução autónoma tendo obtido muito bons resultados. A segunda parte deste documento apresenta vários algoritmos empregues na geração de representações intermédias de dados sensoriais. Estes podem ser utilizados para melhorar técnicas já existentes de reconhecimento de padrões, deteção ou navegação, e por este meio contribuir para futuras aplicações no âmbito dos AD ou ADAS. Dado que os veículos autónomos contêm uma grande quantidade de sensores de diferentes naturezas, representações intermédias são particularmente adequadas, pois podem lidar com problemas relacionados com as diversas naturezas dos dados (2D, 3D, fotométrica, etc.), com o carácter assíncrono dos dados (multiplos sensores a enviar dados a diferentes frequências), ou com o alinhamento dos dados (problemas de calibração, diferentes sensores a disponibilizar diferentes medições para um mesmo objeto). Neste âmbito, são propostas novas técnicas para a computação de uma representação multi-câmara multi-modal de transformação de perspectiva inversa, para a execução de correcção de côr entre imagens de forma a obter mosaicos de qualidade, ou para a geração de uma representação de cena baseada em primitivas poligonais, capaz de lidar com grandes quantidades de dados 3D e 2D, tendo inclusivamente a capacidade de refinar a representação à medida que novos dados sensoriais são recebidos.The main object of this thesis is the study of algorithms for automatic information processing and representation, in particular information provided by onboard sensors (2D and 3D), to be used in the context of driving assistance. The work focuses on some of the problems facing todays Autonomous Driving (AD) systems and Advanced Drivers Assistance Systems (ADAS). The document is composed of two parts. The first part describes the design, construction and development of three robotic prototypes, including remarks about onboard sensors, algorithms and software architectures. These robots were used as test beds for testing and validating the developed techniques; additionally, they have participated in several autonomous driving competitions with very good results. The second part of this document presents several algorithms for generating intermediate representations of the raw sensor data. They can be used to enhance existing pattern recognition, detection or navigation techniques, and may thus benefit future AD or ADAS applications. Since vehicles often contain a large amount of sensors of different natures, intermediate representations are particularly advantageous; they can be used for tackling problems related with the diverse nature of the data (2D, 3D, photometric, etc.), with the asynchrony of the data (multiple sensors streaming data at different frequencies), or with the alignment of the data (calibration issues, different sensors providing different measurements of the same object). Within this scope, novel techniques are proposed for computing a multicamera multi-modal inverse perspective mapping representation, executing color correction between images for obtaining quality mosaics, or to produce a scene representation based on polygonal primitives that can cope with very large amounts of 3D and 2D data, including the ability of refining the representation as new information is continuously received

Utilizing radiation for smart robotic applications using visible, thermal, and polarization images.

The domain of this research is the use of computer vision methodologies in utilizing radiation for smart robotic applications for driving assistance. Radiation can be emitted by an object, reflected or transmitted. Understanding the nature and the properties of the radiation forming an image is essential in interpreting the information in that image which can then be used by a machine e.g. a smart vehicle to make a decision and perform an action. Throughout this work, different types of images are used to help a robotic vehicle make a decision and perform a certain action. This work presents three smart robotic applications; the first one deals with polarization images, the second one deals with thermal images and the third one deals with visible images. Each type of these images is formed by light (radiation) but in a way different from other types where the information embedded in an image depends on the way it was formed and how the light was generated. For polarization imaging, a direct method utilizing shading and polarization for unambiguous shape recovery without the need for nonlinear optimization routines is proposed. The proposed method utilizes simultaneously polarization and shading to find the surface normals, thus eliminating the reconstruction ambiguity. This can be useful to help a smart vehicle gain knowledge about the terrain surface geometry. Regarding thermal imaging, an automatic method for constructing an annotated thermal imaging pedestrian dataset is proposed. This is done by transferring detections from registered visible images simultaneously captured at day-time where pedestrian detection is well developed in visible images. Histogram of Oriented Gradients (HOG) features are extracted from the constructed dataset and then fed to a discriminatively trained deformable part based classifier that can be used to detect pedestrians at night. The resulting classifier was tested for night driving assistance and succeeded in detecting pedestrians even in the situations where visible imaging pedestrian detectors failed because of low light or glare of oncoming traffic. For visible images, a new feature based on HOG is proposed to be used for pedestrian detection. The proposed feature was augmented to two state of the art pedestrian detectors; the discriminatively trained Deformable Part based models (DPM) and the Integral Channel Features (ICF) using fast feature pyramids. The proposed approach is based on computing the image mixed partial derivatives to be used to redefine the gradients of some pixels and to reweigh the vote at all pixels with respect to the original HOG. The approach was tested on the PASCAL2007, INRIA and Caltech datasets and showed to have an outstanding performance