OpenPTrack: Open Source Multi-Camera Calibration and People Tracking for RGB-D Camera Networks

OpenPTrack is an open source software for multi-camera calibration and people tracking in RGB-D camera networks. It allows to track people in big volumes at sensor frame rate and currently supports a heterogeneous set of 3D sensors. In this work, we describe its user-friendly calibration procedure, which consists of simple steps with real-time feedback that allow to obtain accurate results in estimating the camera poses that are then used for tracking people. On top of a calibration based on moving a checkerboard within the tracking space and on a global optimization of cameras and checkerboards poses, a novel procedure which aligns people detections coming from all sensors in a x-y-time space is used for refining camera poses. While people detection is executed locally, in the machines connected to each sensor, tracking is performed by a single node which takes into account detections from all over the network. Here we detail how a cascade of algorithms working on depth point clouds and color, infrared and disparity images is used to perform people detection from different types of sensors and in any indoor light condition. We present experiments showing that a considerable improvement can be obtained with the proposed calibration refinement procedure that exploits people detections and we compare Kinect v1, Kinect v2 and Mesa SR4500 performance for people tracking applications. OpenPTrack is based on the Robot Operating System and the Point Cloud Library and has already been adopted in networks composed of up to ten imagers for interactive arts, education, culture and human\u2013robot interaction applications

Data fusion architecture for intelligent vehicles

Traffic accidents are an important socio-economic problem. Every year, the cost in human lives and the economic consequences are inestimable. During the latest years, efforts to reduce or mitigate this problem have lead to a reduction in casualties. But, the death toll in road accidents is still a problem, which means that there is still much work to be done. Recent advances in information technology have lead to more complex applications, which have the ability to help or even substitute the driver in case of hazardous situations, allowing more secure and efficient driving. But these complex systems require more trustable and accurate sensing technology that allows detecting and identifying the surrounding environment as well as identifying the different objects and users. However, the sensing technology available nowadays is insufficient itself, and thus combining the different available technologies is mandatory in order to fulfill the exigent requirements of safety road applications. In this way, the limitations of every system are overcome. More dependable and reliable information can be thus obtained. These kinds of applications are called Data Fusion (DF) applications. The present document tries to provide a solution for the Data Fusion problem in the Intelligent Transport System (ITS) field by providing a set of techniques and algorithms that allow the combination of information from different sensors. By combining these sensors the basic performances of the classical approaches in ITS can be enhanced, satisfying the demands of safety applications. The works presented are related with two researching fields. Intelligent Transport System is the researching field where this thesis was established. ITS tries to use the recent advances in Information Technology to increase the security and efficiency of the transport systems. Data Fusion techniques, on the other hand, try to give solution to the process related with the combination of information from different sources, enhancing the basic capacities of the systems and adding trustability to the inferences. This work attempts to use the Data Fusion algorithms and techniques to provide solution to classic ITS applications. The sensors used in the present application include a laser scanner and computer vision. First is a well known sensor, widely used, and during more recent years have started to be applied in different ITS applications, showing advanced performance mainly related to its trustability. Second is a recent sensor in automotive applications widely used in all recent ITS advances in the last decade. Thanks to computer vision road security applications (e.g. traffic sign detection, driver monitoring, lane detection, pedestrian detection, etc.) advancements are becoming possible. The present thesis tries to solve the environment reconstruction problem, identifying users of the roads (i.e. pedestrians and vehicles) by the use of Data Fusion techniques. The solution delivers a complete level based solution to the Data Fusion problem. It provides different tools for detecting as well as estimates the degree of danger that involve any detection. Presented algorithms represents a step forward in the ITS world, providing novel Data Fusion based algorithms that allow the detection and estimation of movement of pedestrians and vehicles in a robust and trustable way. To perform such a demanding task other information sources were needed: GPS, inertial systems and context information. Finally, it is important to remark that in the frame of the present thesis, the lack of detection and identification techniques based in radar laser resulted in the need to research and provide more innovative approaches, based in the use of laser scanner, able to detect and identify the different actors involved in the road environment. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Los accidentes de tráfico son un grave problema social y económico, cada año el coste tanto en vidas humanas como económico es incontable, por lo que cualquier acción que conlleve la reducción o eliminación de esta lacra es importante. Durante los últimos años se han hecho avances para mitigar el número de accidentes y reducir sus consecuencias. Estos esfuerzos han dado sus frutos, reduciendo el número de accidentes y sus víctimas. Sin embargo el número de heridos y muertos en accidentes de este tipo es aún muy alto, por lo que no hay que rebajar los esfuerzos encaminados a hacer desaparecer tan importante problema. Los recientes avances en tecnologías de la información han permitido la creación de sistemas de ayuda a la conducción cada vez más complejos, capaces de ayudar e incluso sustituir al conductor, permitiendo una conducción más segura y eficiente. Pero estos complejos sistemas requieren de los sensores más fiables, capaces de permitir reconstruir el entorno, identificar los distintos objetos que se encuentran en él e identificar los potenciales peligros. Los sensores disponibles en la actualidad han demostrado ser insuficientes para tan ardua tarea, debido a los enormes requerimientos que conlleva una aplicación de seguridad en carretera. Por lo tanto, combinar los diferentes sensores disponibles se antoja necesario para llegar a los niveles de eficiencia y confianza que requieren este tipo de aplicaciones. De esta forma, las limitaciones de cada sensor pueden ser superadas, gracias al uso combinado de los diferentes sensores, cada uno de ellos proporcionando información que complementa la obtenida por otros sistemas. Este tipo de aplicaciones se denomina aplicaciones de Fusión Sensorial. El presente trabajo busca aportar soluciones en el entorno de los vehículos inteligentes, mediante técnicas de fusión sensorial, a clásicos problemas relacionados con la seguridad vial. Se buscará combinar diferentes sensores y otras fuentes de información, para obtener un sistema fiable, capaz de satisfacer las exigentes demandas de este tipo de aplicaciones. Los estudios realizados y algoritmos propuestos están enmarcados en dos campos de investigación bien conocidos y populares. Los Sistemas Inteligentes de Transporte (ITS- por sus siglas en ingles- Intelligent Transportation Systems), marco en el que se centra la presente tesis, que engloba las diferentes tecnologías que durante los últimos años han permitido dotar a los sistemas de transporte de mejoras que aumentan la seguridad y eficiencia de los sistemas de transporte tradicionales, gracias a las novedades en el campo de las tecnologías de la información. Por otro lado las técnicas de Fusión Sensorial (DF -por sus siglas en ingles- Data Fusión) engloban las diferentes técnicas y procesos necesarios para combinar diferentes fuentes de información, permitiendo mejorar las prestaciones y dando fiabilidad a los sistemas finales. La presente tesis buscará el empleo de las técnicas de Fusión Sensorial para dar solución a problemas relacionados con Sistemas Inteligentes de Transporte. Los sensores escogidos para esta aplicación son un escáner láser y visión por computador. El primero es un sensor ampliamente conocido, que durante los últimos años ha comenzado a emplearse en el mundo de los ITS con unos excelentes resultados. El segundo de este conjunto de sensores es uno de los sistemas más empleados durante los últimos años, para dotar de cada vez más complejos y versátiles aplicaciones en el mundo de los ITS. Gracias a la visión por computador, aplicaciones tan necesarias para la seguridad como detección de señales de tráfico, líneas de la carreta, peatones, etcétera, que hace unos años parecía ciencia ficción, están cada vez más cerca. La aplicación que se presenta pretende dar solución al problema de reconstrucción de entornos viales, identificando a los principales usuarios de la carretera (vehículos y peatones) mediante técnicas de Fusión Sensorial. La solución implementada busca dar una completa solución a todos los niveles del proceso de fusión sensorial, proveyendo de las diferentes herramientas, no solo para detectar los otros usuarios, sino para dar una estimación del peligro que cada una de estas detecciones implica. Para lograr este propósito, además de los sensores ya comentados han sido necesarias otras fuentes de información, como sensores GPS, inerciales e información contextual. Los algoritmos presentados pretenden ser un importante paso adelante en el mundo de los Sistemas Inteligentes de Transporte, proporcionando novedosos algoritmos basados en tecnologías de Fusión Sensorial que permitirán detectar y estimar el movimiento de los peatones y vehículos de forma fiable y robusta. Finalmente hay que remarcar que en el marco de la presente tesis, la falta de sistemas de detección e identificación de obstáculos basados en radar láser provocó la necesidad de implementar novedosos algoritmos que detectasen e identificasen, en la medida de lo posible y pese a las limitaciones de la tecnología, los diferentes obstáculos que se pueden encontrar en la carretera basándose en este sensor

Seguimento ativo de agentes dinâmicos multivariados usando informação vectorial

Doutoramento em Engenharia MecânicaO objeto principal da presente tese é o estudo de sistemas avançados de segurança, no âmbito da segurança automóvel, baseando-se na previsão de movimentos e ações dos agentes externos. Esta tese propõe tratar os agentes como entidades dinâmicas, com motivações e constrangimentos próprios. Apresenta-se, para tal, novas técnicas de seguimento dos referidos agentes levando em linha de conta as suas especificidades. Em decorrência, estuda-se dedicadamente dois tipos de agentes: os veículos automóveis e os peões. Quanto aos veículos automóveis, propõe-se melhorar a capacidade de previsão de movimentos recorrendo a modelos avançados que representam corretamente os constrangimentos presentes nos veículos. Assim, foram desenvolvidos algoritmos avançados de seguimento de agentes com recurso a modelos de movimento não holonómicos. Estes algoritmos fazem uso de dados vectoriais de distância fornecidos por sensores de distância laser. Para os peões, devido à sua complexidade (designadamente a ausência de constrangimentos de movimentos) propõe-se que a análise da sua linguagem corporal permita detetar atempadamente possíveis intenções de movimentos. Assim, foram desenvolvidos algoritmos de perceção de pose de peões adaptados ao campo da segurança automóvel com recurso a uso de dados de distâncias 3D obtidos com uma câmara stereo. De notar que os diversos algoritmos foram testados em experiências realizadas em ambiente real.The main topic of this thesis is the study of advanced safety systems, in the field of automotive safety, based on the prediction of the movement and actions of external agents. This thesis proposes to treat the agents as dynamic entities with their own motivations as constraints. As so, new target tracking techniques are proposed taking into account the targets’ specificities. Therefore, two different types of agents are dedicatedly studied: automobile vehicles and pedestrians. For the automobile vehicles, a technique to improve motion prediction by the use of advanced motion models is proposed, these models will correctly represent the constrains that exist in this kind of vehicle. With this goal, advanced target tracking algorithms coupled with nonholonomic motion models were developed. These algorithms make use of vectorial range data supplied by laser range sensors. Concerning the pedestrians, due to the problem complexity (mainly due to the lack of any specific motion constraint), it is proposed that the analysis of the pedestrians body language will allow to detected early the pedestrian intentions and movements. As so, pedestrian pose estimation algorithms specially adapted to the field of automotive safety were developed; these algorithms use 3D point cloud data obtained with a stereo camera. The various algorithms were tested in experiments conducted in real conditions

There and Back Again: Self-supervised Multispectral Correspondence Estimation

Across a wide range of applications, from autonomous vehicles to medical imaging, multi-spectral images provide an opportunity to extract additional information not present in color images. One of the most important steps in making this information readily available is the accurate estimation of dense correspondences between different spectra. Due to the nature of cross-spectral images, most correspondence solving techniques for the visual domain are simply not applicable. Furthermore, most cross-spectral techniques utilize spectra-specific characteristics to perform the alignment. In this work, we aim to address the dense correspondence estimation problem in a way that generalizes to more than one spectrum. We do this by introducing a novel cycle-consistency metric that allows us to self-supervise. This, combined with our spectra-agnostic loss functions, allows us to train the same network across multiple spectra. We demonstrate our approach on the challenging task of dense RGB-FIR correspondence estimation. We also show the performance of our unmodified network on the cases of RGB-NIR and RGB-RGB, where we achieve higher accuracy than similar self-supervised approaches. Our work shows that cross-spectral correspondence estimation can be solved in a common framework that learns to generalize alignment across spectra