Data-centric Design and Training of Deep Neural Networks with Multiple Data Modalities for Vision-based Perception Systems

224 p.Los avances en visión artificial y aprendizaje automático han revolucionado la capacidad de construir sistemas que procesen e interpreten datos digitales, permitiéndoles imitar la percepción humana y abriendo el camino a un amplio rango de aplicaciones. En los últimos años, ambas disciplinas han logrado avances significativos,impulsadas por los progresos en las técnicas de aprendizaje profundo(deep learning). El aprendizaje profundo es una disciplina que utiliza redes neuronales profundas (DNNs, por sus siglas en inglés) para enseñar a las máquinas a reconocer patrones y hacer predicciones basadas en datos. Los sistemas de percepción basados en el aprendizaje profundo son cada vez más frecuentes en diversos campos, donde humanos y máquinas colaboran para combinar sus fortalezas.Estos campos incluyen la automoción, la industria o la medicina, donde mejorar la seguridad, apoyar el diagnóstico y automatizar tareas repetitivas son algunos de los objetivos perseguidos.Sin embargo, los datos son uno de los factores clave detrás del éxito de los algoritmos de aprendizaje profundo. La dependencia de datos limita fuertemente la creación y el éxito de nuevas DNN. La disponibilidad de datos de calidad para resolver un problema específico es esencial pero difícil de obtener, incluso impracticable,en la mayoría de los desarrollos. La inteligencia artificial centrada en datos enfatiza la importancia de usar datos de alta calidad que transmitan de manera efectiva lo que un modelo debe aprender. Motivada por los desafíos y la necesidad de los datos, esta tesis formula y valida cinco hipótesis sobre la adquisición y el impacto de los datos en el diseño y entrenamiento de las DNNs.Específicamente, investigamos y proponemos diferentes metodologías para obtener datos adecuados para entrenar DNNs en problemas con acceso limitado a fuentes de datos de gran escala. Exploramos dos posibles soluciones para la obtención de datos de entrenamiento, basadas en la generación de datos sintéticos. En primer lugar, investigamos la generación de datos sintéticos utilizando gráficos 3D y el impacto de diferentes opciones de diseño en la precisión de los DNN obtenidos. Además, proponemos una metodología para automatizar el proceso de generación de datos y producir datos anotados variados, mediante la replicación de un entorno 3D personalizado a partir de un archivo de configuración de entrada. En segundo lugar, proponemos una red neuronal generativa(GAN) que genera imágenes anotadas utilizando conjuntos de datos anotados limitados y datos sin anotaciones capturados en entornos no controlados

Why Don't You Clean Your Glasses? Perception Attacks with Dynamic Optical Perturbations

Camera-based autonomous systems that emulate human perception are increasingly being integrated into safety-critical platforms. Consequently, an established body of literature has emerged that explores adversarial attacks targeting the underlying machine learning models. Adapting adversarial attacks to the physical world is desirable for the attacker, as this removes the need to compromise digital systems. However, the real world poses challenges related to the "survivability" of adversarial manipulations given environmental noise in perception pipelines and the dynamicity of autonomous systems. In this paper, we take a sensor-first approach. We present EvilEye, a man-in-the-middle perception attack that leverages transparent displays to generate dynamic physical adversarial examples. EvilEye exploits the camera's optics to induce misclassifications under a variety of illumination conditions. To generate dynamic perturbations, we formalize the projection of a digital attack into the physical domain by modeling the transformation function of the captured image through the optical pipeline. Our extensive experiments show that EvilEye's generated adversarial perturbations are much more robust across varying environmental light conditions relative to existing physical perturbation frameworks, achieving a high attack success rate (ASR) while bypassing state-of-the-art physical adversarial detection frameworks. We demonstrate that the dynamic nature of EvilEye enables attackers to adapt adversarial examples across a variety of objects with a significantly higher ASR compared to state-of-the-art physical world attack frameworks. Finally, we discuss mitigation strategies against the EvilEye attack.Comment: 15 pages, 11 figure

Cooperative UAV–UGV autonomous power pylon inspection: an investigation of cooperative outdoor vehicle positioning architecture

Realizing autonomous inspection, such as that of power distribution lines, through unmanned aerial vehicle (UAV) systems is a key research domain in robotics. In particular, the use of autonomous and semi-autonomous vehicles to execute the tasks of an inspection process can enhance the efficacy and safety of the operation; however, many technical problems, such as those pertaining to the precise positioning and path following of the vehicles, robust obstacle detection, and intelligent control, must be addressed. In this study, an innovative architecture involving an unmanned aircraft vehicle (UAV) and an unmanned ground vehicle (UGV) was examined for detailed inspections of power lines. In the proposed strategy, each vehicle provides its position information to the other, which ensures a safe inspection process. The results of real-world experiments indicate a satisfactory performance, thereby demonstrating the feasibility of the proposed approach.This research was funded by National Counsel of Technological and Scientific Development of Brazil (CNPq). The authors thank the National Counsel of Technological and Scientific Development of Brazil (CNPq); Coordination for the Improvement of Higher Level People (CAPES); and the Brazilian Ministry of Science, Technology, Innovation, and Communication (MCTIC). The authors would also like express their deepest gratitude to Control Robotics for sharing the Pioneer P3 robot for the experiments. Thanks to Leticia Cantieri for editing the experiment video.info:eu-repo/semantics/publishedVersio

Comunicações ópticas por câmera para sistemas de assistência à condução

Communications, whatever its type, is a pillar of our modern society. More specifically, communications by visible light, that show numerous advantages, from electromagnetic spectral efficiency and regulation freedom to energy saving (since it combine illumination and communication). As such, the automotive world is interested in this technology, in particularly, its application into the Intelligent Transport System (ITS). The objective of this work relies on the study and development of a demonstrator able to support VLC communication means in V2V (Vehicle to Vehicle) scenario, making use of the LED luminaries already implemented in nowadays cars. Since the outdoor implementation is one of the requirements, reception based in OCC (Optical Camera Communication) is a viable solution in this conditions. Also the signal processing/decoding is performed by a CNN (Convolutional Neural Network), this type of algorithm shows a huge decoding flexibility and resilience, which benefits the transmission system performance. All the project was done in collaboration with the integrated circuits systems group of Instituto de Telecomunicações de Aveiro and Exatronic Lda company, based in Aveiro and specialized in innovation and investigation (I+I), engineering and manufacturing of electronics.As comunicações, qualquer que seja o seu tipo, mostram-se como um pilar fundamental para a sociedade. Especificamente as comunicações por luz visível, que apresentam inúmeras vantagens, desde a eficiência espectral e mais liberdade de regulamentação, até à energética pois alia duas caracteristicas distintas (iluminação e comunicação) numa só. Como tal, o mundo automóvel apresenta-se como um dos posíveis interessados na aplicação desta tecnologia, mais propriamente a aplicação como parte integrante do sistema inteligente de transportes (ITS). Este trabalho tem como objectivo o estudo e desenvolvimento de um demonstrador capaz de estabelecer um link de comunicação V2V (Vehicle to vehicle) por meio da modulação da luz visivel emitida pelas iluminárias LED já equipadas actualmente nos veículos. Sendo a implementação exterior um dos requerimentos deste sistema, a rececção através de OCC (Optical Camera Communication) mostra-se assim uma solução viável. Assim como o processamento do sinal recebido, que é efectuado por meio de CNNs (Convolutional Neural Networks), que mostram flexibilidade e resiliência, o que benefecia a capacidade do sistema de transmissão. Todo o projecto foi realizado em colaboração com o grupo de circuitos integrados do Instituto de Telecomunicações de Aveiro e a empresa Exatronic Lda, sediada em Aveiro, e especializada em inovação, investigação (I+I), engenharia e produção de eletrónica.Mestrado em Engenharia Eletrónica e Telecomunicaçõe

Face recognition for vehicle personalization

The objective of this dissertation is to develop a system of practical technologies to implement an illumination robust, consumer grade biometric system based on face recognition to be used in the automotive market. Most current face recognition systems are compromised in accuracy by ambient illumination changes. Especially outdoor applications including vehicle personalization pose the most challenging environment for face recognition. The point of this research is to investigate practical face recognition used for identity management in order to minimize algorithmic complexity while making the system robust to ambient illumination changes. We start this dissertation by proposing an end-to-end face recognition system using near infrared (NIR) spectrum. The advantage of NIR over visible light is that it is invisible to the human eyes while most CCD and CMOS imaging devices show reasonable response to NIR. Therefore, we can build an unobtrusive night-time vision system with active NIR illumination. In day time the active NIR illumination provides more controlled illumination condition. Next, we propose an end-to-end system with active NIR image differencing which takes the difference between successive image frames, one illuminated and one not illuminated, to make the system more robust on illumination changes. Furthermore, we addresses several aspects of the problem in active NIR image differencing which are motion artifact and noise in the difference frame, namely how to efficiently and more accurately align the illuminated frame and ambient frame, and how to combine information in the difference frame and the illuminated frame. Finally, we conclude the dissertation by citing the contributions of the research and discussing the avenues for future work.Ph.D

Driving in the Rain: A Survey toward Visibility Estimation through Windshields

Rain can significantly impair the driver’s sight and affect his performance when driving in wet conditions. Evaluation of driver visibility in harsh weather, such as rain, has garnered considerable research since the advent of autonomous vehicles and the emergence of intelligent transportation systems. In recent years, advances in computer vision and machine learning led to a significant number of new approaches to address this challenge. However, the literature is fragmented and should be reorganised and analysed to progress in this field. There is still no comprehensive survey article that summarises driver visibility methodologies, including classic and recent data-driven/model-driven approaches on the windshield in rainy conditions, and compares their generalisation performance fairly. Most ADAS and AD systems are based on object detection. Thus, rain visibility plays a key role in the efficiency of ADAS/AD functions used in semi- or fully autonomous driving. This study fills this gap by reviewing current state-of-the-art solutions in rain visibility estimation used to reconstruct the driver’s view for object detection-based autonomous driving. These solutions are classified as rain visibility estimation systems that work on (1) the perception components of the ADAS/AD function, (2) the control and other hardware components of the ADAS/AD function, and (3) the visualisation and other software components of the ADAS/AD function. Limitations and unsolved challenges are also highlighted for further research

Particle Filters for Colour-Based Face Tracking Under Varying Illumination

Automatic human face tracking is the basis of robotic and active vision systems used for facial feature analysis, automatic surveillance, video conferencing, intelligent transportation, human-computer interaction and many other applications. Superior human face tracking will allow future safety surveillance systems which monitor drowsy drivers, or patients and elderly people at the risk of seizure or sudden falls and will perform with lower risk of failure in unexpected situations. This area has actively been researched in the current literature in an attempt to make automatic face trackers more stable in challenging real-world environments. To detect faces in video sequences, features like colour, texture, intensity, shape or motion is used. Among these feature colour has been the most popular, because of its insensitivity to orientation and size changes and fast process-ability. The challenge of colour-based face trackers, however, has been dealing with the instability of trackers in case of colour changes due to the drastic variation in environmental illumination. Probabilistic tracking and the employment of particle filters as powerful Bayesian stochastic estimators, on the other hand, is increasing in the visual tracking field thanks to their ability to handle multi-modal distributions in cluttered scenes. Traditional particle filters utilize transition prior as importance sampling function, but this can result in poor posterior sampling. The objective of this research is to investigate and propose stable face tracker capable of dealing with challenges like rapid and random motion of head, scale changes when people are moving closer or further from the camera, motion of multiple people with close skin tones in the vicinity of the model person, presence of clutter and occlusion of face. The main focus has been on investigating an efficient method to address the sensitivity of the colour-based trackers in case of gradual or drastic illumination variations. The particle filter is used to overcome the instability of face trackers due to nonlinear and random head motions. To increase the traditional particle filter\u27s sampling efficiency an improved version of the particle filter is introduced that considers the latest measurements. This improved particle filter employs a new colour-based bottom-up approach that leads particles to generate an effective proposal distribution. The colour-based bottom-up approach is a classification technique for fast skin colour segmentation. This method is independent to distribution shape and does not require excessive memory storage or exhaustive prior training. Finally, to address the adaptability of the colour-based face tracker to illumination changes, an original likelihood model is proposed based of spatial rank information that considers both the illumination invariant colour ordering of a face\u27s pixels in an image or video frame and the spatial interaction between them. The original contribution of this work lies in the unique mixture of existing and proposed components to improve colour-base recognition and tracking of faces in complex scenes, especially where drastic illumination changes occur. Experimental results of the final version of the proposed face tracker, which combines the methods developed, are provided in the last chapter of this manuscript