The AAU Multimodal Annotation Toolboxes: Annotating Objects in Images and Videos

This tech report gives an introduction to two annotation toolboxes that enable the creation of pixel and polygon-based masks as well as bounding boxes around objects of interest. Both toolboxes support the annotation of sequential images in the RGB and thermal modalities. Each annotated object is assigned a classification tag, a unique ID, and one or more optional meta data tags. The toolboxes are written in C++ with the OpenCV and Qt libraries and are operated by using the visual interface and the extensive range of keyboard shortcuts. Pre-built binaries are available for Windows and MacOS and the tools can be built from source under Linux as well. So far, tens of thousands of frames have been annotated using the toolboxes.Comment: 6 pages, 10 figure

Effect of Annotation Errors on Drone Detection with YOLOv3

Following the recent advances in deep networks, object detection and tracking algorithms with deep learning backbones have been improved significantly; however, this rapid development resulted in the necessity of large amounts of annotated labels. Even if the details of such semi-automatic annotation processes for most of these datasets are not known precisely, especially for the video annotations, some automated labeling processes are usually employed. Unfortunately, such approaches might result with erroneous annotations. In this work, different types of annotation errors for object detection problem are simulated and the performance of a popular state-of-the-art object detector, YOLOv3, with erroneous annotations during training and testing stages is examined. Moreover, some inevitable annotation errors in CVPR-2020 Anti-UAV Challenge dataset is also examined in this manner, while proposing a solution to correct such annotation errors of this valuable data set.Comment: Best Paper Award at The 1st Anti-UAV Workshop & Challenge - CVPR Workshops, 202

Implementation of Unmanned aerial vehicles (UAVs) for assessment of transportation infrastructure - Phase II

Technological advances in unmanned aerial vehicle (UAV) technologies continue to enable these tools to become easier to use, more economical, and applicable for transportation-related operations, maintenance, and asset management while also increasing safety and decreasing cost. This Phase 2 project continued to test and evaluate five main UAV platforms with a combination of optical, thermal, and lidar sensors to determine how to implement them into MDOT workflows. Field demonstrations were completed at bridges, a construction site, road corridors, and along highways with data being processed and analyzed using customized algorithms and tools. Additionally, a cost-benefit analysis was conducted, comparing manual and UAV-based inspection methods. The project team also gave a series of technical demonstrations and conference presentations, enabling outreach to interested audiences who gained understanding of the potential implementation of this technology and the advanced research that MDOT is moving to implementation. The outreach efforts and research activities performed under this contract demonstrated how implementing UAV technologies into MDOT workflows can provide many benefits to MDOT and the motoring public; such as advantages in improved cost-effectiveness, operational management, and timely maintenance of Michigan’s transportation infrastructure

Rain Removal in Traffic Surveillance: Does it Matter?

Varying weather conditions, including rainfall and snowfall, are generally regarded as a challenge for computer vision algorithms. One proposed solution to the challenges induced by rain and snowfall is to artificially remove the rain from images or video using rain removal algorithms. It is the promise of these algorithms that the rain-removed image frames will improve the performance of subsequent segmentation and tracking algorithms. However, rain removal algorithms are typically evaluated on their ability to remove synthetic rain on a small subset of images. Currently, their behavior is unknown on real-world videos when integrated with a typical computer vision pipeline. In this paper, we review the existing rain removal algorithms and propose a new dataset that consists of 22 traffic surveillance sequences under a broad variety of weather conditions that all include either rain or snowfall. We propose a new evaluation protocol that evaluates the rain removal algorithms on their ability to improve the performance of subsequent segmentation, instance segmentation, and feature tracking algorithms under rain and snow. If successful, the de-rained frames of a rain removal algorithm should improve segmentation performance and increase the number of accurately tracked features. The results show that a recent single-frame-based rain removal algorithm increases the segmentation performance by 19.7% on our proposed dataset, but it eventually decreases the feature tracking performance and showed mixed results with recent instance segmentation methods. However, the best video-based rain removal algorithm improves the feature tracking accuracy by 7.72%.Comment: Published in IEEE Transactions on Intelligent Transportation System

Anomaly detection in fleet service vehicles: improving object segmentation

Dissertação de mestrado integrado em Engenharia InformáticaThe present dissertation is inserted in a BOSCH project in which the global focus is au tonomous driving. The project is divided in multiple phases, being the main focus of this dissertation object detection and segmentation inside fleet service vehicles. The objective is to detect/segment objects and dirt left inside a vehicle, warning the commuter if they forgot an object inside or the administrators if the vehicles need to be cleaned. To train the models, BOSCH provided an initial dataset containing a small set of annotated images. This dataset contains pictures of a vehicles cockpit with many diverse objects and dirt. One of the goals for BOSCH is to increment this dataset with more images. Hence, in this project several state of the art segmentation methods were thoroughly studied and analysed, with two of them being selected for further exploration: DeepExtremeCut and FgSegNet v2. The main objective is to see to what extent can these methods be used in a semi automatic process to segment more images, thereby increasing the initial dataset. DeepExtremeCut works by using a framework in which, after the model is trained, it allows us to click on four extreme points in the desired object, producing the segmentation. This method produced reliable segmentations, however it requires human intervention both for the initial segmentation and verification of the output. Hence, it was not regarded as a good solution for a future augmentation of the BOSCH dataset. Regarding FgSegNet v2, this later method does not require any initial annotation of the input images. Under this approach only a final verification and possible rectification is required. Therefore, this method meets the requirements defined by BOSCH for a dataset expansion solution. An ablation study is also presented for FgSegNet v2, analysing its three stages: (i) Encoder, (ii) Feature Pooling Module and (iii) Decoder. The result of this study is a proposal of a variation of the aforementioned method called Mod FgSegNet. It was also compared with state of the art methods in public datasets. Three datasets are used for testing: CDNet2014, SBI2015 and CityScapes. In CDNet2014 we got an overall improvement when compared to the state of the art, particularly in the LowFrameRate subset. Regarding SBI2015 the overall results are lower in comparison with the top state of art, while in CityScapes some promising results are presented.A presente dissertação está inserida num projeto da BOSCH, em que o foco global é a condução autónoma. Este projeto foi dividido em múltiplas fases, sendo o foco principal desta dissertação e a detecção e a segmentação de objetos. O objetivo é detectar / segmentar objetos e lixo deixados no interior de um veículo, avisando o passageiro se ele se esqueceu de algum objeto no interior ou os administradores se os veículos precisam de ser limpos. Para lidar com este tópico, vários métodos de segmentação do estado da arte foram exaustivamente estudados e analisados, nos quais dois deles em particular foram explorados, ou seja, o DeepExtremeCut e o FgSegNet. Algumas melhorias foram feitas no desempenho deste último, permitindo um potencial aumento semiautomático no tamanho de um dataset fornecido pela BOSCH, uma vez que não possuía imagens suficientes. Este dataset contém fotos da cabine de um veículo com diferentes objetos e lixo. O DeepExtremeCut funciona através do uso de uma framework no qual, após o treino do modelo, permite clicar em quatro pontos extremos do objeto desejado, produzindo a segmentação. Este método produz segmentações confiáveis, embora não corresponda a uma segmentação automática, visto que existe a necessidade de selecionar todos os objetos, ainda pode ser útil quando a segmentação automática de um método diferente não estiver a funcionar em casos particulares. Em relação ao FgSegNet v2, e apresentado um estudo de ablação, sendo feita uma análise das suas três etapas: (i) Encoder, (ii) Feate Pooling Module e (iii) Decoder. O resultado deste estudo e uma proposta de variação do referido método no dataset da BOSCH, de forma a utilizá-lo potencialmente em vários projetos dentro da empresa, chamado Mod_FgSegNet. Também foi comparado com métodos do estado de arte em datasets públicos. Os três datasets usados para teste são: CDNet 2014, SBI2015 e CityScapes. No CDNet2014, obtivemos uma melhoria geral em comparação com o estado da arte, principalmente no subconjunto LowFrameRate. Em relação ao ao SBI2015, os resultados gerais foram inferiores em comparação com o estado da arte de ponta, enquanto que no CityScapes alguns resultados promissores foram apresentados