Bounding Box-Free Instance Segmentation Using Semi-Supervised Learning for Generating a City-Scale Vehicle Dataset

Vehicle classification is a hot computer vision topic, with studies ranging from ground-view up to top-view imagery. In remote sensing, the usage of top-view images allows for understanding city patterns, vehicle concentration, traffic management, and others. However, there are some difficulties when aiming for pixel-wise classification: (a) most vehicle classification studies use object detection methods, and most publicly available datasets are designed for this task, (b) creating instance segmentation datasets is laborious, and (c) traditional instance segmentation methods underperform on this task since the objects are small. Thus, the present research objectives are: (1) propose a novel semi-supervised iterative learning approach using GIS software, (2) propose a box-free instance segmentation approach, and (3) provide a city-scale vehicle dataset. The iterative learning procedure considered: (1) label a small number of vehicles, (2) train on those samples, (3) use the model to classify the entire image, (4) convert the image prediction into a polygon shapefile, (5) correct some areas with errors and include them in the training data, and (6) repeat until results are satisfactory. To separate instances, we considered vehicle interior and vehicle borders, and the DL model was the U-net with the Efficient-net-B7 backbone. When removing the borders, the vehicle interior becomes isolated, allowing for unique object identification. To recover the deleted 1-pixel borders, we proposed a simple method to expand each prediction. The results show better pixel-wise metrics when compared to the Mask-RCNN (82% against 67% in IoU). On per-object analysis, the overall accuracy, precision, and recall were greater than 90%. This pipeline applies to any remote sensing target, being very efficient for segmentation and generating datasets.Comment: 38 pages, 10 figures, submitted to journa

Pedestrian Detection in Crowded Environments through Bayesian Prediction of Sequential Probability Matrices

In order to safely navigate populated environments, an autonomous vehicle must be able to detect human shapes using its sensory systems, so that it can properly avoid a collision. In this paper, we introduce a Bayesian approach to the Viola-Jones algorithm, as a method to automatically detect pedestrians in image sequences. We present a probabilistic interpretation of the basic execution of the original tool and develop a technique to produce approximate convolutions of probability matrices with multiple local maxima

Detection, Quantification and Classification of Ripened Tomatoes: A Comparative Analysis of Image Processing and Machine Learning

In this paper, specifically for detection of ripe/unripe tomatoes with/without defects in the crop field, two distinct methods are described and compared. One is a machine learning approach, known as ‘Cascaded Object Detector’ and the other is a composition of traditional customized methods, individually known as ‘Colour Transformation’, ‘Colour Segmentation’ and ‘Circular Hough Transformation’. The (Viola Jones) Cascaded Object Detector generates ‘histogram of oriented gradient’ (HOG) features to detect tomatoes. For ripeness checking, the RGB mean is calculated with a set of rules. However, for traditional methods, color thresholding is applied to detect tomatoes either from a natural or solid background and RGB colour is adjusted to identify ripened tomatoes. In this work, Colour Segmentation is applied in the detection of tomatoes with defects, which has not previously been applied under machine learning techniques. The function modules of this algorithm are fed formatted images, captured by a camera mounted on a mobile robot. This robot was designed, built and operated in a tomato field to identify and quantify both green and ripened tomatoes as well as to detect damaged/blemished ones. This algorithm is shown to be optimally feasible for any micro-controller based miniature electronic devices in terms of its run time complexity of O(n3) for traditional method in best and average cases. Comparisons show that the accuracy of the machine learning method is 95%, better than that of the Colour Segmentation Method using MATLAB. This result is potentially significant for farmers in crop fields to identify the condition of tomatoes quickly

Pedestrian and Vehicle Detection in Autonomous Vehicle Perception Systems—A Review

Autonomous Vehicles (AVs) have the potential to solve many traffic problems, such as accidents, congestion and pollution. However, there are still challenges to overcome, for instance, AVs need to accurately perceive their environment to safely navigate in busy urban scenarios. The aim of this paper is to review recent articles on computer vision techniques that can be used to build an AV perception system. AV perception systems need to accurately detect non-static objects and predict their behaviour, as well as to detect static objects and recognise the information they are providing. This paper, in particular, focuses on the computer vision techniques used to detect pedestrians and vehicles. There have been many papers and reviews on pedestrians and vehicles detection so far. However, most of the past papers only reviewed pedestrian or vehicle detection separately. This review aims to present an overview of the AV systems in general, and then review and investigate several detection computer vision techniques for pedestrians and vehicles. The review concludes that both traditional and Deep Learning (DL) techniques have been used for pedestrian and vehicle detection; however, DL techniques have shown the best results. Although good detection results have been achieved for pedestrians and vehicles, the current algorithms still struggle to detect small, occluded, and truncated objects. In addition, there is limited research on how to improve detection performance in difficult light and weather conditions. Most of the algorithms have been tested on well-recognised datasets such as Caltech and KITTI; however, these datasets have their own limitations. Therefore, this paper recommends that future works should be implemented on more new challenging datasets, such as PIE and BDD100K.EPSRC DTP PhD studentshi

Towards Autonomy: Cost-effective Scheduling for Long-range Autonomous Valet Parking (LAVP)

Continuous and effective developments in Autonomous Vehicles (AVs) are happening on daily basis. Industries nowadays, are interested in introducing less costly and highly controllable AVs to public. Current so-called AVP solutions are still limited to a very short range (e.g., even only work at the entrance of car parks). This paper proposes a parking scheduling scheme for long-range AVP (LAVP) case, by considering mobility of Autonomous Vehicles (AVs), fuel consumption and journey time. In LAVP, Car Parks (CPs) are used to accommodate increasing numbers of AVs, and placed outside city center, in order to avoid traffic congestions and ensure road safety in public places. Furthermore, with positioning of reference points to guide user-centric long-term driving and drop-off/pick-up passengers, simulation results under the Helsinki city scenario shows the benefits of LAVP. The advantage of LAVP system is also reflected through both analysis and simulation

Automatic vehicle detection and tracking in aerial video

This thesis is concerned with the challenging tasks of automatic and real-time vehicle detection and tracking from aerial video. The aim of this thesis is to build an automatic system that can accurately localise any vehicles that appear in aerial video frames and track the target vehicles with trackers. Vehicle detection and tracking have many applications and this has been an active area of research during recent years; however, it is still a challenge to deal with certain realistic environments. This thesis develops vehicle detection and tracking algorithms which enhance the robustness of detection and tracking beyond the existing approaches. The basis of the vehicle detection system proposed in this thesis has different object categorisation approaches, with colour and texture features in both point and area template forms. The thesis also proposes a novel Self-Learning Tracking and Detection approach, which is an extension to the existing Tracking Learning Detection (TLD) algorithm. There are a number of challenges in vehicle detection and tracking. The most difficult challenge of detection is distinguishing and clustering the target vehicle from the background objects and noises. Under certain conditions, the images captured from Unmanned Aerial Vehicles (UAVs) are also blurred; for example, turbulence may make the vehicle shake during flight. This thesis tackles these challenges by applying integrated multiple feature descriptors for real-time processing. In this thesis, three vehicle detection approaches are proposed: the HSV-GLCM feature approach, the ISM-SIFT feature approach and the FAST-HoG approach. The general vehicle detection approaches used have highly flexible implicit shape representations. They are based on training samples in both positive and negative sets and use updated classifiers to distinguish the targets. It has been found that the detection results attained by using HSV-GLCM texture features can be affected by blurring problems; the proposed detection algorithms can further segment the edges of the vehicles from the background. Using the point descriptor feature can solve the blurring problem, however, the large amount of information contained in point descriptors can lead to processing times that are too long for real-time applications. So the FAST-HoG approach combining the point feature and the shape feature is proposed. This new approach is able to speed up the process that attains the real-time performance. Finally, a detection approach using HoG with the FAST feature is also proposed. The HoG approach is widely used in object recognition, as it has a strong ability to represent the shape vector of the object. However, the original HoG feature is sensitive to the orientation of the target; this method improves the algorithm by inserting the direction vectors of the targets. For the tracking process, a novel tracking approach was proposed, an extension of the TLD algorithm, in order to track multiple targets. The extended approach upgrades the original system, which can only track a single target, which must be selected before the detection and tracking process. The greatest challenge to vehicle tracking is long-term tracking. The target object can change its appearance during the process and illumination and scale changes can also occur. The original TLD feature assumed that tracking can make errors during the tracking process, and the accumulation of these errors could cause tracking failure, so the original TLD proposed using a learning approach in between the tracking and the detection by adding a pair of inspectors (positive and negative) to constantly estimate errors. This thesis extends the TLD approach with a new detection method in order to achieve multiple-target tracking. A Forward and Backward Tracking approach has been proposed to eliminate tracking errors and other problems such as occlusion. The main purpose of the proposed tracking system is to learn the features of the targets during tracking and re-train the detection classifier for further processes. This thesis puts particular emphasis on vehicle detection and tracking in different extreme scenarios such as crowed highway vehicle detection, blurred images and changes in the appearance of the targets. Compared with currently existing detection and tracking approaches, the proposed approaches demonstrate a robust increase in accuracy in each scenario

Efficient resource allocation for automotive active vision systems

Individual mobility on roads has a noticeable impact upon peoples' lives, including traffic accidents resulting in severe, or even lethal injuries. Therefore the main goal when operating a vehicle is to safely participate in road-traffic while minimising the adverse effects on our environment. This goal is pursued by road safety measures ranging from safety-oriented road design to driver assistance systems. The latter require exteroceptive sensors to acquire information about the vehicle's current environment. In this thesis an efficient resource allocation for automotive vision systems is proposed. The notion of allocating resources implies the presence of processes that observe the whole environment and that are able to effeciently direct attentive processes. Directing attention constitutes a decision making process dependent upon the environment it operates in, the goal it pursues, and the sensor resources and computational resources it allocates. The sensor resources considered in this thesis are a subset of the multi-modal sensor system on a test vehicle provided by Audi AG, which is also used to evaluate our proposed resource allocation system. This thesis presents an original contribution in three respects. First, a system architecture designed to efficiently allocate both high-resolution sensor resources and computational expensive processes based upon low-resolution sensor data is proposed. Second, a novel method to estimate 3-D range motion, e cient scan-patterns for spin image based classifiers, and an evaluation of track-to-track fusion algorithms present contributions in the field of data processing methods. Third, a Pareto efficient multi-objective resource allocation method is formalised, implemented, and evaluated using road traffic test sequences

En Introduksjon til Kunstig Syn i Autonom Kjøring

Autonom kjøring er en av de fremtredende teknologiene i dagens samfunn. Et bredt spekter av applikasjoner bruker derfor denne teknologien for fordelene den gir. For eksempel vil en autonom kjørende robot frigjøre arbeidskraft og øke produktiviteten i bransjer som krever rask transport. For å oppnå disse fordelene krever det imidlertid utvikling av pålitelig og nøyaktig programvare og algoritmer som skal implementeres i disse autonome kjøresytemene. Ettersom dette feltet har vokst gjennom årene, har forskjellige selskaper implementert denne teknologien med stor suksess. Dermed gjør det økte fokuset på autonom kjøre teknologi dette til et aktuelt tema å forske på. Siden utvikling av et autonomt kjøresystem er et krevende tema, fokuserer dette prosjektet kun på hvordan kunstig syn kan brukes i autonome kjøresystemer. Først og fremst utvikles en kunstig syns basert programvare for autonom kjøring. Programvaren er først implementert på et lite forhåndslaget kjøretøy i bok størrelse. Dette systemet brukes deretter til å teste programvarens funksjonalitet. Autonome kjørefunksjoner som fungerer tilfredsstillende på det lille test kjøretøyet blir også testet på et større kjøretøy for å se om programvaren fungerer for andre systemer. Videre er den en utviklede programvaren begrenset til enkelte autonome kjørehandlinger. Dette inkluderer handlinger som å stoppe når en hindring eller et stoppskilt er oppdaget, kjøring på en enkel vei og parkering. Selv om dette bare er noen få autonome kjøre funksjoner, er de grunnleggende operasjoner som kan gjøre det autonome kjøresystemet allerede anvendelig for forskjellige brukstilfeller. Ulike kunstig syn metode for gjenstands deteksjon har blitt implementert for å oppdage ulike typer gjenstander som hindringer og skilt for å bestemme kjøretøyets miljø. Programvaren inkluderer også bruk av en linje deteksjonsmetode for å oppdage vei- og parkerings linjer som brukes til å sentrere og parkere kjøretøyet. Dessuten skapes et fuglebilde av den fysiske verden fra kamera bilder som skal brukes som et miljøkart for å planlegge den mest optimale rute i forskjellige scenarier. Til slutt blir disse implementeringene kombinert for å bygge kjørelogikken til kjøretøyet, noe som gjør det i stand til å utføre kjørehandlingene nevnt i forrige avsnitt. Ved bruk av den utviklede programvaren for kjøreoppgave, deteksjon av hindringer, viste resultatet at selv om de faktiske hindringene ble oppdaget, var det scenarier der blokkader ble oppdaget selv om det ikke var noen. På den annen side var den utviklede funksjonen med å stoppe når et stoppskilt blir oppdaget svært nøyaktig og pålitelig ettersom den utførte som forventet. Når det gjelder de resterende to implementerte handlingene, sentrering og parkering av kjøretøyet, slet systemet med å oppnå et lovende resultat. Til tross for det viste de fysiske valideringstestene uten bruk av kjøretøymodell positive resultater, men med mindre avvik fra ønsket resultat. Samlet sett har programvaren potensial for å bli anvendelig i mer krevende scenarier, men det er behov for videre utvikling for å fikse noen problemområder først.Autonomous driving is one of the rising technology in today’s society. Thus, a wide range of applications uses this technology for the benefits it yields. For instance, an autonomous driving robot will free up the labor force and increase productivity in industries that require rapid transportation. However, to gain these benefits, it requires the development of reliable and accurate software and algorithms to be implemented in these autonomous driving systems. As this field has been growing over the years, different companies have implemented this technology with great success. Thus, the increased focus on autonomous driving technology makes this a relevant topic to perform research on. As developing an autonomous driving system is a demanding topic, this project focuses solely on how computer vision can be used in autonomous driving systems. First and foremost, a computer-vision based autonomous driving software is developed. The software is first imple- mented on a small premade book-size vehicle. This system is then used to test the software’s functionality. Autonomous driving functions that perform satisfactorily on the small test vehicle are also tested on a larger vehicle to see if the software works for other systems. Furthermore, the developed software is limited to some autonomous driving actions. This includes actions such as stopping when a hindrance or a stop sign is detected, driving on a simple road, and parking. Although these are only a few autonomous driving actions, they are fundamental operations that can make the autonomous driving system already applicable to different use cases. Different computer vision methods for object detection have been implemented for detecting different types of objects such as hindrances and signs to determine the vehicle’s environment. The software also includes the usage of a line detection method for detecting road and parking lines that are used for centering and parking the vehicle. Moreover, a bird-view of the physical world is created from the camera output to be used as an environment map to plan the most optimal path in different scenarios. Finally, these implementations are combined to build the driving logic of the vehicle, making it able to perform the driving actions mentioned in the previous paragraph. When utilizing the developed software for the driving task, hindrance detection, the result showed that although the actual hindrances were detected, there were scenarios where block- ades were detected even though there were none. On the other hand, the developed function of stopping when a stop sign is detected was highly accurate and reliable as it performed as expected. With regard to the remaining two implemented actions, centering and parking the vehicle, the system struggled to achieve a promising result. Despite that, the physical validation tests without the use of a vehicle model showed positive outcomes although with minor deviation from the desired result. Overall, the software showed potential to be developed even further to be applicable in more demanding scenarios, however, the current issues must be addressed first

En Introduksjon til Kunstig Syn i Autonom Kjøring

Autonom kjøring er en av de fremtredende teknologiene i dagens samfunn. Et bredt spekter av applikasjoner bruker derfor denne teknologien for fordelene den gir. For eksempel vil en autonom kjørende robot frigjøre arbeidskraft og øke produktiviteten i bransjer som krever rask transport. For å oppnå disse fordelene krever det imidlertid utvikling av pålitelig og nøyaktig programvare og algoritmer som skal implementeres i disse autonome kjøresytemene. Ettersom dette feltet har vokst gjennom årene, har forskjellige selskaper implementert denne teknologien med stor suksess. Dermed gjør det økte fokuset på autonom kjøre teknologi dette til et aktuelt tema å forske på. Siden utvikling av et autonomt kjøresystem er et krevende tema, fokuserer dette prosjektet kun på hvordan kunstig syn kan brukes i autonome kjøresystemer. Først og fremst utvikles en kunstig syns basert programvare for autonom kjøring. Programvaren er først implementert på et lite forhåndslaget kjøretøy i bok størrelse. Dette systemet brukes deretter til å teste programvarens funksjonalitet. Autonome kjørefunksjoner som fungerer tilfredsstillende på det lille test kjøretøyet blir også testet på et større kjøretøy for å se om programvaren fungerer for andre systemer. Videre er den en utviklede programvaren begrenset til enkelte autonome kjørehandlinger. Dette inkluderer handlinger som å stoppe når en hindring eller et stoppskilt er oppdaget, kjøring på en enkel vei og parkering. Selv om dette bare er noen få autonome kjøre funksjoner, er de grunnleggende operasjoner som kan gjøre det autonome kjøresystemet allerede anvendelig for forskjellige brukstilfeller. Ulike kunstig syn metode for gjenstands deteksjon har blitt implementert for å oppdage ulike typer gjenstander som hindringer og skilt for å bestemme kjøretøyets miljø. Programvaren inkluderer også bruk av en linje deteksjonsmetode for å oppdage vei- og parkerings linjer som brukes til å sentrere og parkere kjøretøyet. Dessuten skapes et fuglebilde av den fysiske verden fra kamera bilder som skal brukes som et miljøkart for å planlegge den mest optimale rute i forskjellige scenarier. Til slutt blir disse implementeringene kombinert for å bygge kjørelogikken til kjøretøyet, noe som gjør det i stand til å utføre kjørehandlingene nevnt i forrige avsnitt. Ved bruk av den utviklede programvaren for kjøreoppgave, deteksjon av hindringer, viste resultatet at selv om de faktiske hindringene ble oppdaget, var det scenarier der blokkader ble oppdaget selv om det ikke var noen. På den annen side var den utviklede funksjonen med å stoppe når et stoppskilt blir oppdaget svært nøyaktig og pålitelig ettersom den utførte som forventet. Når det gjelder de resterende to implementerte handlingene, sentrering og parkering av kjøretøyet, slet systemet med å oppnå et lovende resultat. Til tross for det viste de fysiske valideringstestene uten bruk av kjøretøymodell positive resultater, men med mindre avvik fra ønsket resultat. Samlet sett har programvaren potensial for å bli anvendelig i mer krevende scenarier, men det er behov for videre utvikling for å fikse noen problemområder først.Autonomous driving is one of the rising technology in today's society. Thus, a wide range of applications uses this technology for the benefits it yields. For instance, an autonomous driving robot will free up the labor force and increase productivity in industries that require rapid transportation. However, to gain these benefits, it requires the development of reliable and accurate software and algorithms to be implemented in these autonomous driving systems. As this field has been growing over the years, different companies have implemented this technology with great success. Thus, the increased focus on autonomous driving technology makes this a relevant topic to perform research on. As developing an autonomous driving system is a demanding topic, this project focuses solely on how computer vision can be used in autonomous driving systems. First and foremost, a computer-vision based autonomous driving software is developed. The software is first implemented on a small premade book-size vehicle. This system is then used to test the software's functionality. Autonomous driving functions that perform satisfactorily on the small test vehicle are also tested on a larger vehicle to see if the software works for other systems. Furthermore, the developed software is limited to some autonomous driving actions. This includes actions such as stopping when a hindrance or a stop sign is detected, driving on a simple road, and parking. Although these are only a few autonomous driving actions, they are fundamental operations that can make the autonomous driving system already applicable to different use cases. Different computer vision methods for object detection have been implemented for detecting different types of objects such as hindrances and signs to determine the vehicle's environment. The software also includes the usage of a line detection method for detecting road and parking lines that are used for centering and parking the vehicle. Moreover, a bird-view of the physical world is created from the camera output to be used as an environment map to plan the most optimal path in different scenarios. Finally, these implementations are combined to build the driving logic of the vehicle, making it able to perform the driving actions mentioned in the previous paragraph. When utilizing the developed software for the driving task, hindrance detection, the result showed that although the actual hindrances were detected, there were scenarios where blockades were detected even though there were none. On the other hand, the developed function of stopping when a stop sign is detected was highly accurate and reliable as it performed as expected. With regard to the remaining two implemented actions, centering and parking the vehicle, the system struggled to achieve a promising result. Despite that, the physical validation tests without the use of a vehicle model showed positive outcomes although with minor deviation from the desired result. Overall, the software showed potential to be developed even further to be applicable in more demanding scenarios, however, the current issues must be addressed first