Computer Vision for Fish Monitoring: Challenges and Possibilities

This master's thesis focuses on the evaluation and exploration of detection and tracking algorithms for fish in a dense underwater environment. The primary objectives were to achieve precise and accurate fish detection and to track fish over an extended period. The thesis explores the performance of two object detection algorithms, YOLOv4 and YOLOv8, as well as their integration with the DeepSORT tracking algorithm. The algorithms were trained and evaluated using a dataset collected from a densely populated underwater fish tank. The dataset was manually annotated using bounding box annotation techniques to accurately label the objects of interest. The results demonstrated the effectiveness of both YOLOv4 and YOLOv8 in detecting fish in densely populated environments. However, YOLOv8 achieved a significantly higher mAP50-95 score, indicating better localization and detection accuracy. It proved more adept at precisely locating the position of detected fish, leading to improved overall detection performance. In terms of fish tracking the combination of DeepSORT and YOLOv8 showed the best overall performance, as evidenced by higher MOTA and IDF1 scores, and lower MOTP scores. However, tracking individual fish over extended periods presented challenges due to occlusions and rapid trajectory changes, leading to a high number of identity switches. By evaluating and exploring the effectiveness of detection and tracking algorithms, this thesis contributes to the advancement of fish monitoring techniques in aquaculture. The findings provide valuable insights into the performance of YOLOv4 and YOLOv8 and the potential of DeepSORT for accurate and reliable fish detection and tracking. The results and methodologies presented in this study lay the groundwork for further research and development in the field, aiming to enhance fish welfare, optimize resource management, and improve efficiency in aquaculture practices

Detecting Invasive Insects Using Uncewed Aerial Vehicles and Variational Autoencoders

In this thesis, we use machine learning techniques to address limitations in our ability to monitor pest insect migrations. Invasive insect populations, such as the brown marmorated stink bug (BMSB), cause significant economic and environmental damages. In order to mitigate these damages, tracking BMSB migration is vital, but it also poses a challenge. The current state-of-the-art solution to track insect migrations is called mark-release-recapture. In mark-release-recapture, a researcher marks insects with a fluorescent powder, releases them back into the wild, and searches for the insects using ultra-violet flashlights at suspected migration destination locations. However, this involves a significant amount of labor and has a low recapture rate. By automating the insect search step, the recapture rate can be improved, reducing the amount of labor required in the process and improving the quality of the data. We propose a solution to the BMSB migration tracking problem using an unmanned aerial vehicle (UAV) to collect video data of the area of interest. Our system uses an ultra violet (UV) lighting array and digital cameras mounted on the bottom of the UAV, as well as artificial intelligence algorithms such as convolutional neural networks (CNN), and multiple hypotheses tracking (MHT) techniques. Specifically, we propose a novel computer vision method for insect detection using a Convolutional Variational Auto Encoder (CVAE). Our experimental results show that our system can detect BMSB with high precision and recall, outperforming the current state-of-the-art. Additionally, we associate insect observations using MHT, improving detection results and accurately counting real-world insects

Urban Traffic Monitoring from LIDAR Data with a Two-Level Marked Point Process Model

In this report we present a new object based hierarchical model for joint probabilistic extraction of vehicles and coherent vehicle groups - called traffic segments - in airborne and terrestrial LIDAR point clouds collected from crowded urban areas. Firstly, the 3D point set is segmented into terrain, vehicle, roof, vegetation and clutter classes. Then the points with the corresponding class labels and intensity values are projected to the ground plane. In the obtained 2D class and intensity maps we approximate the top view projections of vehicles by rectangles. Since our tasks are simultaneously the extraction of the rectangle population which describes the position, size and orientation of the vehicles and grouping the vehicles into the traffic segments, we propose a hierarchical, Two-Level Marked Point Process (L2MPP) model for the problem. The output vehicle and traffic segment configurations are extracted by an iterative stochastic optimization algorithm. We have tested the proposed method with real aerial and terrestrial LiDAR measurements. Our aerial data set contains 471 vehicles, and we provide quantitative object and pixel level comparions results versus two state-of-the-art solutions

Dynamic scene understanding: Pedestrian tracking from aerial devices.

Multiple Object Tracking (MOT) is the problem that involves following the trajectory of multiple objects in a sequence, generally a video. Pedestrians are among the most interesting subjects to track and recognize for many purposes such as surveillance, and safety. In the recent years, Unmanned Aerial Vehicles (UAV’s) have been viewed as a viable option for monitoring public areas, as they provide a low-cost method of data collection while covering large and difficult-to-reach areas. In this thesis, we present an online pedestrian tracking and re-identification from aerial devices framework. This framework is based on learning a compact directional statistic distribution (von-Mises-Fisher distribution) for each person ID using a deep convolutional neural network. The distribution characteristics are trained to be invariant to clothes appearances and to transformations. In real world scenarios, during deployment, new pedestrian and objects can appear in the scene and the model should detect them as Out Of Distribution (OOD). Thus, our frameworks also includes an OOD detection adopted from [16] called Virtual Outlier Synthetic (VOS), that detects OOD based on synthesising virtual outlier in the embedding space in an online manner. To validate, analyze and compare our approach, we use a large real benchmark data that contain detection tracking and identity annotations. These targets are captured at different viewing angles, different places, and different times by a ”DJI Phantom 4” drone. We validate the effectiveness of the proposed framework by evaluating their detection, tracking and long term identification performance as well as classification performance between In Distribution (ID) and OOD. We show that the the proposed methods in the framework can learn models that achieve their objectives

Geometric Graphs: Matching, Similarity, and Indexing

For many applications, such as drug discovery, road network analysis, and image processing, it is critical to study spatial properties of objects in addition to object relationships. Geometric graphs provide a suitable modeling framework for such applications, where vertices are located in some 2D space. As a result, searching for similar objects is tackled by estimating the similarity of the structure of different graphs. In this case, inexact graph matching approaches are typically employed. However, computing the optimal solution to the graph matching problem is proved to be a very complex task. In addition to this, approximate approaches face many problems such as poor scalability with respect to graph size and less tolerance to changes in graph structure or labels. In this thesis, we propose a framework to tackle the inexact graph matching problem for geometric graphs in 2D space. It consists of a pipeline of three components that we design to cope with the requirements of several application domains. The first component of our framework is an approach to estimate the similarity of vertices. It is based on the string edit distance and handles any labeling information assigned to the vertices and edges. Based on this, we build the second component of our framework. It consists of two algorithms to tackle the inexact graph matching problem. The first algorithm adopts a probabilistic scheme, where we propose a density function that estimates the probability of the correspondences between vertices of different graphs. Then, a match between the two graphs is computed utilizing the expectation maximization technique. The second graph matching algorithm follows a continuous optimization scheme to iteratively improve the match between two graphs. For this, we propose a vertex embedding approach so that the similarity of different vertices can be easily estimated by the Euclidean distance. The third component of our framework is a graph indexing structure, which helps to efficiently search a graph database for similar graphs. We propose several lower bound graph distances that are used to prune non-similar graphs and reduce the response time. Using representative geometric graphs extracted from a variety of applications domains, such as chemoinformatics, character recognition, road network analysis, and image processing, we show that our approach outperforms existing graph matching approaches in terms of matching quality, classification accuracy, and runtime

Sensor Fusion for Object Detection and Tracking in Autonomous Vehicles

Autonomous driving vehicles depend on their perception system to understand the environment and identify all static and dynamic obstacles surrounding the vehicle. The perception system in an autonomous vehicle uses the sensory data obtained from different sensor modalities to understand the environment and perform a variety of tasks such as object detection and object tracking. Combining the outputs of different sensors to obtain a more reliable and robust outcome is called sensor fusion. This dissertation studies the problem of sensor fusion for object detection and object tracking in autonomous driving vehicles and explores different approaches for utilizing deep neural networks to accurately and efficiently fuse sensory data from different sensing modalities. In particular, this dissertation focuses on fusing radar and camera data for 2D and 3D object detection and object tracking tasks. First, the effectiveness of radar and camera fusion for 2D object detection is investigated by introducing a radar region proposal algorithm for generating object proposals in a two-stage object detection network. The evaluation results show significant improvement in speed and accuracy compared to a vision-based proposal generation method. Next, radar and camera fusion is used for the task of joint object detection and depth estimation where the radar data is used in conjunction with image features to generate object proposals, but also provides accurate depth estimation for the detected objects in the scene. A fusion algorithm is also proposed for 3D object detection where where the depth and velocity data obtained from the radar is fused with the camera images to detect objects in 3D and also accurately estimate their velocities without requiring any temporal information. Finally, radar and camera sensor fusion is used for 3D multi-object tracking by introducing an end-to-end trainable and online network capable of tracking objects in real-time