TractorEYE: Vision-based Real-time Detection for Autonomous Vehicles in Agriculture

Agricultural vehicles such as tractors and harvesters have for decades been able to navigate automatically and more efficiently using commercially available products such as auto-steering and tractor-guidance systems. However, a human operator is still required inside the vehicle to ensure the safety of vehicle and especially surroundings such as humans and animals. To get fully autonomous vehicles certified for farming, computer vision algorithms and sensor technologies must detect obstacles with equivalent or better than human-level performance. Furthermore, detections must run in real-time to allow vehicles to actuate and avoid collision.This thesis proposes a detection system (TractorEYE), a dataset (FieldSAFE), and procedures to fuse information from multiple sensor technologies to improve detection of obstacles and to generate a map. TractorEYE is a multi-sensor detection system for autonomous vehicles in agriculture. The multi-sensor system consists of three hardware synchronized and registered sensors (stereo camera, thermal camera and multi-beam lidar) mounted on/in a ruggedized and water-resistant casing. Algorithms have been developed to run a total of six detection algorithms (four for rgb camera, one for thermal camera and one for a Multi-beam lidar) and fuse detection information in a common format using either 3D positions or Inverse Sensor Models. A GPU powered computational platform is able to run detection algorithms online. For the rgb camera, a deep learning algorithm is proposed DeepAnomaly to perform real-time anomaly detection of distant, heavy occluded and unknown obstacles in agriculture. DeepAnomaly is -- compared to a state-of-the-art object detector Faster R-CNN -- for an agricultural use-case able to detect humans better and at longer ranges (45-90m) using a smaller memory footprint and 7.3-times faster processing. Low memory footprint and fast processing makes DeepAnomaly suitable for real-time applications running on an embedded GPU. FieldSAFE is a multi-modal dataset for detection of static and moving obstacles in agriculture. The dataset includes synchronized recordings from a rgb camera, stereo camera, thermal camera, 360-degree camera, lidar and radar. Precise localization and pose is provided using IMU and GPS. Ground truth of static and moving obstacles (humans, mannequin dolls, barrels, buildings, vehicles, and vegetation) are available as an annotated orthophoto and GPS coordinates for moving obstacles. Detection information from multiple detection algorithms and sensors are fused into a map using Inverse Sensor Models and occupancy grid maps. This thesis presented many scientific contribution and state-of-the-art within perception for autonomous tractors; this includes a dataset, sensor platform, detection algorithms and procedures to perform multi-sensor fusion. Furthermore, important engineering contributions to autonomous farming vehicles are presented such as easily applicable, open-source software packages and algorithms that have been demonstrated in an end-to-end real-time detection system. The contributions of this thesis have demonstrated, addressed and solved critical issues to utilize camera-based perception systems that are essential to make autonomous vehicles in agriculture a reality

Lidar-based Obstacle Detection and Recognition for Autonomous Agricultural Vehicles

Today, agricultural vehicles are available that can drive autonomously and follow exact route plans more precisely than human operators. Combined with advancements in precision agriculture, autonomous agricultural robots can reduce manual labor, improve workflow, and optimize yield. However, as of today, human operators are still required for monitoring the environment and acting upon potential obstacles in front of the vehicle. To eliminate this need, safety must be ensured by accurate and reliable obstacle detection and avoidance systems.In this thesis, lidar-based obstacle detection and recognition in agricultural environments has been investigated. A rotating multi-beam lidar generating 3D point clouds was used for point-wise classification of agricultural scenes, while multi-modal fusion with cameras and radar was used to increase performance and robustness. Two research perception platforms were presented and used for data acquisition. The proposed methods were all evaluated on recorded datasets that represented a wide range of realistic agricultural environments and included both static and dynamic obstacles.For 3D point cloud classification, two methods were proposed for handling density variations during feature extraction. One method outperformed a frequently used generic 3D feature descriptor, whereas the other method showed promising preliminary results using deep learning on 2D range images. For multi-modal fusion, four methods were proposed for combining lidar with color camera, thermal camera, and radar. Gradual improvements in classification accuracy were seen, as spatial, temporal, and multi-modal relationships were introduced in the models. Finally, occupancy grid mapping was used to fuse and map detections globally, and runtime obstacle detection was applied on mapped detections along the vehicle path, thus simulating an actual traversal.The proposed methods serve as a first step towards full autonomy for agricultural vehicles. The study has thus shown that recent advancements in autonomous driving can be transferred to the agricultural domain, when accurate distinctions are made between obstacles and processable vegetation. Future research in the domain has further been facilitated with the release of the multi-modal obstacle dataset, FieldSAFE

Deep Learning Based Parking Vacancy Detection for Smart Cities

Parking shortage is a major problem in modern cities. Drivers cruising in search of a parking space directly translate into frustration, traffic congestion, and excessive carbon emission. We introduce a simple and effective deep learning-based parking space notification (PSN) system to inform drivers of new parking availabilities and re-occupancy of the freed spaces. Our system is particularly designed to target areas with severe parking shortages (i.e., nearly all parking spaces are occupied), a situation that allows us to convert the problem of detecting parking vacancies into recognizing vehicles leaving from their stationary positions. Our PSN system capitalizes on a calibrated Mask R-CNN model and a unique adaptation of the IoU concept to track the changes of vehicle positions in a video stream. We evaluated PSN using videos from a CCTV camera installed at a private parking lot and publicly available YouTube videos. The PSN system successfully captured all new parking vacancies arising from leaving vehicles with no false positive detections. Prompt notification messages were sent to users via cloud messaging services

Hybrid Ventilation System and Soft-Sensors for Maintaining Indoor Air Quality and Thermal Comfort in Buildings

Maintaining both indoor air quality (IAQ) and thermal comfort in buildings along with optimized energy consumption is a challenging problem. This investigation presents a novel design for hybrid ventilation system enabled by predictive control and soft-sensors to achieve both IAQ and thermal comfort by combining predictive control with demand controlled ventilation (DCV). First, we show that the problem of maintaining IAQ, thermal comfort and optimal energy is a multi-objective optimization problem with competing objectives, and a predictive control approach is required to smartly control the system. This leads to many implementation challenges which are addressed by designing a hybrid ventilation scheme supported by predictive control and soft-sensors. The main idea of the hybrid ventilation system is to achieve thermal comfort by varying the ON/OFF times of the air conditioners to maintain the temperature within user-defined bands using a predictive control and IAQ is maintained using Healthbox 3.0, a DCV device. Furthermore, this study also designs soft-sensors by combining the Internet of Things (IoT)-based sensors with deep-learning tools. The hardware realization of the control and IoT prototype is also discussed. The proposed novel hybrid ventilation system and the soft-sensors are demonstrated in a real research laboratory, i.e., Center for Research in Automatic Control Engineering (C-RACE) located at Kalasalingam University, India. Our results show the perceived benefits of hybrid ventilation, predictive control, and soft-sensors

Convergence of Intelligent Data Acquisition and Advanced Computing Systems

This book is a collection of published articles from the Sensors Special Issue on "Convergence of Intelligent Data Acquisition and Advanced Computing Systems". It includes extended versions of the conference contributions from the 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS’2019), Metz, France, as well as external contributions