Applying Deep Learning to Estimate Fruit Yield in Agriculture 4.0 Systems

Over the last few years, with the advances in Information and Communications Technology (ICT) and the increasing human needs, industry has been reshaping itself. A new industrial revolution is emerging, and it is called Industry 4.0. This revolution intends to digitize the market and make it as intelligent as possible. As the history tells, every time there is an industry revolution, the agricultural sector benefits from it. Agriculture 4.0 is ongoing, and it is marked by intelligence and data. It aims to make the agricultural sector more efficient, that is: producing more outputs (such as food, fibers, fuel and other raw materials) while using less inputs (e.g. water, fertilizers, pesticides). Additionally, it envisions to promote food security, by reducing food loss and waste during the “Farm to Fork” journey. A major challenge in the agricultural sector is forecasting food storage and marketing activities prior to harvesting. Nowadays, most farmers manually count fruits before harvesting, in order to estimate the production yield of their fields, as a means to manage storage and marketing activities. Manually counting fruits in large fields is a laborious, costly and time-consuming effort, which is often also error prone. A consequence of this outdated methodology is that it leads to food wastage, which can affect food security. The developed work along this dissertation is an entry point to a system that is capable of estimating the production yield of a whole orchard, while being capable of respecting the required time constraints of each case study. With data taken with a smartphone, the developed system was able to accurately estimate the number of fruits present in tree sides, registering accuracies up to 98%. The high accuracy and speed results were possible due to the combination of state-of-the-art object detection and tracking techniques. To achieve this, a large model of Scaled YOLOv4 was combined with an online Multiple Object Tracking (MOT) framework based on Simple Online and Realtime Tracking with a Deep Association Metric (Deep SORT). Furthermore, this results validated the viability of implementing a proposed system, capable of estimating the fruit yield of a whole tree and, consequently, the production yield of the whole orchard, that is both low in complexity, easy-to-use, fast and reliable.O avanço das ICTs, juntamente com as necessidades humanas, está a proporcionar uma nova revolução industrial, designada de Indústria 4.0. Esta revolução visa uma digitalização do mercado, assim como torná-lo mais inteligente. Sempre que uma revolução industrial toma lugar, o setor agrícola beneficia disso, herdando as tecnologias que fazem parte de tal revolução. A agricultura 4.0 está em progresso, e é marcada por inteligência e informação. Esta revolução tem como objetivo tornar o setor agrícola mais eficiente, isto é: produzir mais (por exemplo comida, fibras, combustível e outras matérias-primas) com menos (e.g. água, fertilizantes, pesticidas). Adicionalmente, esta revolução visa a promoção da segurança alimentar, através da redução da perda e do desperdício de comida. Um grande problema no setor agrícola reside no planeamento de armazenamento e marketing de alimentos, antes da sua colheita. Na realidade, a maioria dos agricultores realiza o processo de contagem de frutos, do seu campo agrícola, manualmente, a fim de planear o espaço necessário para armazenar os mesmos e planear as suas vendas. A contagem manual de frutos é uma tarefa dispendiosa, que consome uma grande porção de tempo, tediosa, e propícia a erros. Uma consequência desta metodologia de trabalho é o desperdício alimentar, o qual leva ao comprometimento da segurança alimentar. O trabalho desenvolvido ao longo desta dissertação é um ponto de partida para um sistema que é capaz de estimar o rendimento de produção de um pomar inteiro, e ao mesmo tempo capaz de respeitar as restrições temporais de cada caso de estudo. Através de dados adquiridos com um smartphone, o sistema desenvolvido é capaz de estimar o número de frutos presentes em faces de árvores, registando eficácias tão altas como 98%. Os resultados obtidos foram possíveis devido às técnicas implementadas, que contaram com a combinação de metodologias de estado de arte de deteção e rastreamento de objetos. Um modelo da arquitetura Scaled YOLOv4 foi combinado com uma framework baseada em Deep SORT capaz de rastrear múltiplos objetos numa sequência de imagens. Os resultados obtidos validam a viabilidade da implementação de um sistema proposto, que ambiciona ser simples, fácil de usar, rápido e fiável na contagem de frutos de uma árvore inteira e, consequentemente, na estimação do rendimento de produção de um pomar inteiro

Driver attention analysis and drowsiness detection using mobile devices

Drowsiness and lack of attention are some of the most fatal and underrated accident causes while driving. In this thesis a non intrusive classifier based on features from drivers' facial movements has been developed, focusing on detection strategies that could be deployed on low-complexity devices, like smartphones. Different classification architectures will be proposed and studied in order to understand which implementation performed the best in terms of detection accuracy.openEmbargo temporaneo per motivi di segretezza e/o di proprietà dei risultati e informazioni di enti esterni o aziende private che hanno partecipato alla realizzazione del lavoro di ricerca relativo alla tes

Point Cloud Processing for Environmental Analysis in Autonomous Driving using Deep Learning

Autonomous self-driving cars need a very precise perception system of their environment, working for every conceivable scenario. Therefore, different kinds of sensor types, such as lidar scanners, are in use. This thesis contributes highly efficient algorithms for 3D object recognition to the scientific community. It provides a Deep Neural Network with specific layers and a novel loss to safely localize and estimate the orientation of objects from point clouds originating from lidar sensors. First, a single-shot 3D object detector is developed that outputs dense predictions in only one forward pass. Next, this detector is refined by fusing complementary semantic features from cameras and joint probabilistic tracking to stabilize predictions and filter outliers. The last part presents an evaluation of data from automotive-grade lidar scanners. A Generative Adversarial Network is also being developed as an alternative for target-specific artificial data generation.One of the main objectives of leading automotive companies is autonomous self-driving cars. They need a very precise perception system of their environment, working for every conceivable scenario. Therefore, different kinds of sensor types are in use. Besides cameras, lidar scanners became very important. The development in that field is significant for future applications and system integration because lidar offers a more accurate depth representation, independent from environmental illumination. Especially algorithms and machine learning approaches, including Deep Learning and Artificial Intelligence based on raw laser scanner data, are very important due to the long range and three-dimensional resolution of the measured point clouds. Consequently, a broad field of research with many challenges and unsolved tasks has been established. This thesis aims to address this deficit and contribute highly efficient algorithms for 3D object recognition to the scientific community. It provides a Deep Neural Network with specific layers and a novel loss to safely localize and estimate the orientation of objects from point clouds. First, a single shot 3D object detector is developed that outputs dense predictions in only one forward pass. Next, this detector is refined by fusing complementary semantic features from cameras and a joint probabilistic tracking to stabilize predictions and filter outliers. In the last part, a concept for deployment into an existing test vehicle focuses on the semi-automated generation of a suitable dataset. Subsequently, an evaluation of data from automotive-grade lidar scanners is presented. A Generative Adversarial Network is also being developed as an alternative for target-specific artificial data generation. Experiments on the acquired application-specific and benchmark datasets show that the presented methods compete with a variety of state-of-the-art algorithms while being trimmed down to efficiency for use in self-driving cars. Furthermore, they include an extensive set of standard evaluation metrics and results to form a solid baseline for future research.Eines der Hauptziele führender Automobilhersteller sind autonome Fahrzeuge. Sie benötigen ein sehr präzises System für die Wahrnehmung der Umgebung, dass für jedes denkbare Szenario überall auf der Welt funktioniert. Daher sind verschiedene Arten von Sensoren im Einsatz, sodass neben Kameras u. a. auch Lidar Sensoren ein wichtiger Bestandteil sind. Die Entwicklung auf diesem Gebiet ist für künftige Anwendungen von höchster Bedeutung, da Lidare eine genauere, von der Umgebungsbeleuchtung unabhängige, Tiefendarstellung bieten. Insbesondere Algorithmen und maschinelle Lernansätze wie Deep Learning, die Rohdaten über Lernzprozesse direkt verarbeiten können, sind aufgrund der großen Reichweite und der dreidimensionalen Auflösung der gemessenen Punktwolken sehr wichtig. Somit hat sich ein weites Forschungsfeld mit vielen Herausforderungen und ungelösten Problemen etabliert. Diese Arbeit zielt darauf ab, dieses Defizit zu verringern und effiziente Algorithmen zur 3D-Objekterkennung zu entwickeln. Sie stellt ein tiefes Neuronales Netzwerk mit spezifischen Schichten und einer neuartigen Fehlerfunktion zur sicheren Lokalisierung und Schätzung der Orientierung von Objekten aus Punktwolken bereit. Zunächst wird ein 3D-Detektor entwickelt, der in nur einem Vorwärtsdurchlauf aus einer Punktwolke alle Objekte detektiert. Anschließend wird dieser Detektor durch die Fusion von komplementären semantischen Merkmalen aus Kamerabildern und einem gemeinsamen probabilistischen Tracking verfeinert, um die Detektionen zu stabilisieren und Ausreißer zu filtern. Im letzten Teil wird ein Konzept für den Einsatz in einem bestehenden Testfahrzeug vorgestellt, das sich auf die halbautomatische Generierung eines geeigneten Datensatzes konzentriert. Hierbei wird eine Auswertung auf Daten von Automotive-Lidaren vorgestellt. Als Alternative zur zielgerichteten künstlichen Datengenerierung wird ein weiteres generatives Neuronales Netzwerk untersucht. Experimente mit den erzeugten anwendungsspezifischen- und Benchmark-Datensätzen zeigen, dass sich die vorgestellten Methoden mit dem Stand der Technik messen können und gleichzeitig auf Effizienz für den Einsatz in selbstfahrenden Autos optimiert sind. Darüber hinaus enthalten sie einen umfangreichen Satz an Evaluierungsmetriken und -ergebnissen, die eine solide Grundlage für die zukünftige Forschung bilden

Multi-Object Tracking System based on LiDAR and RADAR for Intelligent Vehicles applications

El presente Trabajo Fin de Grado tiene como objetivo el desarrollo de un Sistema de Detección y Multi-Object Tracking 3D basado en la fusión sensorial de LiDAR y RADAR para aplicaciones de conducción autónoma basándose en algoritmos tradicionales de Machine Learning. La implementación realizada está basada en Python, ROS y cumple requerimientos de tiempo real. En la etapa de detección de objetos se utiliza el algoritmo de segmentación del plano RANSAC, para una posterior extracción de Bounding Boxes mediante DBSCAN. Una Late Sensor Fusion mediante Intersection over Union 3D y un sistema de tracking BEV-SORT completan la arquitectura propuesta.This Final Degree Project aims to develop a 3D Multi-Object Tracking and Detection System based on the Sensor Fusion of LiDAR and RADAR for autonomous driving applications based on traditional Machine Learning algorithms. The implementation is based on Python, ROS and complies with real-time requirements. In the Object Detection stage, the RANSAC plane segmentation algorithm is used, for a subsequent extraction of Bounding Boxes using DBSCAN. A Late Sensor Fusion using Intersection over Union 3D and a BEV-SORT tracking system complete the proposed architecture.Grado en Ingeniería en Electrónica y Automática Industria

Recognition, Analysis, and Assessments of Human Skills using Wearable Sensors

One of the biggest social issues in mature societies such as Europe and Japan is the aging population and declining birth rate. These societies have a serious problem with the retirement of the expert workers, doctors, and engineers etc. Especially in the sectors that require long time to make experts in fields like medicine and industry; the retirement and injuries of the experts, is a serious problem. The technology to support the training and assessment of skilled workers (like doctors, manufacturing workers) is strongly required for the society. Although there are some solutions for this problem, most of them are video-based which violates the privacy of the subjects. Furthermore, they are not easy to deploy due to the need for large training data. This thesis provides a novel framework to recognize, analyze, and assess human skills with minimum customization cost. The presented framework tackles this problem in two different domains, industrial setup and medical operations of catheter-based cardiovascular interventions (CBCVI). In particular, the contributions of this thesis are four-fold. First, it proposes an easy-to-deploy framework for human activity recognition based on zero-shot learning approach, which is based on learning basic actions and objects. The model recognizes unseen activities by combinations of basic actions learned in a preliminary way and involved objects. Therefore, it is completely configurable by the user and can be used to detect completely new activities. Second, a novel gaze-estimation model for attention driven object detection task is presented. The key features of the model are: (i) usage of the deformable convolutional layers to better incorporate spatial dependencies of different shapes of objects and backgrounds, (ii) formulation of the gaze-estimation problem in two different way, as a classification as well as a regression problem. We combine both formulations using a joint loss that incorporates both the cross-entropy as well as the mean-squared error in order to train our model. This enhanced the accuracy of the model from 6.8 by using only the cross-entropy loss to 6.4 for the joint loss. The third contribution of this thesis targets the area of quantification of quality of i actions using wearable sensor. To address the variety of scenarios, we have targeted two possibilities: a) both expert and novice data is available , b) only expert data is available, a quite common case in safety critical scenarios. Both of the developed methods from these scenarios are deep learning based. In the first one, we use autoencoders with OneClass SVM, and in the second one we use the Siamese Networks. These methods allow us to encode the expert’s expertise and to learn the differences between novice and expert workers. This enables quantification of the performance of the novice in comparison to the expert worker. The fourth contribution, explicitly targets medical practitioners and provides a methodology for novel gaze-based temporal spatial analysis of CBCVI data. The developed methodology allows continuous registration and analysis of gaze data for analysis of the visual X-ray image processing (XRIP) strategies of expert operators in live-cases scenarios and may assist in transferring experts’ reading skills to novices

Інтелектуальна система виробництва друкованих плат

Робота публікується згідно наказу ректора від 27.05.2021 р. №311/од "Про розміщення кваліфікаційних робіт вищої освіти в репозиторії університету". Керівник дипломної роботи: д.т.н., проф., завідувач кафедри авіаційних комп’ютерно-інтегрованих комплексів, Синєглазов Віктор МихайловичThe main development of the project is the creation of an original, combined, automated system for the production of printed circuit boards in order to eliminate rejects and increase efficiency in time. Software development includes the operation of the YOLO neural network for improved detection and finding liquid and non-liquid areas with component clearance on the camera module under the control of the raspberry сontroller. Decision making takes place in a couple of steps, after which the machine understands what action should be taken, auto centering, reset, undo, etc. The practical significance of the work is the combination of several microcontrollers with the founder in the shop and improvement by means of new algorithms. Basic research has shown how switching from manual to automatic operation using remote communication can significantly reduce the time spent on core department processes. Using emulation with optional microcontroller connections, the problem of limited installer resources and the implementation of more complex algorithms in the installer's operation was solved. For greater project accuracy, the Mirae Mx-200 installer was considered, with statistics displayed for real projects. The physical and software connections allow the machine to be controlled remotely.Головною розробкою проекту є створення оригінальної, комбінованої, автоматизованої системи для виробництва друкованих плат з метою ліквідації браку та збільшення ефективності в часі. Розробка програмного забезпечення включає в себе роботу нейронної мережі YOLO для поліпшення задачі ідентифікування та знаходження ліквідних і неліквідних зон з компонентного просвічування на модулі камери під управлінням контролера расбері. Прийняття рішень після чого машина розуміє яку дію варто зробити, авто центрування, скидання, скасування і т.д. Практичним значенням роботи є комбінація декількох мікроконтролерів з установником в цеху та їх удосконалення шляхом нових алгоритмів. Основні дослідження показали як перехід з ручної роботи на автоматичну за допомогою віддаленого зв’язку можуть значно зменшити часові витрати на основні процеси департаменту. Використовуючи емуляцію з додатковим підключенням мікроконтролерів було вирішено проблему обмеженості ресурсів установника та впровадження більш складніших алгоритмів в його дію. Для більшої точності проекту, було розглянуто установник Mirae Mx-200, з відображенням статистики реальних проектів. Фізичне та програмне підключення дає змогу керувати машиною дистанційно

UAV or Drones for Remote Sensing Applications in GPS/GNSS Enabled and GPS/GNSS Denied Environments

The design of novel UAV systems and the use of UAV platforms integrated with robotic sensing and imaging techniques, as well as the development of processing workflows and the capacity of ultra-high temporal and spatial resolution data, have enabled a rapid uptake of UAVs and drones across several industries and application domains.This book provides a forum for high-quality peer-reviewed papers that broaden awareness and understanding of single- and multiple-UAV developments for remote sensing applications, and associated developments in sensor technology, data processing and communications, and UAV system design and sensing capabilities in GPS-enabled and, more broadly, Global Navigation Satellite System (GNSS)-enabled and GPS/GNSS-denied environments.Contributions include:UAV-based photogrammetry, laser scanning, multispectral imaging, hyperspectral imaging, and thermal imaging;UAV sensor applications; spatial ecology; pest detection; reef; forestry; volcanology; precision agriculture wildlife species tracking; search and rescue; target tracking; atmosphere monitoring; chemical, biological, and natural disaster phenomena; fire prevention, flood prevention; volcanic monitoring; pollution monitoring; microclimates; and land use;Wildlife and target detection and recognition from UAV imagery using deep learning and machine learning techniques;UAV-based change detection

The Conference on High Temperature Electronics

The status of and directions for high temperature electronics research and development were evaluated. Major objectives were to (1) identify common user needs; (2) put into perspective the directions for future work; and (3) address the problem of bringing to practical fruition the results of these efforts. More than half of the presentations dealt with materials and devices, rather than circuits and systems. Conference session titles and an example of a paper presented in each session are (1) User requirements: High temperature electronics applications in space explorations; (2) Devices: Passive components for high temperature operation; (3) Circuits and systems: Process characteristics and design methods for a 300 degree QUAD or AMP; and (4) Packaging: Presently available energy supply for high temperature environment