Detection of abnormal passenger behaviors on ships, using RGBD cameras

El objetivo de este trabajo fin de Máster (TFM) es el diseño, implementación, y evaluación de un sistema inteligente de videovigilancia, que permita la detección, seguimiento y conteo de personas, así como la detección de estampidas, para grandes embarcaciones. El sistema desarrollado debe ser portable, y funcionar en tiempo real. Para ello se ha realizado un estudio de las tecnologías disponibles en sistemas embebidos, para elegir las que mejor se adecúan al objetivo del TFM. Se ha desarrollado un sistema de detección de personas basado en una MobileNet-SSD, complementado con un banco de filtros de Kalman para el seguimiento. Además, se ha incorporado un detector de estampidas basado en el análisis de la entropía del flujo óptico. Todo ello se ha implementado y evaluado en un dispositivo embebido que incluye una unidad VPU. Los resultados obtenidos han permitido validar la propuesta.The aim of this Final Master Thesis (TFM) is the design, implementation and evaluation of an intelligent video surveillance system that allows the detection, monitoring and counting of people, as well as the detection of stampedes, for large ships. The developed system must be portable and work in real time. To this end, a study has been carried out of the technologies available in embedded systems, in order to choose those that best suit the objective of the TFM. A people detection system based on a MobileNetSSD has been developed, complemented by a Kalman filter bank for monitoring. In addition, a stampede detector based on optical flow entropy analysis has been incorporated. All this has been implemented and evaluated in an embedded device that includes a Vision Processing Unit (VPU) unit. The results obtained have allowed the validation of the proposal.Máster Universitario en Ingeniería de Telecomunicación (M125

Image Analysis and Machine Learning in Agricultural Research

Agricultural research has been a focus for academia and industry to improve human well-being. Given the challenges in water scarcity, global warming, and increased prices of fertilizer, and fossil fuel, improving the efficiency of agricultural research has become even more critical. Data collection by humans presents several challenges including: 1) the subjectiveness and reproducibility when doing the visual evaluation, 2) safety when dealing with high toxicity chemicals or severe weather events, 3) mistakes cannot be avoided, and 4) low efficiency and speed. Image analysis and machine learning are more versatile and advantageous in evaluating different plant characteristics, and this could help with agricultural data collection. In the first chapter, information related to different types of imaging (e.g., RGB, multi/hyperspectral, and thermal imaging) was explored in detail for its advantages in different agriculture applications. The process of image analysis demonstrated how target features were extracted for analysis including shape, edge, texture, and color. After acquiring features information, machine learning can be used to automatically detect or predict features of interest such as disease severity. In the second chapter, case studies of different agricultural applications were demonstrated including: 1) leaf damage symptoms, 2) stress evaluation, 3) plant growth evaluation, 4) stand/insect counting, and 5) evaluation for produce quality. Case studies showed that the use of image analysis is often more advantageous than visual rating. Advantages of image analysis include increased objectivity, speed, and more reproducibly reliable results. In the third chapter, machine learning was explored using romaine lettuce images from RD4AG to automatically grade for bolting and compactness (two of the important parameters for lettuce quality). Although the accuracy is at 68.4 and 66.6% respectively, a much larger data base and many improvements are needed to increase the model accuracy and reliability. With the advancement in cameras, computers with high computing power, and the development of different algorithms, image analysis and machine learning have the potential to replace part of the labor and improve the current data collection procedure in agricultural research. Advisor: Gary L. Hei

3D Sensor Placement and Embedded Processing for People Detection in an Industrial Environment

Papers I, II and III are extracted from the dissertation and uploaded as separate documents to meet post-publication requirements for self-arciving of IEEE conference papers.At a time when autonomy is being introduced in more and more areas, computer vision plays a very important role. In an industrial environment, the ability to create a real-time virtual version of a volume of interest provides a broad range of possibilities, including safety-related systems such as vision based anti-collision and personnel tracking. In an offshore environment, where such systems are not common, the task is challenging due to rough weather and environmental conditions, but the result of introducing such safety systems could potentially be lifesaving, as personnel work close to heavy, huge, and often poorly instrumented moving machinery and equipment. This thesis presents research on important topics related to enabling computer vision systems in industrial and offshore environments, including a review of the most important technologies and methods. A prototype 3D sensor package is developed, consisting of different sensors and a powerful embedded computer. This, together with a novel, highly scalable point cloud compression and sensor fusion scheme allows to create a real-time 3D map of an industrial area. The question of where to place the sensor packages in an environment where occlusions are present is also investigated. The result is algorithms for automatic sensor placement optimisation, where the goal is to place sensors in such a way that maximises the volume of interest that is covered, with as few occluded zones as possible. The method also includes redundancy constraints where important sub-volumes can be defined to be viewed by more than one sensor. Lastly, a people detection scheme using a merged point cloud from six different sensor packages as input is developed. Using a combination of point cloud clustering, flattening and convolutional neural networks, the system successfully detects multiple people in an outdoor industrial environment, providing real-time 3D positions. The sensor packages and methods are tested and verified at the Industrial Robotics Lab at the University of Agder, and the people detection method is also tested in a relevant outdoor, industrial testing facility. The experiments and results are presented in the papers attached to this thesis.publishedVersio

Egocentric vision-based passive dietary intake monitoring

Egocentric (first-person) perception captures and reveals how people perceive their surroundings. This unique perceptual view enables passive and objective monitoring of human-centric activities and behaviours. In capturing egocentric visual data, wearable cameras are used. Recent advances in wearable technologies have enabled wearable cameras to be lightweight, accurate, and with long battery life, making long-term passive monitoring a promising solution for healthcare and human behaviour understanding. In addition, recent progress in deep learning has provided an opportunity to accelerate the development of passive methods to enable pervasive and accurate monitoring, as well as comprehensive modelling of human-centric behaviours. This thesis investigates and proposes innovative egocentric technologies for passive dietary intake monitoring and human behaviour analysis. Compared to conventional dietary assessment methods in nutritional epidemiology, such as 24-hour dietary recall (24HR) and food frequency questionnaires (FFQs), which heavily rely on subjects’ memory to recall the dietary intake, and trained dietitians to collect, interpret, and analyse the dietary data, passive dietary intake monitoring can ease such burden and provide more accurate and objective assessment of dietary intake. Egocentric vision-based passive monitoring uses wearable cameras to continuously record human-centric activities with a close-up view. This passive way of monitoring does not require active participation from the subject, and records rich spatiotemporal details for fine-grained analysis. Based on egocentric vision and passive dietary intake monitoring, this thesis proposes: 1) a novel network structure called PAR-Net to achieve accurate food recognition by mining discriminative food regions. PAR-Net has been evaluated with food intake images captured by wearable cameras as well as those non-egocentric food images to validate its effectiveness for food recognition; 2) a deep learning-based solution for recognising consumed food items as well as counting the number of bites taken by the subjects from egocentric videos in an end-to-end manner; 3) in light of privacy concerns in egocentric data, this thesis also proposes a privacy-preserved solution for passive dietary intake monitoring, which uses image captioning techniques to summarise the image content and subsequently combines image captioning with 3D container reconstruction to report the actual food volume consumed. Furthermore, a novel framework that integrates food recognition, hand tracking and face recognition has also been developed to tackle the challenge of assessing individual dietary intake in food sharing scenarios with the use of a panoramic camera. Extensive experiments have been conducted. Tested with both laboratory (captured in London) and field study data (captured in Africa), the above proposed solutions have proven the feasibility and accuracy of using the egocentric camera technologies with deep learning methods for individual dietary assessment and human behaviour analysis.Open Acces

Sistema de contagem de fluxo de pessoas usando imagens de profundidade e processamento embarcado

Orientador: Roberto de Alencar LotufoDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: A crescente disponibilidade de diferentes tipos de sensores, a custos progressivamente menores, tem permitido a melhoria de diversas soluções envolvendo a detecção automática de pessoas, além de criar novas demandas. Este trabalho propõe o estudo e a melhoria de técnicas de contagem automática de fluxo de pessoas usando imagens de profundidade, com ênfase em soluções para a implementação dos algoritmos em processadores embarcados, compactos e de baixo custo. O hardware utilizado consiste de um sensor Kinect e de um Raspberry Pi 3 para o processamento. Inicialmente os experimentos envolveram simulações offline, seguidas de testes em cenários reais, usando três instalações diferentesAbstract: The increasing availability of different types of sensors at progressively lower costs is allowing the improvement of many solutions involving automatic people detection, and it is also creating newer demands. In this work we propose to study and improve techniques of automatic people-flow counting using depth images, focusing on solutions to implement the algorithms on compact, low-power, embedded processors. The hardware we used consists of a Kinect sensor and a Raspberry Pi 3 for processing. The experiments initially involved offline simulations, followed by tests in real case scenarios, using three different installationsMestradoEngenharia de ComputaçãoMestre em Engenharia Elétric

Bildebehandling og Autonomi

Abstrakt. Denne bachelor oppgaven omhandler utvikling og implementering av programvare for en undervanns robot (ROV), som tar i bruk blide behandling og data syn for å utføre autonome oppgaver. Dette gjøres som en en del av et tverrfaglig prosjekt i student organisasjonen UiS Subsea. ROV-en er bygd med mål om å delta i MATE ROV World Championship, som holdes i Colorado, USA, den 20-24. juni 2023. Denne bachelor oppgaven beskriver hvordan kamera-strømmer mottas på topside systemet, fra ROV-en, hvordan bilde behandling blir brukt til å løse autonome oppgaver, hvordan programmet implementeres i et brukergrensesnitt (GUI) og hvordan kjøre kommandoer sendes ned til ROV-en. Et modulært system har blitt laget, som mottar flere kamera strømmer, utfører autonome oppgaver og sender styre kommandoer. Programmet er blitt implementert inn i en GUI og testet på land. Tydelige definerte bildebehandlings oppgaver er gitt ut av MATE. Alle oppgavene er løst, utenom 3D modellering. Programmet er testet på land og oppfører seg som ønsket. Derimot, er ikke programmene testet i vann. Grunnen til dette er at ROV-en ikke var ferdigstilt i tide for vår gruppe å teste programmene våre. Testing og forbedring av programmet vill fortsette i forberedelser til MATE. Grunnet leveringsfristen for bachelor oppgaven, blir ikke dette inkludert i denne rapporten. Se vår GitHub repository for koden: https://github.com/UiS-Subsea/Bachelor_Bildebehandling Se GUI gruppens GitHub repository for implementeringen av vår kode inn i deres kode: https://github.com/UiS-Subsea/Bachelor_GUIAbstract. This bachelor’s thesis is about creating and implementing a software program on an underwater robot (ROV), that utilizes image processing and computer vision to perform autonomous tasks. This is done as part of an interdisciplinary project in the student organization UiS Subsea. The ROV is built with the purpose of competing in the MATE ROV World Championship, which is held in Colorado, USA, on the 20-24th of June 2023. This thesis describes how camera feeds are received on the topside system, from the ROV, how image processing is utilized to solve autonomous tasks, the implementation of the program into a graphical user interface (GUI), and the sending of driving commands down to the ROV. Additionally, an attempt was made to use computer vision to create a 3D model of a coral head. A modular program has been made, receiving multiple camera feeds, performing autonomous tasks, and sending driving commands. This program was implemented into a GUI, and tested on land. Clearly defined image-processing tasks were given by MATE. All of them were solved except for 3D modeling. The program was tested on land and behaved as intended. However, the programs were not tested in water. This was due to the ROV not being finished in time for our group to test our programs. Testing and improving the program will continue in preparation for MATE. Due to the submission date for the bachelor’s thesis, this will not be included in this paper. See our GitHub repository for the code: https://github.com/UiS-Subsea/Bachelor_Bildebehandling See the GUI group’s repository for our code implemented into their code: https://github.com/UiS-Subsea/Bachelor_GU

Bildebehandling og Autonomi

Abstrakt. Denne bachelor oppgaven omhandler utvikling og implementering av programvare for en undervanns robot (ROV), som tar i bruk blide behandling og data syn for å utføre autonome oppgaver. Dette gjøres som en en del av et tverrfaglig prosjekt i student organisasjonen UiS Subsea. ROV-en er bygd med mål om å delta i MATE ROV World Championship, som holdes i Colorado, USA, den 20-24. juni 2023. Denne bachelor oppgaven beskriver hvordan kamera-strømmer mottas på topside systemet, fra ROV-en, hvordan bilde behandling blir brukt til å løse autonome oppgaver, hvordan programmet implementeres i et brukergrensesnitt (GUI) og hvordan kjøre kommandoer sendes ned til ROV-en. Et modulært system har blitt laget, som mottar flere kamera strømmer, utfører autonome oppgaver og sender styre kommandoer. Programmet er blitt implementert inn i en GUI og testet på land. Tydelige definerte bildebehandlings oppgaver er gitt ut av MATE. Alle oppgavene er løst, utenom 3D modellering. Programmet er testet på land og oppfører seg som ønsket. Derimot, er ikke programmene testet i vann. Grunnen til dette er at ROV-en ikke var ferdigstilt i tide for vår gruppe å teste programmene våre. Testing og forbedring av programmet vill fortsette i forberedelser til MATE. Grunnet leveringsfristen for bachelor oppgaven, blir ikke dette inkludert i denne rapporten. Se vår GitHub repository for koden: https://github.com/UiS-Subsea/Bachelor_Bildebehandling Se GUI gruppens GitHub repository for implementeringen av vår kode inn i deres kode: https://github.com/UiS-Subsea/Bachelor_GUIAbstract. This bachelor’s thesis is about creating and implementing a software program on an underwater robot (ROV), that utilizes image-processing and computer vision to perform autonomous tasks. This is done as part of an interdisciplinary project in the student organization UiS Subsea. The ROV is built with the purpose of competing in the MATE ROV World Championship, which is held in Colorado, USA, on the 20-24th of June 2023. This thesis describes how camera feeds are received on the topside system, from the ROV, how image processing is utilized to solve autonomous tasks, the implementation of the program into a graphical user interface (GUI), and the sending of driving commands down to the ROV. Additionally, an attempt was made to use computer vision to create a 3D model of a coral head. A modular program has been made, receiving multiple camera feeds, performing autonomous tasks, and sending driving commands. This program was implemented into a GUI, and tested on land. Clearly defined image-processing tasks were given by MATE. All of them were solved except for 3D modeling. The program was tested on land and behaved as intended. However, the programs were not tested in water. This was due to the ROV not being finished in time for our group to test our programs. Testing and improving the program will continue in preparation for MATE. Due to the submission date for the bachelor’s thesis, this will not be included in this paper. See our GitHub repository for the code: https://github.com/UiS-Subsea/Bachelor_Bildebehandling See the GUI group’s repository for our code implemented into their code: https://github.com/UiS-Subsea/Bachelor_GU

Bildebehandling og Autonomi

Abstrakt. Denne bachelor oppgaven omhandler utvikling og implementering av programvare for en undervanns robot (ROV), som tar i bruk blide behandling og data syn for å utføre autonome oppgaver. Dette gjøres som en en del av et tverrfaglig prosjekt i student organisasjonen UiS Subsea. ROV-en er bygd med mål om å delta i MATE ROV World Championship, som holdes i Colorado, USA, den 20-24. juni 2023. Denne bachelor oppgaven beskriver hvordan kamera-strømmer mottas på topside systemet, fra ROV-en, hvordan bilde behandling blir brukt til å løse autonome oppgaver, hvordan programmet implementeres i et brukergrensesnitt (GUI) og hvordan kjøre kommandoer sendes ned til ROV-en. Et modulært system har blitt laget, som mottar flere kamera strømmer, utfører autonome oppgaver og sender styre kommandoer. Programmet er blitt implementert inn i en GUI og testet på land. Tydelige definerte bildebehandlings oppgaver er gitt ut av MATE. Alle oppgavene er løst, utenom 3D modellering. Programmet er testet på land og oppfører seg som ønsket. Derimot, er ikke programmene testet i vann. Grunnen til dette er at ROV-en ikke var ferdigstilt i tide for vår gruppe å teste programmene våre. Testing og forbedring av programmet vill fortsette i forberedelser til MATE. Grunnet leveringsfristen for bachelor oppgaven, blir ikke dette inkludert i denne rapporten. Se vår GitHub repository for koden: https://github.com/UiS-Subsea/Bachelor_Bildebehandling Se GUI gruppens GitHub repository for implementeringen av vår kode inn i deres kode: https://github.com/UiS-Subsea/Bachelor_GUIAbstract. This bachelor’s thesis is about creating and implementing a software program on an underwater robot (ROV), that utilizes image processing and computer vision to perform autonomous tasks. This is done as part of an interdisciplinary project in the student organization UiS Subsea. The ROV is built with the purpose of competing in the MATE ROV World Championship, which is held in Colorado, USA, on the 20-24th of June 2023. This thesis describes how camera feeds are received on the topside system, from the ROV, how image processing is utilized to solve autonomous tasks, the implementation of the program into a graphical user interface (GUI), and the sending of driving commands down to the ROV. Additionally, an attempt was made to use computer vision to create a 3D model of a coral head. A modular program has been made, receiving multiple camera feeds, performing autonomous tasks, and sending driving commands. This program was implemented into a GUI, and tested on land. Clearly defined image-processing tasks were given by MATE. All of them were solved except for 3D modeling. The program was tested on land and behaved as intended. However, the programs were not tested in water. This was due to the ROV not being finished in time for our group to test our programs. Testing and improving the program will continue in preparation for MATE. Due to the submission date for the bachelor’s thesis, this will not be included in this paper. See our GitHub repository for the code: https://github.com/UiS-Subsea/Bachelor_Bildebehandling See the GUI group’s repository for our code implemented into their code: https://github.com/UiS-Subsea/Bachelor_GU

A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles

The spread of Unmanned Aerial Vehicles (UAVs) in the last decade revolutionized many applications fields. Most investigated research topics focus on increasing autonomy during operational campaigns, environmental monitoring, surveillance, maps, and labeling. To achieve such complex goals, a high-level module is exploited to build semantic knowledge leveraging the outputs of the low-level module that takes data acquired from multiple sensors and extracts information concerning what is sensed. All in all, the detection of the objects is undoubtedly the most important low-level task, and the most employed sensors to accomplish it are by far RGB cameras due to costs, dimensions, and the wide literature on RGB-based object detection. This survey presents recent advancements in 2D object detection for the case of UAVs, focusing on the differences, strategies, and trade-offs between the generic problem of object detection, and the adaptation of such solutions for operations of the UAV. Moreover, a new taxonomy that considers different heights intervals and driven by the methodological approaches introduced by the works in the state of the art instead of hardware, physical and/or technological constraints is proposed