Towards Vision-Based Smart Hospitals: A System for Tracking and Monitoring Hand Hygiene Compliance

One in twenty-five patients admitted to a hospital will suffer from a hospital acquired infection. If we can intelligently track healthcare staff, patients, and visitors, we can better understand the sources of such infections. We envision a smart hospital capable of increasing operational efficiency and improving patient care with less spending. In this paper, we propose a non-intrusive vision-based system for tracking people's activity in hospitals. We evaluate our method for the problem of measuring hand hygiene compliance. Empirically, our method outperforms existing solutions such as proximity-based techniques and covert in-person observational studies. We present intuitive, qualitative results that analyze human movement patterns and conduct spatial analytics which convey our method's interpretability. This work is a step towards a computer-vision based smart hospital and demonstrates promising results for reducing hospital acquired infections.Comment: Machine Learning for Healthcare Conference (MLHC

Computer vision based classification of fruits and vegetables for self-checkout at supermarkets

The field of machine learning, and, in particular, methods to improve the capability of machines to perform a wider variety of generalised tasks are among the most rapidly growing research areas in today’s world. The current applications of machine learning and artificial intelligence can be divided into many significant fields namely computer vision, data sciences, real time analytics and Natural Language Processing (NLP). All these applications are being used to help computer based systems to operate more usefully in everyday contexts. Computer vision research is currently active in a wide range of areas such as the development of autonomous vehicles, object recognition, Content Based Image Retrieval (CBIR), image segmentation and terrestrial analysis from space (i.e. crop estimation). Despite significant prior research, the area of object recognition still has many topics to be explored. This PhD thesis focuses on using advanced machine learning approaches to enable the automated recognition of fresh produce (i.e. fruits and vegetables) at supermarket self-checkouts. This type of complex classification task is one of the most recently emerging applications of advanced computer vision approaches and is a productive research topic in this field due to the limited means of representing the features and machine learning techniques for classification. Fruits and vegetables offer significant inter and intra class variance in weight, shape, size, colour and texture which makes the classification challenging. The applications of effective fruit and vegetable classification have significant importance in daily life e.g. crop estimation, fruit classification, robotic harvesting, fruit quality assessment, etc. One potential application for this fruit and vegetable classification capability is for supermarket self-checkouts. Increasingly, supermarkets are introducing self-checkouts in stores to make the checkout process easier and faster. However, there are a number of challenges with this as all goods cannot readily be sold with packaging and barcodes, for instance loose fresh items (e.g. fruits and vegetables). Adding barcodes to these types of items individually is impractical and pre-packaging limits the freedom of choice when selecting fruits and vegetables and creates additional waste, hence reducing customer satisfaction. The current situation, which relies on customers correctly identifying produce themselves leaves open the potential for incorrect billing either due to inadvertent error, or due to intentional fraudulent misclassification resulting in financial losses for the store. To address this identified problem, the main goals of this PhD work are: (a) exploring the types of visual and non-visual sensors that could be incorporated into a self-checkout system for classification of fruits and vegetables, (b) determining a suitable feature representation method for fresh produce items available at supermarkets, (c) identifying optimal machine learning techniques for classification within this context and (d) evaluating our work relative to the state-of-the-art object classification results presented in the literature. An in-depth analysis of related computer vision literature and techniques is performed to identify and implement the possible solutions. A progressive process distribution approach is used for this project where the task of computer vision based fruit and vegetables classification is divided into pre-processing and classification techniques. Different classification techniques have been implemented and evaluated as possible solution for this problem. Both visual and non-visual features of fruit and vegetables are exploited to perform the classification. Novel classification techniques have been carefully developed to deal with the complex and highly variant physical features of fruit and vegetables while taking advantages of both visual and non-visual features. The capability of classification techniques is tested in individual and ensemble manner to achieved the higher effectiveness. Significant results have been obtained where it can be concluded that the fruit and vegetables classification is complex task with many challenges involved. It is also observed that a larger dataset can better comprehend the complex variant features of fruit and vegetables. Complex multidimensional features can be extracted from the larger datasets to generalise on higher number of classes. However, development of a larger multiclass dataset is an expensive and time consuming process. The effectiveness of classification techniques can be significantly improved by subtracting the background occlusions and complexities. It is also worth mentioning that ensemble of simple and less complicated classification techniques can achieve effective results even if applied to less number of features for smaller number of classes. The combination of visual and nonvisual features can reduce the struggle of a classification technique to deal with higher number of classes with similar physical features. Classification of fruit and vegetables with similar physical features (i.e. colour and texture) needs careful estimation and hyper-dimensional embedding of visual features. Implementing rigorous classification penalties as loss function can achieve this goal at the cost of time and computational requirements. There is a significant need to develop larger datasets for different fruit and vegetables related computer vision applications. Considering more sophisticated loss function penalties and discriminative hyper-dimensional features embedding techniques can significantly improve the effectiveness of the classification techniques for the fruit and vegetables applications

Computer-aided technologies for food risk assessment

Nowadays, the relationship among food, health, and economy is an emerging topic that engages the modern global society from an interdisciplinary perspective. The Food Risk assessment has been formalized and incorporated into the specific discipline addressed to everyone involved with food from production to consumption, including growers, processors, regulators, distributors, retailers and consumers. However, both intentional and unintentional actions committed for economic gains could make an attempt on people’s health. In recent years, many tools have been developed to help the authorities involved in controls and consumers too. The integration of multidisciplinary techniques has favorably supported the study and the development of tools related to the field of the Systems Biology as well as the application of state-of-the-art techniques deriving from other application fields such as the Computer Science. To counteract and operate with reaction and prevention in my Ph.D. I investigate the use of original Computer-Aided technologies in two particular instances. The first one refers to a Food Traceability issue related to dairy product control. I studied and implemented a heuristic procedure that allows food inspectors to highlight possible adulterations in cheese production into the small farm environment. The procedure is mainly based on Short Tandem Repeat investigation to compare the DNA fingerprint among cows, milk, and cheese. The second one regards the Food Fraud discipline. I developed a mobile application to counteract the problem of fish species substitution and mislabelling. The infrastructure implemented is composed of a cloud remote server where both image analysis and machine learning algorithm take part. The main breakthrough on this topic has been reached with a deep learning classification system which allowed to obtain an improvement in the global accuracy to correctly identify the fish species. Eventually, in the last topic, I deal with the problem of fish fillets identification. The main outcome of this preliminary study is the application of a portable Near Infra-Res molecular sensor that was specifically trained to discriminate the fish fillets available in a sample database

Automatic detection of persuasion attempts on social networks

The rise of social networks and the increasing amount of time people spend on them have created a perfect place for the dissemination of false narratives, propaganda, and manipulated content. In order to prevent the spread of disinformation, content moderation is needed, however it is unfeasible to do it manually due to the large number of daily posts. This dissertation aims at solving this problem by creating a system for automatic detection of persuasion techniques, as proposed in a SemEval challenge. We start by reviewing classic machine learning and natural language processing approaches and go through more sophisticated deep learning approaches which are more suited for this type of complex problem. The classic machine learning approaches are used to create a baseline for the problem. The architecture proposed, using deep learning techniques, is built on top of a DistilBERT transformer followed by Convolutional Neural Networks. We study how our usage of different loss functions, pre-processing the text, freezing DistilBERT layers and performing hyperparameter search impact the performance of our system. We discovered that we could optimize our architecture by freezing the two initial DistilBERT’s layers and using asymmetric loss to tackle the class imbalance on the dataset presented. This study resulted in three final models with the same architecture but using different parameters where the first showed signs of overfitting, one did not show sings of overfitting but did not seem to converge and other seemed to converge but yielded the worst performance of all three. They presented a micro f1-score of 0.551, 0.526 and 0.509 and were placed in 3rd, 6th and 11th place respectively in the overall table. The models can only classify textual elements as the multimodal component is not implemented on this iteration but only discussed; Sumário: Deteção automática de tentativas de persuasão em redes sociais - O crescimento das redes sociais e o aumento do tempo que as pessoas passam nelas criaram um lugar perfeito para a disseminação de falsas narrativas, propaganda e conteúdo manipulado. Para evitar a disseminação da desinformação, é necessária a moderação do conteúdo, porém é inviável fazê-la manualmente devido ao grande número de conteúdo diário. Esta dissertação visa resolver este problema através da criação de um sistema de deteção automática de técnicas de persuasão, conforme proposto num desafio da SemEval. Começamos por rever as abordagens clássicas de aprendizagem automática e processamento de linguagem natural, passamos de seguida por abordagens mais sofisticadas de aprendizagem profunda que são mais adequadas para esse tipo de problema complexo. As abordagens clássicas de aprendizagem automática são usadas para criar um ponto de partida para o problema. A arquitetura proposta, utilizando técnicas de aprendizagem profunda, é construída sobre um transformer DistilBERT seguido de redes neuronais convolucionais. Estudamos de que forma o uso de diferentes funções ativação, pré-processamento do texto, congelamento de camadas do DistilBERT e realização de pesquisa de hiperparâmetros afetam o desempenho do nosso sistema. Descobrimos que poderíamos otimizar nossa arquitetura congelando as duas camadas iniciais do DistilBERT e usando asymmetric loss para lidar com o desequilíbrio de classes no conjunto de dados apresentado. Este estudo resultou em três modelos finais com a mesma arquitetura, mas usando parâmetros diferentes, onde o primeiro mostrou sinais de overfitting, um não mostrou sinais de overfitting mas não parece convergir e outro parece convergir, mas produziu o pior desempenho de todos os três. Apresentaram micro f1-score de 0.551, 0.526 e 0.509 e ficaram em 3º, 6º e 11º lugares, respectivamente, na tabela geral. Os modelos podem apenas classificar elementos textuais, pois o componente multimodal não é implementado nesta iteração, mas apenas discutido

Deep Learning Detection in the Visible and Radio Spectrums

Deep learning models with convolutional neural networks are being used to solve some of the most difficult problems in computing today. Complicating factors to the use and development of deep learning models include lack of availability of large volumes of data, lack of problem specific samples, and the lack variations in the specific samples available. The costs to collect this data and to compute the models for the task of detection remains a inhibitory condition for all but the most well funded organizations. This thesis seeks to approach deep learning from a cost reduction and hybrid perspective — incorporating techniques of transfer learning, training augmentation, synthetic data generation, morphological computations, as well as statistical and thresholding model fusion — in the task of detection in two domains: visible spectrum detection of target spacecraft, and radio spectrum detection of radio frequency interference in 2D astronomical time-frequency data. The effects of training augmentation on object detection performance is studied in the visible spectrum, as well as the effect of image degradation on detection performance. Supplementing training on degraded images significantly improves the detection results, and in scenarios with low factors of degradation, the baseline results are exceeded. Morphological operations on degraded data shows promise in reducing computational requirements in some detection tasks. The proposed Mask R-CNN model is able to detect and localize properly on spacecraft images degraded by high levels of pixel loss. Deep learning models such as U-Net have been leveraged for the task of radio frequency interference labeling (flagging). Model variations on U-Net architecture design such as layer size and composition are continuing to be explored, however, the examination of deep learning models combined with statistical tests and thresholding techniques for radio frequency interference mitigation is in its infancy. For the radio spectrum domain, the use of the U-Net model combined with various statistical tests and the SumThreshold technique in an output fusion model is tested against a baseline of SumThreshold alone, for the detection of radio frequency interference. This thesis also contributes an improved dataset for spacecraft detection, and a simple technique for the generation of synthetic channelized voltage data for simulating radio astronomy spectra recordings in a 2D time-frequency plot