Wearable Technologies and AI at the Far Edge for Chronic Heart Failure Prevention and Management: A Systematic Review and Prospects

Smart wearable devices enable personalized at-home healthcare by unobtrusively collecting patient health data and facilitating the development of intelligent platforms to support patient care and management. The accurate analysis of data obtained from wearable devices is crucial for interpreting and contextualizing health data and facilitating the reliable diagnosis and management of critical and chronic diseases. The combination of edge computing and artificial intelligence has provided real-time, time-critical, and privacy-preserving data analysis solutions. However, based on the envisioned service, evaluating the additive value of edge intelligence to the overall architecture is essential before implementation. This article aims to comprehensively analyze the current state of the art on smart health infrastructures implementing wearable and AI technologies at the far edge to support patients with chronic heart failure (CHF). In particular, we highlight the contribution of edge intelligence in supporting the integration of wearable devices into IoT-aware technology infrastructures that provide services for patient diagnosis and management. We also offer an in-depth analysis of open challenges and provide potential solutions to facilitate the integration of wearable devices with edge AI solutions to provide innovative technological infrastructures and interactive services for patients and doctors

Lightweight Object Detection Ensemble Framework for Autonomous Vehicles in Challenging Weather Conditions

The computer vision systems driving autonomous vehicles are judged by their ability to detect objects and obstacles in the vicinity of the vehicle in diverse environments. Enhancing this ability of a self-driving car to distinguish between the elements of its environment under adverse conditions is an important challenge in computer vision. For example, poor weather conditions like fog and rain lead to image corruption which can cause a drastic drop in object detection (OD) performance. The primary navigation of autonomous vehicles depends on the effectiveness of the image processing techniques applied to the data collected from various visual sensors. Therefore, it is essential to develop the capability to detect objects like vehicles and pedestrians under challenging conditions such as like unpleasant weather. Ensembling multiple baseline deep learning models under different voting strategies for object detection and utilizing data augmentation to boost the models' performance is proposed to solve this problem. The data augmentation technique is particularly useful and works with limited training data for OD applications. Furthermore, using the baseline models significantly speeds up the OD process as compared to the custom models due to transfer learning. Therefore, the ensembling approach can be highly effective in resource-constrained devices deployed for autonomous vehicles in uncertain weather conditions. The applied techniques demonstrated an increase in accuracy over the baseline models and were able to identify objects from the images captured in the adverse foggy and rainy weather conditions. The applied techniques demonstrated an increase in accuracy over the baseline models and reached 32.75% mean average precision (mAP) and 52.56% average precision (AP) in detecting cars in the adverse fog and rain weather conditions present in the dataset. The effectiveness of multiple voting strategies for bounding box predictions on the dataset is also demonstrated. These strategies help increase the explainability of object detection in autonomous systems and improve the performance of the ensemble techniques over the baseline models

Runtime adaptive iomt node on multi-core processor platform

The Internet of Medical Things (IoMT) paradigm is becoming mainstream in multiple clinical trials and healthcare procedures. Thanks to innovative technologies, latest-generation communication networks, and state-of-the-art portable devices, IoTM opens up new scenarios for data collection and continuous patient monitoring. Two very important aspects should be considered to make the most of this paradigm. For the first aspect, moving the processing task from the cloud to the edge leads to several advantages, such as responsiveness, portability, scalability, and reliability of the sensor node. For the second aspect, in order to increase the accuracy of the system, state-of-the-art cognitive algorithms based on artificial intelligence and deep learning must be integrated. Sensory nodes often need to be battery powered and need to remain active for a long time without a different power source. Therefore, one of the challenges to be addressed during the design and development of IoMT devices concerns energy optimization. Our work proposes an implementation of cognitive data analysis based on deep learning techniques on resource-constrained computing platform. To handle power efficiency, we introduced a component called Adaptive runtime Manager (ADAM). This component takes care of reconfiguring the hardware and software of the device dynamically during the execution, in order to better adapt it to the workload and the required operating mode. To test the high computational load on a multi-core system, the Orlando prototype board by STMicroelectronics, cognitive analysis of Electrocardiogram (ECG) traces have been adopted, considering single-channel and six-channel simultaneous cases. Experimental results show that by managing the sensory node configuration at runtime, energy savings of at least 15% can be achieved

Train me if you can: decentralized learning on the deep edge

The end of Moore’s Law aligned with data privacy concerns is forcing machine learning (ML) to shift from the cloud to the deep edge. In the next-generation ML systems, the inference and part of the training process will perform at the edge, while the cloud stays responsible for major updates. This new computing paradigm, called federated learning (FL), alleviates the cloud and network infrastructure while increasing data privacy. Recent advances empowered the inference pass of quantized artificial neural networks (ANNs) on Arm Cortex-M and RISC-V microcontroller units (MCUs). Nevertheless, the training remains confined to the cloud, imposing the transaction of high volumes of private data over a network and leading to unpredictable delays when ML applications attempt to adapt to adversarial environments. To fill this gap, we make the first attempt to evaluate the feasibility of ANN training in Arm Cortex-M MCUs. From the available optimization algorithms, stochastic gradient descent (SGD) has the best trade-off between accuracy, memory footprint, and latency. However, its original form and the variants available in the literature still do not fit the stringent requirements of Arm Cortex-M MCUs. We propose L-SGD, a lightweight implementation of SGD optimized for maximum speed and minimal memory footprint in this class of MCUs. We developed a floating-point version and another that operates over quantized weights. For a fully-connected ANN trained on the MNIST dataset, L-SGD (float-32) is 4.20× faster than the SGD while requiring only 2.80% of the memory with negligible accuracy loss. Results also show that quantized training is still unfeasible to train an ANN from the scratch but is a lightweight solution to perform minor model fixes and counteract the fairness problem in typical FL systems.This work has been supported by FCT - Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020. This work has also been supported by FCT within the PhD Scholarship Project Scope: SFRH/BD/146780/2019

High-Performance VOC Quantification for IAQ Monitoring Using Advanced Sensor Systems and Deep Learning

With air quality being one target in the sustainable development goals set by the United Nations, accurate monitoring also of indoor air quality is more important than ever. Chemiresistive gas sensors are an inexpensive and promising solution for the monitoring of volatile organic com pounds, which are of high concern indoors. To fully exploit the potential of these sensors, advanced operating modes, calibration, and data evaluation methods are required. This contribution outlines a systematic approach based on dynamic operation (temperature-cycled operation), randomized calibration (Latin hypercube sampling), and the use of advances in deep neural networks originally developed for natural language processing and computer vision, applying this approach to volatile organic compound measurements for indoor air quality monitoring. This paper discusses the pros and cons of deep neural networks for volatile organic compound monitoring in a laboratory en vironment by comparing the quantification accuracy of state-of-the-art data evaluation methods with a 10-layer deep convolutional neural network (TCOCNN). The overall performance of both methods was compared for complex gas mixtures with several volatile organic compounds, as well as interfering gases and changing ambient humidity in a comprehensive lab evaluation. Furthermore, both were tested under realistic conditions in the field with additional release tests of volatile organic compounds. The results obtained during field testing were compared with analytical measurements, namely the gold standard gas chromatography mass spectrometry analysis based on Tenax sampling, as well as two mobile systems, a gas chromatograph with photo-ionization detection for volatile organic compound monitoring and a gas chromatograph with a reducing compound photometer for the monitoring of hydrogen. The results showed that the TCOCNN outperforms state-of-the-art data evaluation methods, for example for critical pollutants such as formaldehyde, achieving an uncertainty of around 11 ppb even in complex mixtures, and offers a more robust volatile organic compound quantification in a laboratory environment, as well as in real ambient air for most targets

Ultrasonic measurements and machine learning methods to monitor industrial processes

The process manufacturing sector is increasingly using the collection and analysis of data to improve productivity, sustainability, and product quality. The endpoint of this transformation is processes that automatically adapt to demands in real-time. In-line and on-line sensors underpin this transition by automatically collecting the real-time data required to inform decision-making. Each sensing technique possesses its own advantages and disadvantages making them suitable for specific applications. Therefore, a wide range of sensing solutions must be developed to monitor the diverse and often highly variable operations in process manufacturing. Ultrasonic (US) sensors measure the interaction of mechanical waves with materials. They have benefits of being in-line, real-time, non-destructive, low in cost, small in size, able to monitor opaque materials, and can be applied non-invasively. Machine Learning (ML) is the use of computer algorithms to learn patterns in data to perform a task such as making predictions or decisions. The correlations in the data that the ML models learn during training have not been explicitly programmed by human operators. Therefore, ML is used to automatically learn from and analyse data. There are four main types of ML: supervised, unsupervised, semi-supervised, and reinforcement learning. Supervised and unsupervised ML are both used in this thesis. Supervised ML maps inputs to outputs during training with the aim being to create a model that accurately predicts the outputs of data that was not previously used during training. In contrast, unsupervised learning only uses input data in which patterns are discovered. Supervised ML is being increasingly combined with sensor measurements as it offers several distinct advantages over conventional calibration methods, these include: reduced time for development, potential for more accurate fitting, methods to encourage generalisation across parameter ranges, direct correlations to important process information rather than material properties, and ability for continuous retraining as more data becomes available. The aim of this thesis was to develop ML methods to facilitate the optimal deployment of US sensors for process monitoring applications in industrial environments. To achieve this, the thesis evaluates US sensing techniques and ML methods across three types of process manufacturing operations: material mixing, cleaning of pipe fouling, and alcoholic fermentation of beer. Two US sensing techniques were investigated: a non-invasive, reflection-mode technique, and a transmission-based method using an invasive US probe with reflector plate. The non-invasive, reflection-mode technique is more amenable to industrial implementation than the invasive probe given it can be externally retrofitted to existing vessels. Different feature extraction and feature selection methods, algorithms, and hyperparameter ranges were explored to determine the optimal ML pipeline for process monitoring using US sensors. This facilitates reduced development time of US sensor and ML combinations when deployed in industrial settings by recommending a pipeline that has been trialled over a range of process monitoring applications. Furthermore, methods to leverage previously collected datasets were developed to negate or reduce the burden of collecting labelled data (the outputs required during ML model training and often acquired by using reference measurements) for every new process monitoring application. These included unlabelled and labelled domain adaptation approaches. Both US sensing techniques investigated were found to be similarly accurate for process monitoring. To monitor the development of homogeneity during the blending of honey and water the non-invasive, reflection-mode technique achieved up to 100 % accuracy to classify whether the materials were mixed or non-mixed and an R2 of 0.977 to predict the time remaining (or time since) complete mixing was achieved. To monitor the structural changes during the mixing of flour and water, the aformentioned sensing method achieved an accuracy of 92.5 % and an R2 of 0.968 for the same classification and regression tasks. Similarly, the sensing method achieved an accuracy of up to 98.2 % when classifying whether fouling had been removed from pipe sections and R2 values of up 0.947 were achieved when predicting the time remaining until mixing was complete. The non-invasive, reflection-mode method also achieved R2 values of 0.948, Mean Squared Error (MSE) values of 0.283, and Mean Absolute Error (MAE) values of 0.146 to predict alcohol by volume percentage of alcohol during beer fermentation. In comparison, the transmission-based sensing method achieved R2 values of 0.952, MSE values of 0.265, and MAE values of 0.136 for the same task. Furthermore, the transmission-based method achieved accuracies of up to 99.8 % and 99.9 % to classify whether ethanol production had started and whether ethanol production had finished during an industrial beer fermentation process. The material properties that affect US wave propagation are strongly temperature dependent. However, ML models that omitted the process temperature were comparable in accuracy to those which included it as an input. For example, when monitoring laboratory scale fermentation processes, the highest performing models using the process temperature as a feature achieved R2 values of 0.952, MSE values of 0.265, and MAE values of 0.136 to predict the current alcohol concentration, compared with R2 values of 0.948, MSE values of 0.283, and MAE values of 0.146 when omitting the temperature. Similarly, when transferring models between mixing processes, accuracies of 92.2 % and R2 values of 0.947 were achieved when utilising the process temperature compared with 92.1% and 0.942 when omitting the temperature. When transferring models between cleaning processes, inclusion of the process temperature as a feature degraded model accuracy during classification tasks as omitting the temperature produced the highest accuracies for 6 out of 8 tasks. Mixed results were obtained for regression tasks where including the process temperature increased model accuracy for 3 out of 8 tasks. Overall, these results indicate that US sensing, for some applications, is able to achieve comparable accuracy when the process temperature is not available. The choice of whether to include the temperature as a feature should be made during the model validation stage to determine whether it improves prediction accuracy. The optimal feature extraction, feature selection, and ML algorithm permutation was determined as follows: Features were extracted by Convolutional Neural Networks (CNNs) followed by Principal Component Analysis (PCA) and inputted into deep neural networks with Long Short-Term Memory (LSTM) layers. The CNN was pre-trained on an auxiliary task using previously collected US datasets to learn features of the waveforms. The auxiliary task was to classify the dataset from which each US waveform originated. PCA was applied to reduce the dimensionality of the input data and enable the use of additional features, such as the US time of flight or measures of variation between consecutively acquired waveforms. This CNN and PCA feature extraction method was shown to produce more informative features from the US waveform compared to a traditional, coarse feature extraction approach, achieving higher accuracy on 65 % of tasks evaluated. The coarse feature method used commonly extracted parameters from US waveforms such as the energy, standard deviation, and skewness. LSTM units were used to learn the trajectory of the process features and so enable the use of information from previous timesteps to inform model prediction. Using LSTM units was shown to outperform neural networks with feature gradients used as inputs to incorporate information from previous timesteps for all process monitoring applications. Multi-task learning also showed improvements in learning feature trajectories and model accuracy (improving regression accuracy for 8 out of 18 tasks), however, at the expense of a greater number of hyperparameters to optimise. The choice to use multi-task learning should be evaluated during the validation stage of model development. Unlabelled and labelled domain adaptation were investigated to transfer ML knowledge between similar processes. Unlabelled domain adaptation was used to transfer trained ML models between similar mixing and similar cleaning processes to negate the need to collect labelled data for a new task. Transfer Component Analysis was compared to a Single Feature transfer method. Transferring a single feature was found to be optimal, achieving classification accuracies of up to 96.0% and 98.4% to predict whether the mixing or cleaning processes were complete and R2 of up to 0.947 and 0.999 to predict the time remaining for each process, respectively. The Single Feature method was most accurate as it was most representative of the changing material properties at the sensor measurement area. Training ML models across a greater process parameter range (a greater range of temperatures; 19.3 to 22.1°C compared with 19.8 to 21.2°C) or multiple datasets improved transfer learning to further datasets by enabling the models to adapt to a wider range of feature distributions. Labelled domain adaptation increased model accuracy on an industrial fermentation dataset by transferring ML knowledge from a laboratory fermentation dataset. Federated learning was investigated to maintain dataset privacy when applying transfer learning between datasets. The federated learning methodology performed better than the other methods tested, achieving higher accuracy for 14 out of 16 machine learning tasks compared with the base case model which was trained using data solely from the industrial fermentation. This was attributed to federated learning improving the gradient descent operation during network optimisation. During the federated learning training strategy, the local models were trained for a full epoch on each dataset before network weights were sent to the global model. In contrast, during the non-federated learning strategy, batches from each dataset were interspersed. Therefore, it is recommended that the order that the data is passed to the model during training should be evaluated during the validation stage. Overall, there are two main contributions from this thesis: Development of the ML pipeline for process monitoring using US sensors, and the development of unlabelled and labelled domain adaptation methods for process monitoring using US sensors. The development of an ML pipeline facilitates reduced time for the deployment of US sensor and ML combinations in industrial settings by recommending a method that has been trialled over a range of process monitoring applications. The unlabelled and labelled domain adaptation methods were developed to leverage previously collected datasets. This negates or reduces the burden of collecting labelled data in industrial environments. Furthermore, the pipeline and domain adaptation methodologies are evaluated using a non-invasive, reflection-mode US sensing technique. This technique is industrially relevant as it can be externally retrofitted onto existing process equipment. The novelty contained within this thesis can be summarised as follows: • The use of CNNs and LSTM layers for process monitoring using US sensor data: CNNs were used to extract spatial-invariant features from US sensor data to overcome problems of features shifting in the time domain due to changes in temperature or sound velocity. LSTM units were used for their ability to analyse sequences and understand temporal dependencies, critical for monitoring processes that develop over time. Feature extraction using CNNs was shown to produce more informative features from the US waveform compared to a traditional, coarse feature extraction approach, achieving higher accuracy on 65 % of tasks evaluated. LSTM units were shown to outperform neural networks with feature gradients used as inputs to incorporate information from previous timesteps for all process monitoring applications. • Evaluating the omission of the process temperature as a feature for process monitoring using US sensor data: This indicates whether the US sensor and ML combinations could be used in industrial applications where measurement of the process temperature is not available. Overall, it was found that ML models which omitted the process temperature were comparable in accuracy to those which included it as an input (for example, R2 values of 0.952, MSE values of 0.265, and MAE values of 0.136 when including temperature compared with R2 values of 0.948, MSE values of 0.283, and MAE values of 0.146 were obtained when omitting the temperature to predict the current alcohol concentration during laboratory scale fermentation processes). • The use of labelled and unlabelled domain adaptation for US data for process monitoring: Unlabelled domain adaptation was used to transfer trained ML models between similar mixing and similar cleaning processes to negate the need to collect labelled data for a new task. Labelled domain adaptation increased model accuracy on an industrial fermentation dataset by transferring ML knowledge from a laboratory fermentation dataset. • The use of labelled and unlabelled domain adaptation on features extracted from US waveforms: This allows the domain adaptation methods to be used for diverse US waveforms as, instead of aligning the US sensor data, the US waveform features are used which provide information about the process being monitored as they develop over time. • The use of federated learning and multi-task learning with US data: Federated learning was investigated to maintain dataset privacy when applying transfer learning between datasets. Multi-task learning was investigated to aid LSTM unit learning of the process trajectory. The federated learning methodology performed better than the other methods tested, achieving higher accuracy for 14 out of 16 ML tasks compared with the base case model. Multi-task learning also showed improvements in learning feature trajectories and model accuracy (improving regression accuracy for 8 out of 18 tasks evaluated), however, at the expense of a greater number of hyperparameters to optimise. • The use of data augmentation for US data for process monitoring applications: Data augmentation was a component of the convolutional feature extraction method developed in this thesis. Data augmentation artificially increased the dataset size to train the convolutional feature extractor while ensuring that features specific to each waveform, rather than the position or magnitude of features, were learned. This improved the feature-learning auxiliary task the CNN was trained to perform which classified the dataset from which each previously collected US waveform originated