Multispectral Video Fusion for Non-contact Monitoring of Respiratory Rate and Apnea

Continuous monitoring of respiratory activity is desirable in many clinical applications to detect respiratory events. Non-contact monitoring of respiration can be achieved with near- and far-infrared spectrum cameras. However, current technologies are not sufficiently robust to be used in clinical applications. For example, they fail to estimate an accurate respiratory rate (RR) during apnea. We present a novel algorithm based on multispectral data fusion that aims at estimating RR also during apnea. The algorithm independently addresses the RR estimation and apnea detection tasks. Respiratory information is extracted from multiple sources and fed into an RR estimator and an apnea detector whose results are fused into a final respiratory activity estimation. We evaluated the system retrospectively using data from 30 healthy adults who performed diverse controlled breathing tasks while lying supine in a dark room and reproduced central and obstructive apneic events. Combining multiple respiratory information from multispectral cameras improved the root mean square error (RMSE) accuracy of the RR estimation from up to 4.64 monospectral data down to 1.60 breaths/min. The median F1 scores for classifying obstructive (0.75 to 0.86) and central apnea (0.75 to 0.93) also improved. Furthermore, the independent consideration of apnea detection led to a more robust system (RMSE of 4.44 vs. 7.96 breaths/min). Our findings may represent a step towards the use of cameras for vital sign monitoring in medical applications

Remote Assessment of the Cardiovascular Function Using Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology

Employment of artificial intelligence mechanisms for e-Health systems in order to obtain vital signs and detect diseases from medical images improving the processes of online consultations and diagnosis

Nowadays e-Health web applications allow doctors to access different types of features, such as knowing which medication the patient has consumed or performing online consultations. Internet systems for healthcare can be improved by using artificial intelligence mechanisms for the process of detecting diseases and obtaining biological data, allowing medical professionals to have important information that facilitates the diagnosis process and the choice of the correct treatment for each particular person. The proposed research work aims to present an innovative approach when compared to traditional platforms, by providing online vital signs in real time, access to a web stethoscope, to a medical image uploader that predicts if a certain disease is present, through deep learning methods, and also allows the visualization of all historical data of a patient. This dissertation has the objective of defending the concept of online consultations, providing complementary functionalities to the traditional methods for performing medical diagnoses through the use of software engineering practices. The process of obtaining vital signs was done via artificial intelligence using a computer camera as sensor. This methodology requires that the user is at a state of rest during the measurements. This investigation led to the conclusion that, in the future, many medical processes will most likely be done online, where this practice is considered extremely helpful for the analysis and treatment of contagious diseases, or cases that require constant monitoring.No quotidiano, as aplicações Web e-Saúde permitem aos médicos acesso a diferentes tipos de funcionalidades, como saber qual a medicação que o doente consumiu ou a realização de consultas online. Os sistemas via internet para a saúde podem ser melhorados, utilizando mecanismos de inteligência artificial para os processos de deteção de doenças e de obtenção de dados biológicos, permitindo que os médicos tenham informações importantes que facilitam o processo de diagnóstico ou a escolha do tratamento correto para um determinado utente. O trabalho de investigação proposto pretende apresentar uma abordagem inovadora na comparação com as plataformas tradicionais, ao disponibilizar sinais vitais online em tempo real, acesso a um estetoscópio web, a um uploader de imagens médicas que prevê se uma determinada doença está presente, através de métodos de aprendizagem profunda, bem como permite visualizar todos os dados históricos de um paciente. Esta dissertação visa defender o conceito de consultas virtuais, providenciando funcionalidades complementares aos processos tradicionais de realização de um diagnóstico médico, através da utilização de práticas de engenharia de software. O processo de obtenção de sinais vitais foi feito através de inteligência artificial para visão computacional utilizando uma câmara de computador. Esta metodologia requer que o utilizador esteja em estado de repouso durante a obtenção dos dados medidos. Esta investigação permitiu concluir que, no futuro, muitos processos médicos atuais provavelmente serão feitos online, sendo esta prática considerada extremamente útil na análise e tratamento de doenças contagiosas, ou de casos que requerem acompanhamento constante

Pulse Oxigraphy: And other new in-depth perspectives through the near infrared window

The aim of this thesis was to investigate the feasability of contactless imaging pulse oximetry (proposed term: pulse oxigraphy). The patent disclosed in chapter 2 claims that such pulse oxigraphy can be achieved with camera-derived photoplethysmographic pulse waves at three wavelengths, preferably being 660, 810 and 940nm. From the absorption curves of hemoglobin and oxyhemoglobin it can be easily derived that two of these wavelengths (660 and 940nm) contain oxygenation-related information, and they have proven to be useful for conventional pulse oximetry (in transmission- mode as well as in reflectance-mode). The additional third wavelength (810nm) lies at a so-called isobestic point where the absorption curves of hemoglobin and oxyhemoglobin intersect. Thus, images and/or plethysmographic pulse waves recorded at 810nm do not contain oxygenation-related information, which is useful for reference purposes when dealing with shadows, reflections, movement artifacts and variations in geometry. With regard to pulse oxigraphy the following results were obtained: In chapter 3 we proved that it is possible to derive photoplethysmographic pulse waves containing the heart rythm of a living person at all three required wavelengths from camera recordings collected at a distance of 72 cm. To investigate and validate the capabilities for pulse oxigraphy with this set up, direct measurements on volunteers were sub optimal, because of: Signal-to-noise issues, sequentially recorded heartbeats for oxygen saturation calculations, and lack of a method to induce prolonged stable and adjustable oxygen saturation levels

The Role of Edge Robotics As-a-Service in Monitoring COVID-19 Infection

Deep learning technology has been widely used in edge computing. However, pandemics like covid-19 require deep learning capabilities at mobile devices (detect respiratory rate using mobile robotics or conduct CT scan using a mobile scanner), which are severely constrained by the limited storage and computation resources at the device level. To solve this problem, we propose a three-tier architecture, including robot layers, edge layers, and cloud layers. We adopt this architecture to design a non-contact respiratory monitoring system to break down respiratory rate calculation tasks. Experimental results of respiratory rate monitoring show that the proposed approach in this paper significantly outperforms other approaches. It is supported by computation time costs with 2.26 ms per frame, 27.48 ms per frame, 0.78 seconds for convolution operation, similarity calculation, processing one-minute length respiratory signals, respectively. And the computation time costs of our three-tier architecture are less than that of edge+cloud architecture and cloud architecture. Moreover, we use our three-tire architecture for CT image diagnosis task decomposition. The evaluation of a CT image dataset of COVID-19 proves that our three-tire architecture is useful for resolving tasks on deep learning networks by edge equipment. There are broad application scenarios in smart hospitals in the future

Frame registration for motion compensation in imaging photoplethysmography

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. Imaging photoplethysmography (iPPG) is an emerging technology used to assess microcirculation and cardiovascular signs by collecting backscattered light from illuminated tissue using optical imaging sensors. An engineering approach is used to evaluate whether a silicone cast of a human palm might be effectively utilized to predict the results of image registration schemes for motion compensation prior to their application on live human tissue. This allows us to establish a performance baseline for each of the algorithms and to isolate performance and noise fluctuations due to the induced motion from the temporally changing physiological signs. A multi-stage evaluation model is developed to qualitatively assess the influence of the region of interest (ROI), system resolution and distance, reference frame selection, and signal normalization on extracted iPPG waveforms from live tissue. We conclude that the application of image registration is able to deliver up to 75% signal-to-noise (SNR) improvement (4.75 to 8.34) over an uncompensated iPPG signal by employing an intensity-based algorithm with a moving reference frame

Employment of artificial intelligence mechanisms for e-Health systems in order to obtain vital signs improving the processes of online consultations and diagnosis

A large number of web-based e-Health applications have been developed through time, allowing doctors to access different types of functionalities, like knowing which medication the patient has consumed or performing online consultations. Internet systems for healthcare can be improved by using artificial intelligence mechanisms for the process of detecting diseases and obtaining biological data, allowing medical professionals to have important information that facilitates the diagnosis process and the choice of the correct treatment for each particular person. The proposed research work aims to present an innovative approach when compared to traditional platforms, by providing online vital signs in real time and allowing the visualization of all historical data of a patient. It aims to defend the concept of promoting online consultations, providing complementary functionalities to the traditional methods for performing medical diagnoses through the use of software engineering practices. This investigation led to the conclusion that, in the future, many medical processes will most likely be done online, where this practice is considered extremely helpful for the analysis and treatment of contagious diseases, or cases that require constant monitoring.info:eu-repo/semantics/acceptedVersio

Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging

Background: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera-based PhotoPlethysmoGraphy (PPG) and a low-cost thermal camera can be used to create cheap, convenient and mobile monitoring systems. However, to ensure reliable monitoring results, a person has to remain still for several minutes while a measurement is being taken. This is very cumbersome and makes its use in real-life mobile situations quite impractical. // Objective: We propose a system which combines PPG and thermography with the aim of improving cardiovascular signal quality and capturing stress responses quickly. // Methods: Using a smartphone camera with a low cost thermal camera added on, we built a novel system which continuously and reliably measures two different types of cardiovascular events: i) blood volume pulse and ii) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 healthy participants, involved in a series of stress-inducing mental workload tasks, measured their physiological responses to stressors over a short window of time (20 seconds) immediately after each task. Participants reported their level of perceived mental stress using a 10-cm Visual Analogue Scale (VAS). We used normalized K-means clustering to reduce interpersonal differences in the self-reported ratings. For the instant stress inference task, we built novel low-level feature sets representing variability of cardiovascular patterns. We then used the automatic feature learning capability of artificial Neural Networks (NN) to improve the mapping between the extracted set of features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. // Results: First, we found that the measured PPG signals presented high quality cardiac cyclic information (relative power Signal Quality Index, pSQI: M=0.755, SD=0.068). We also found that the measured thermal changes of the nose tip presented high quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (respiratory pSQI: from M=0.714 to M=0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the two cardiovascular signals (i.e. heart rate variability and thermal directionality metrics) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing pre-crafted features based-methods. In addition, the 17-fold Leave-One-Subject-Out (LOSO) cross-validation results showed that combination of both modalities produced higher accuracy in comparison with the use of PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art automatic stress recognition methods that require long term measurements (usually, at least a period of 2 minutes is required for an accuracy of around 80% from LOSO). Lastly, we explored effects of different widely-used data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. // Conclusions: Results demonstrate the feasibility of using smartphone-based imaging for instant mental stress recognition. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental healthcare solutions in the wild