DistancePPG: Robust non-contact vital signs monitoring using a camera

Vital signs such as pulse rate and breathing rate are currently measured using contact probes. But, non-contact methods for measuring vital signs are desirable both in hospital settings (e.g. in NICU) and for ubiquitous in-situ health tracking (e.g. on mobile phone and computers with webcams). Recently, camera-based non-contact vital sign monitoring have been shown to be feasible. However, camera-based vital sign monitoring is challenging for people with darker skin tone, under low lighting conditions, and/or during movement of an individual in front of the camera. In this paper, we propose distancePPG, a new camera-based vital sign estimation algorithm which addresses these challenges. DistancePPG proposes a new method of combining skin-color change signals from different tracked regions of the face using a weighted average, where the weights depend on the blood perfusion and incident light intensity in the region, to improve the signal-to-noise ratio (SNR) of camera-based estimate. One of our key contributions is a new automatic method for determining the weights based only on the video recording of the subject. The gains in SNR of camera-based PPG estimated using distancePPG translate into reduction of the error in vital sign estimation, and thus expand the scope of camera-based vital sign monitoring to potentially challenging scenarios. Further, a dataset will be released, comprising of synchronized video recordings of face and pulse oximeter based ground truth recordings from the earlobe for people with different skin tones, under different lighting conditions and for various motion scenarios.Comment: 24 pages, 11 figure

A photoplethysmography smartphone-based method for heart rate variability assessment: device model and breathing influences

© 2019 Elsevier. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/A measurement method of heart rate and heart rate variability (HRV) based on smartphone has been developed and validated. The method is based on photoplethysmography (PPG) acquired with the smartphone camera (SPPG). SPPG was compared with the electrocardiogram (ECG), used as the gold standard, and with an external PPG sensor. Twenty-three healthy subjects were measured using two different smartphone models in three different breathing conditions. The error of the first differentiation between SPPG and ECG series is minimized with the fiducial point at maximum first derivative of the SPPG. The obtained standard deviation of error (SDE) between SPPG and ECG is around 5.4 ms and it is similar to SDE between PPG and ECG. Good agreement between SPPG and ECG for NN, SDNN and RMSSD have been found but it is insufficient agreement for LF/HF. Similar levels of agreement for SPPG-ECG and PPG-ECG have been obtained for the HRV indices. Finally, the differences between smartphone models for HRV indices are slight. Therefore, the smartphone can be used for measuring accurately the following HRV indices: NN, SDNN and RMSSD.Peer ReviewedPostprint (published version

Respiratory rate derived from smartphone-camera-acquired pulse photoplethysmographic signals

A method for deriving respiratory rate from smartphone-camera-acquired pulse photoplethysmographic (SCPPG) signal is presented. Our method exploits respiratory information by examining the pulse wave velocity and dispersion from the SCPPG waveform and we term these indices as the pulse width variability (PWV). A method to combine information from several derived respiration signals is also presented and it is used to combine PWV information with other methods such as pulse amplitude variability (PAV), pulse rate variability (PRV), and respiration-induced amplitude and frequency modulations (AM and FM) in SCPPG signals Evaluation is performed on a database containing SCPPG signals recorded from 30 subjects during controlled respiration experiments at rates from 0.2 to 0.6 Hz with an increment of 0.1 Hz, using three different devices: iPhone 4S, iPod 5, and HTC One M8. Results suggest that spontaneous respiratory rates (0.2–0.4 Hz) can be estimated from SCPPG signals by the PWV- and PRVbased methods with low relative error (median of order 0.5% and interquartile range of order 2.5%). The accuracy can be improved by combining PWV and PRV with other methods such as PAV, AM and/or FM methods. Combination of these methods yielded low relative error for normal respiratory rates, and Institute of Physics and Engineering in Medicine maintained good performance at higher rates (0.5–0.6 Hz) when using the iPhone 4S or iPod 5 devices

CardioCam: Leveraging Camera on Mobile Devices to Verify Users While Their Heart is Pumping

With the increasing prevalence of mobile and IoT devices (e.g., smartphones, tablets, smart-home appliances), massive private and sensitive information are stored on these devices. To prevent unauthorized access on these devices, existing user verification solutions either rely on the complexity of user-defined secrets (e.g., password) or resort to specialized biometric sensors (e.g., fingerprint reader), but the users may still suffer from various attacks, such as password theft, shoulder surfing, smudge, and forged biometrics attacks. In this paper, we propose, CardioCam, a low-cost, general, hard-to-forge user verification system leveraging the unique cardiac biometrics extracted from the readily available built-in cameras in mobile and IoT devices. We demonstrate that the unique cardiac features can be extracted from the cardiac motion patterns in fingertips, by pressing on the built-in camera. To mitigate the impacts of various ambient lighting conditions and human movements under practical scenarios, CardioCam develops a gradient-based technique to optimize the camera configuration, and dynamically selects the most sensitive pixels in a camera frame to extract reliable cardiac motion patterns. Furthermore, the morphological characteristic analysis is deployed to derive user-specific cardiac features, and a feature transformation scheme grounded on Principle Component Analysis (PCA) is developed to enhance the robustness of cardiac biometrics for effective user verification. With the prototyped system, extensive experiments involving 25 subjects are conducted to demonstrate that CardioCam can achieve effective and reliable user verification with over 99% average true positive rate (TPR) while maintaining the false positive rate (FPR) as low as 4%

BEst (Biomarker Estimation): Health Biomarker Estimation Non-invasively and Ubiquitously

This dissertation focuses on the non-invasive assessment of blood-hemoglobin levels. The primary goal of this research is to investigate a reliable, affordable, and user-friendly point-of-care solution for hemoglobin-level determination using fingertip videos captured by a smartphone. I evaluated videos obtained from five patient groups, three from the United States and two from Bangladesh, under two sets of lighting conditions. In the last group, based on human tissue optical transmission modeling data, I used near-infrared light-emitting diode sources of three wavelengths. I developed novel image processing techniques for fingertip video analysis to estimate hemoglobin levels. I studied video images creating image histogram and subdividing each image into multiple blocks. I determined the region of interest in a video and created photoplethysmogram signals. I created features from image histograms and PPG signals. I used the Partial Least Squares Regression and Support Vector Machine Regression tools to analyze input features and to build hemoglobin prediction models. Using data from the last and largest group of patients studied, I was able to develop a model with a strong linear correlation between estimated and clinically-measured hemoglobin levels. With further data and methodological refinements, the approach I have developed may be able to define a clinically accurate public health applicable tool for hemoglobin level and other blood constituent assessment

Efficient Deep Learning-based Estimation of the Vital Signs on Smartphones

Nowadays, due to the widespread use of smartphones in everyday life and the improvement of computational capabilities of these devices, many complex tasks can now be deployed on them. Concerning the need for continuous monitoring of vital signs, especially for the elderly or those with certain types of diseases, the development of algorithms that can estimate vital signs using smartphones has attracted researchers worldwide. Such algorithms estimate vital signs (heart rate and oxygen saturation level) by processing an input PPG signal. These methods often apply multiple pre-processing steps to the input signal before the prediction step. This can increase the computational complexity of these methods, meaning only a limited number of mobile devices can run them. Furthermore, multiple pre-processing steps also require the design of a couple of hand-crafted stages to obtain an optimal result. This research proposes a novel end-to-end solution to mobile-based vital sign estimation by deep learning. The proposed method does not require any pre-processing. Due to the use of fully convolutional architecture, the parameter count of our proposed model is, on average, a quarter of the ordinary architectures that use fully-connected layers as the prediction heads. As a result, the proposed model has less over-fitting chance and computational complexity. A public dataset for vital sign estimation, including 62 videos collected from 35 men and 27 women, is also provided. The experimental results demonstrate state-of-the-art estimation accuracy.Comment: 6 pages, 9 figures, 4 table

Mobile Personal Healthcare System for Non-Invasive, Pervasive and Continuous Blood Pressure Monitoring: A Feasibility Study

Background: Smartphone-based blood pressure (BP) monitor using photoplethysmogram (PPG) technology has emerged as a promising approach to empower users with self-monitoring for effective diagnosis and control ofhypertension (HT). Objective: This study aimed to develop a mobile personal healthcare system for non-invasive, pervasive, and continuous estimation of BP level and variability to be user-friendly to elderly. Methods: The proposed approach was integrated by a self-designed cuffless, calibration-free, wireless and wearable PPG-only sensor, and a native purposely-designed smartphone application using multilayer perceptron machine learning techniques from raw signals. We performed a pilot study with three elder adults (mean age 61.3 ± 1.5 years; 66% women) to test usability and accuracy of the smartphone-based BP monitor. Results: The employed artificial neural network (ANN) model performed with high accuracy in terms of predicting the reference BP values of our validation sample (n=150). On average, our approach predicted BP measures with accuracy \u3e90% and correlations \u3e0.90 (P \u3c .0001). Bland-Altman plots showed that most of the errors for BP prediction were less than 10 mmHg. Conclusions: With further development and validation, the proposed system could provide a cost-effective strategy to improve the quality and coverage of healthcare, particularly in rural zones, areas lacking physicians, and solitary elderly populations

Non-invasive techniques for respiratory information extraction based on pulse photoplethysmogram and electrocardiogram

El objetivo principal de esta tesis es el desarrollo de métodos no invasivos para la extracción de información respiratoria a partir de dos señales biomédicas ampliamente utilizadas en la rutina clínica: el electrocardiograma (ECG) y la señal fotopletismográfica de pulso (PPG). La motivación de este estudio es la conveniencia de monitorizar información respiratoria a partir de dispositivos no invasivos que permita sustituir las técnicas actuales que podrían interferir con la respiración natural y que presentan inconvenientes en algunas aplicaciones como la prueba de esfuerzo y los estudios del sueño. Además, si estos dispositivos no invasivos son los ya utilizados en la rutina clínica, la información respiratoria extraída de ellos representa un valor añadido que permite tener una visión más completa del paciente. DESARROLLO TEÓRICO Esta tesis se divide en 6 capítulos. El Capítulo 1 introduce la problemática, motivaciones y objetivos del estudio. También introduce el origen fisiológico de las señales estudiadas ECG y PPG, y cómo y por qué tienen información autonómica y respiratoria que se puede extraer de ellas. El Capítulo 2 aborda la obtención de información respiratoria a partir del ECG. Se han propuesto varios métodos para la obtención de la respiración a partir del ECG (EDR, del inglés ¿ECG derived respiration?). Su rendimiento se suele ver muy afectado en entornos altamente no estacionarios y ruidosos como la prueba de esfuerzo. No obstante, se han propuesto algunas alternativas, como una basada en el ángulo de rotación del eje eléctrico (obtenido del ECG), que es el que mejor funciona en prueba de esfuerzo según nuestros conocimientos. Este método requiere de tres derivaciones ortogonales y es muy dependiente de cada una de ellas, i.e., el método no es aplicable o su rendimiento se reduce significativamente si hay algún problema en alguna de las derivaciones requeridas. En el Capítulo 2 se propone un método EDR nuevo basado en las pendientes del QRS y el ángulo de la onda R. El Capítulo 3 aborda a obtención de información respiratoria a partir de la señal PPG. Se propone un método nuevo para obtener la tasa respiratoria a partir de la señal PPG. Explota una modulación respiratoria en la variabilidad de anchura de pulso (PWV) relacionada con la velocidad y dispersión de la onda de pulso. El Capítulo 4 aborda la extracción de información respiratoria a partir de señales PPG registradas con smarthpones (SCPPG), mediante la adaptación de los métodos basados en la señal PPG presentados en el Capítulo 3. En el Capítulo 5 se propone un método para el diagnóstico del síndrome de apnea obstructiva del sueño (OSAS) en niños basado únicamente en la señal PPG. El OSAS es una disfunción relacionada con la respiración y el sueño que se diagnostica mediante polisomnografía (PSG). La PSG es el registro nocturno de muchas señales durante el sueño, siendo muy difícil de aplicar en entornos ambulatorios. El método que presenta esta tesis está enfocado a diagnosticar el OSAS en niños utilizando únicamente la señal PPG que permitiría considerar un diagnóstico ambulatorio con sus ventajas económicas y sociales. Finalmente, el Capítulo 6 resume las contribuciones originales y las conclusiones principales de esta tesis, y propone posibles extensiones del trabajo. CONCLUSIÓN El método presentado en el Capítulo 2 para estimar la tasa respiratoria a partir de las pendientes del complejo QRS y el ángulo de la onda R en el ECG demostró ser robusto en entornos altamente no estacionarios y ruidosos y por tanto ser aplicable durante ejercicio incluyendo entrenamiento deportivo. Además, es independiente de un conjunto específico de derivaciones y, por tanto, un problema en alguna de ellas no implica una reducción considerable del rendimiento. El método presentado en el Capítulo 3 para estimar la tasa respiratoria a partir de la PWV extraída de la señal PPG está mucho menos afectada por el tono simpático que otros métodos presentados en la literatura que suelen basarse en la amplitud y/o la tasa de pulso. Esto permite una mayor precisión que otros métodos basados en PPG. Además, se propone un método para combinar información de diferentes señales respiratorias, y se utiliza para estimar la tasa respiratoria a partir de la PWV en combinación con otros métodos basados en la señal PPG, mejorando la precisión de la estimación incluso en comparación con otros métodos en la literatura que requieren el ECG o la presión sanguínea. Los métodos propuestos en el Capítulo 4 para estimar la tasa respiratoria mediante señales SCPPG estimaron de forma precisa la tasa respiratoria en sus rangos espontáneos habituales (0.2-0.4 Hz) e incluso a tasas más altas (hasta 0.5 Hz o 0.6 Hz, dependiendo del dispositivo utilizado). El único requerimiento es que el smartphone tenga un luz tipo flash y una cámara para grabar una yema del dedo sobre ella. La popularidad de los smartphones los convierte en dispositivos de acceso y aceptación r¿apidos. Así, para la población general es potencialmente aceptable un método que funciona en smartphones, pudiendo facilitar la medida de algunas constantes vitales utilizando solo la yema del dedo. El método presentado en el Capítulo 5 para el diagnóstico del OSAS en niños a partir de la PPG obtuvo una precisión suficiente para la clínica, aunque antes de ser aplicado en dicho entorno, el método debería ser validado en una base de datos más grande.The main objective of this thesis is to develop non-invasive methods for respiration information extraction from two biomedical signals which are widely adopted in clinical routine: the electrocardiogram (ECG) and the pulse photoplethysmographic (PPG) signal. This study is motivated by the desirability of monitoring respiratory information from non-invasive devices allowing to substitute the current respiration-monitoring techniques which may interfere with natural breathing and which are unmanageable in some applications such as stress test or sleep studies. Furthermore, if these noninvasive devices are those already used in the clinical routine, the respiratory information obtained from them represents an added value which allows a more complete overview of the patient status. This thesis is divided into 6 chapters. Chapter 1 of this thesis introduces the problematic, motivations and objectives of this study. It also introduces the physiological origin of studied ECG and PPG signals, and why and how they carry autonomic- and respiration-related information which can be extracted from them. Chapter 2 of this thesis addresses the derivation of respiratory information from ECG signal. Several ECG derived respiration (EDR) methods have been presented in literature. Their performance usually decrease considerably in highly non-stationary and noisy environments such as stress test. However, some alternatives aimed to this kind of environments have been presented, such as one based on electrical axis rotation angles (obtained from the ECG), which to the best of our knowledge was the best suited for stress test. This method requires three orthogonal leads, and it is very dependent on each one of those leads, i.e., the performance of the method is significantly decreased if there is any problem at any one of the required leads. A novel EDR method based on QRS slopes and R-wave angle is presented in this thesis. The proposed method demonstrated to be robust in highly non-stationary and noisy environments and so to be applicable to exercise conditions including sports training. Furthermore, it is independent on a specific lead set, and so, a problem at any lead do not imply a significantly reduction of the performance. Chapter 3 addresses the derivation of respiratory information from PPG signals. A novel method for deriving respiratory rate from PPG signal is presented. It exploits respiration-related modulations in pulse width variability (PWV) which is related to pulse wave velocity and dispersion. The proposed method is much less affected by the sympathetic tone than other methods in literature which are usually based on pulses amplitude and/or rate. This leads to highest accuracy than other PPG-based method. Furthermore, a method for combining information from several respiratory signals was developed and used to obtain a respiratory rate estimation from the proposed PWV-based in combination with other known PPG-based methods, improving the accuracy of the estimation and outperforming other methods in literature which involve ECG or BP recording. Chapter 4 addresses the derivation of respiratory information from smartphone- camera-acquired-PPG (SCPPG) signals by adapting the methods for deriving respiratory rate from PPG signal presented in Chapter 3. The proposed method accurately estimates respiratory rate from SCPPG signals at its normal spontaneous ranges (0.2-0.4 Hz) and even at higher rates (up to 0.5 Hz or 0.6 Hz, depending on the used device). The only requirement is that these smartphones and tablets contain a flashlight and a video camera to image a fingertip pressed to it. As smartphones and tablets have become common, they meet the criteria of ready access and acceptance. Hence, a mobile phone/tablet approach has the potential to be widely-accepted by the general population and can facilitate the capability to measure some of the vital signs using only fingertip of the subject. Chapter 5 of this thesis proposes a methodology for obstructive sleep apnea syndrome (OSAS) screening in children just based on PPG signal. OSAS is a sleep-respiration-related dysfunction for which polysomnography (PSG) is the gold standard for diagnosis. PSG consists of overnight recording of many signals during sleep, therefore, it is quite involved and difficult to use in ambulatory scenario. The method presented in this thesis is aimed to diagnose the OSAS in children based just on PPG signal which would allow us to consider an ambulatory diagnosis with both its social and economic advantages. Finally, Chapter 6 summarizes the original contributions and main conclusions of the thesis, and proposes possible extensions of the work