BEst (Biomarker Estimation): Health Biomarker Estimation Non-invasively and Ubiquitously

This dissertation focuses on the non-invasive assessment of blood-hemoglobin levels. The primary goal of this research is to investigate a reliable, affordable, and user-friendly point-of-care solution for hemoglobin-level determination using fingertip videos captured by a smartphone. I evaluated videos obtained from five patient groups, three from the United States and two from Bangladesh, under two sets of lighting conditions. In the last group, based on human tissue optical transmission modeling data, I used near-infrared light-emitting diode sources of three wavelengths. I developed novel image processing techniques for fingertip video analysis to estimate hemoglobin levels. I studied video images creating image histogram and subdividing each image into multiple blocks. I determined the region of interest in a video and created photoplethysmogram signals. I created features from image histograms and PPG signals. I used the Partial Least Squares Regression and Support Vector Machine Regression tools to analyze input features and to build hemoglobin prediction models. Using data from the last and largest group of patients studied, I was able to develop a model with a strong linear correlation between estimated and clinically-measured hemoglobin levels. With further data and methodological refinements, the approach I have developed may be able to define a clinically accurate public health applicable tool for hemoglobin level and other blood constituent assessment

Measuring 2D:4D finger length ratios with Smartphone Cameras

Finger length ratios have received much attention among researchers as the 2D:4D ratio has been linked to several physical and mental characteristics. This study explores the feasibility of using a Smartphone as an instrument for measuring finger length ratios. The approach taken in this study is to use the Smartphone camera to take freehand photos of the hand which is subsequently subjected to image analysis. Measurement procedures include hand near and far from the body, palms up or down, or hands in mid air versus hands resting on a flat surface. Experimental evaluations show that the most accurate measurements are achieved by resting the hand on a surface with the palm facing up. These results are comparable to those achieved with conventional procedures with an error of 1%

Improving Mobile MOOC Learning via Implicit Physiological Signal Sensing

Massive Open Online Courses (MOOCs) are becoming a promising solution for delivering high- quality education on a large scale at low cost in recent years. Despite the great potential, today’s MOOCs also suffer from challenges such as low student engagement, lack of personalization, and most importantly, lack of direct, immediate feedback channels from students to instructors. This dissertation explores the use of physiological signals implicitly collected via a "sensorless" approach as a rich feedback channel to understand, model, and improve learning in mobile MOOC contexts. I first demonstrate AttentiveLearner, a mobile MOOC system which captures learners' physiological signals implicitly during learning on unmodified mobile phones. AttentiveLearner uses on-lens finger gestures for video control and monitors learners’ photoplethysmography (PPG) signals based on the fingertip transparency change captured by the back camera. Through series of usability studies and follow-up analyses, I show that the tangible video control interface of AttentiveLearner is intuitive to use and easy to operate, and the PPG signals implicitly captured by AttentiveLearner can be used to infer both learners’ cognitive states (boredom and confusion levels) and divided attention (multitasking and external auditory distractions). Building on top of AttentiveLearner, I design, implement, and evaluate a novel intervention technology, Context and Cognitive State triggered Feed-Forward (C2F2), which infers and responds to learners’ boredom and disengagement events in real time via a combination of PPG-based cognitive state inference and learning topic importance monitoring. C2F2 proactively reminds a student of important upcoming content (feed-forward interventions) when disengagement is detected. A 48-participant user study shows that C2F2 on average improves learning gains by 20.2% compared with a non-interactive baseline system and is especially effective for bottom performers (improving their learning gains by 41.6%). Finally, to gain a holistic understanding of the dynamics of MOOC learning, I investigate the temporal dynamics of affective states of MOOC learners in a 22 participant study. Through both a quantitative analysis of the temporal transitions of affective states and a qualitative analysis of subjective feedback, I investigate differences between mobile MOOC learning and complex learning activities in terms of affect dynamics, and discuss pedagogical implications in detail

Una contribución a la evaluación de la adherencia a hábitos de vida saludables basado en aplicaciones móviles

The adherence to a healthy lifestyle plays a key role for increasing life expectancy and living better. The main habits of healthy lifestyle are: physical activity, diet and sleep quality. Nowadays, many people use a smartphone and carry it all day. The objective of this thesis is to demonstrate the feasibility of the evaluation of the adherence to a healthy lifestyle by means of smartphone applications and sensors, whether internal or externally connected. On the one hand, the accelerometer sensor is used to evaluate the physical activity and the associated energy expenditure. In previous research, we can found classifiers of physical activity from data of this sensor but the measurements were performed in a laboratory environment or with smartphone fixed to a specific position. From the collected data during a week of 26 subjects, a 75.6% of F1-score of the classification of activities has been achieved and a 3.18% of error in the energy expenditure estimation. On the other hand, the heart rate variability (HRV) can serve as indicator of behaviours related to health and physical condition. A system has been designed to evaluate the HRV using the rear camera of the smartphone as a sensor. For this purpose, the photoplethysmography technique has been used. In previous research, this technique has been used in smartphones in order to obtain the heart rate but it has not been assessed the beat-to-beat HRV. The proposed system uses the GPU for image processing in real time. The obtained results have been compared with the electrocardiogram and with a reference photoplethysmography device. For that, the standard deviation of error made for the beat detection and the level of agreement of HRV indices have been assessed. This assessment has been performed with 23 subjects and the results obtained for two different smartphone models have been compared. The standard deviation of error of heart rate detection between smartphone and electrocardiogram obtained was 5.4 ms, while between electrocardiogram and reference photoplethysmography device was 4.9 ms. On the other hand, an application for the ensemble analysis of physical activity and heart rate has been developed. Using this application, the data of 11 people was collected, they have divided in two groups of 5 and 6 people during 3 and 6 weeks respectively. From the analysis of the collected data, it has been found that the level of physical activity decreases over the time and there is some association between the constancy of the practice of physical activity and changes in mood. However, these association should be taken with caution due to the reduced number of subjects which were involved in this study. Therefore, the developed system is a starting point in order to evaluate the adherence to a healthy lifestyle in a unified way with an single application. Finally, one of the consequences of leading an unhealthy lifestyle is the decreasing of quality of sleep that can cause daytime sleepiness. This can be a serious health risk, for example if it occurs while driving. To prevent this, an early drowsiness detection system based on the analysis of respiratory signal and respiratory rate variability has been proposed and validated. The designed algorithm has been assessed with 15 subjects and a specificity of 96.6% and a sensitivity of 90.3% has been obtained.La adherencia a un estilo de vida saludable es un factor muy importante para alargar años de vida y aumentar su calidad. Los principales hábitos de vida saludable son: la actividad física, la dieta y la calidad del sueño. Hoy en día muchas personas utilizan un smartphone y lo llevan encima todo el día. El objetivo de esta tesis es demostrar la viabilidad de la evaluación de la adherencia a hábitos de vida saludables mediante aplicaciones móviles y sensores ya sean del propio smartphone o conectados externamente. Para ello, se utiliza el sensor de acelerometría para evaluar la actividad física y el gasto calórico asociado. En trabajos previos podemos encontrar clasificadores de actividad física a partir de los datos de estos sensores pero las medidas las realizan en un entorno de laboratorio o con el smartphone ubicado en una posición determinada. A partir de los datos de 26 sujetos recogidos durante una semana se ha alcanzado un 75.6% de F1-score de la clasificación de actividades y un 3.18% de error de estimación de gasto calórico. Por otro lado, la variabilidad de la frecuencia cardíaca (VFC) puede servir de indicador de conductas relacionadas con la salud y la condición física. Se ha diseñado un sistema para evaluar la VFC utilizando como sensor la cámara trasera del smartphone. Para ello se ha utilizado la técnica de fotopletismografía. En trabajos previos se ha utilizado esta técnica en smartphones para obtener el ritmo cardíaco pero no se ha comparado la variabilidad del ritmo cardíaco latido a latido. El sistema propuesto utiliza la GPU para procesar la imágenes en tiempo real. Los resultados obtenidos se han comparado con el electrocardiograma y con un dispositivo de fotopletismografía de referencia. Para ello, se ha evaluado la desviación estándar del error cometido en la detección del latido cardíaco y el grado de acuerdo de los índices de VFC. Esta evaluación se ha realizado en 23 sujetos y se han comparado los resultados obtenidos con dos modelos de smartphone. La desviación estándar del error en la detección del latido cardíaco obtenida entre el smartphone y el electrocardiograma es de 5.4 ms, mientras que entre el dispositivo de referencia de fotopletismografía y el electrocardiograma es de 4.9 ms. Por otro lado, se ha desarrollado una aplicación para el análisis conjunto de la actividad física y el ritmo cardíaco. Se recogieron los datos de 11 personas utilizando esta aplicación, divididas en dos grupos de 5 y 6 personas durante 3 y 6 semanas respectivamente. A partir del análisis de los datos recogidos se ha encontrado que el nivel de la actividad física desciende a lo largo del tiempo y que existe alguna asociación entre la constancia en la práctica de la actividad física y los cambios en el estado de ánimo. Sin embargo, estas asociaciones se han de tomar con precaución debido al reducido número de sujetos que han participado en este estudio. Por lo tanto, el sistema desarrollado supone un punto de partida para evaluar la adherencia a un estilo de vida saludable de forma unificada en una única aplicación. Finalmente, una de las consecuencias de llevar un estilo de vida poco saludable es el empobrecimiento de la calidad del sueño que puede provocar la somnolencia diurna. Esto puede resultar un grave peligro para la salud, por ejemplo si se produce mientras se está al volante. Para prevenir esto, se ha propuesto y validado un sistema de detección de somnolencia temprana a partir del análisis de la señal respiratoria basado en la variabilidad del ritmo respiratorio. El algoritmo diseñado ha sido validado con 15 sujetos y se ha obtenido una especificidad del 96.6% y una sensibilidad del 90.3%

Una contribución a la evaluación de la adherencia a hábitos de vida saludables basado en aplicaciones móviles

The adherence to a healthy lifestyle plays a key role for increasing life expectancy and living better. The main habits of healthy lifestyle are: physical activity, diet and sleep quality. Nowadays, many people use a smartphone and carry it all day. The objective of this thesis is to demonstrate the feasibility of the evaluation of the adherence to a healthy lifestyle by means of smartphone applications and sensors, whether internal or externally connected. On the one hand, the accelerometer sensor is used to evaluate the physical activity and the associated energy expenditure. In previous research, we can found classifiers of physical activity from data of this sensor but the measurements were performed in a laboratory environment or with smartphone fixed to a specific position. From the collected data during a week of 26 subjects, a 75.6% of F1-score of the classification of activities has been achieved and a 3.18% of error in the energy expenditure estimation. On the other hand, the heart rate variability (HRV) can serve as indicator of behaviours related to health and physical condition. A system has been designed to evaluate the HRV using the rear camera of the smartphone as a sensor. For this purpose, the photoplethysmography technique has been used. In previous research, this technique has been used in smartphones in order to obtain the heart rate but it has not been assessed the beat-to-beat HRV. The proposed system uses the GPU for image processing in real time. The obtained results have been compared with the electrocardiogram and with a reference photoplethysmography device. For that, the standard deviation of error made for the beat detection and the level of agreement of HRV indices have been assessed. This assessment has been performed with 23 subjects and the results obtained for two different smartphone models have been compared. The standard deviation of error of heart rate detection between smartphone and electrocardiogram obtained was 5.4 ms, while between electrocardiogram and reference photoplethysmography device was 4.9 ms. On the other hand, an application for the ensemble analysis of physical activity and heart rate has been developed. Using this application, the data of 11 people was collected, they have divided in two groups of 5 and 6 people during 3 and 6 weeks respectively. From the analysis of the collected data, it has been found that the level of physical activity decreases over the time and there is some association between the constancy of the practice of physical activity and changes in mood. However, these association should be taken with caution due to the reduced number of subjects which were involved in this study. Therefore, the developed system is a starting point in order to evaluate the adherence to a healthy lifestyle in a unified way with an single application. Finally, one of the consequences of leading an unhealthy lifestyle is the decreasing of quality of sleep that can cause daytime sleepiness. This can be a serious health risk, for example if it occurs while driving. To prevent this, an early drowsiness detection system based on the analysis of respiratory signal and respiratory rate variability has been proposed and validated. The designed algorithm has been assessed with 15 subjects and a specificity of 96.6% and a sensitivity of 90.3% has been obtained.La adherencia a un estilo de vida saludable es un factor muy importante para alargar años de vida y aumentar su calidad. Los principales hábitos de vida saludable son: la actividad física, la dieta y la calidad del sueño. Hoy en día muchas personas utilizan un smartphone y lo llevan encima todo el día. El objetivo de esta tesis es demostrar la viabilidad de la evaluación de la adherencia a hábitos de vida saludables mediante aplicaciones móviles y sensores ya sean del propio smartphone o conectados externamente. Para ello, se utiliza el sensor de acelerometría para evaluar la actividad física y el gasto calórico asociado. En trabajos previos podemos encontrar clasificadores de actividad física a partir de los datos de estos sensores pero las medidas las realizan en un entorno de laboratorio o con el smartphone ubicado en una posición determinada. A partir de los datos de 26 sujetos recogidos durante una semana se ha alcanzado un 75.6% de F1-score de la clasificación de actividades y un 3.18% de error de estimación de gasto calórico. Por otro lado, la variabilidad de la frecuencia cardíaca (VFC) puede servir de indicador de conductas relacionadas con la salud y la condición física. Se ha diseñado un sistema para evaluar la VFC utilizando como sensor la cámara trasera del smartphone. Para ello se ha utilizado la técnica de fotopletismografía. En trabajos previos se ha utilizado esta técnica en smartphones para obtener el ritmo cardíaco pero no se ha comparado la variabilidad del ritmo cardíaco latido a latido. El sistema propuesto utiliza la GPU para procesar la imágenes en tiempo real. Los resultados obtenidos se han comparado con el electrocardiograma y con un dispositivo de fotopletismografía de referencia. Para ello, se ha evaluado la desviación estándar del error cometido en la detección del latido cardíaco y el grado de acuerdo de los índices de VFC. Esta evaluación se ha realizado en 23 sujetos y se han comparado los resultados obtenidos con dos modelos de smartphone. La desviación estándar del error en la detección del latido cardíaco obtenida entre el smartphone y el electrocardiograma es de 5.4 ms, mientras que entre el dispositivo de referencia de fotopletismografía y el electrocardiograma es de 4.9 ms. Por otro lado, se ha desarrollado una aplicación para el análisis conjunto de la actividad física y el ritmo cardíaco. Se recogieron los datos de 11 personas utilizando esta aplicación, divididas en dos grupos de 5 y 6 personas durante 3 y 6 semanas respectivamente. A partir del análisis de los datos recogidos se ha encontrado que el nivel de la actividad física desciende a lo largo del tiempo y que existe alguna asociación entre la constancia en la práctica de la actividad física y los cambios en el estado de ánimo. Sin embargo, estas asociaciones se han de tomar con precaución debido al reducido número de sujetos que han participado en este estudio. Por lo tanto, el sistema desarrollado supone un punto de partida para evaluar la adherencia a un estilo de vida saludable de forma unificada en una única aplicación. Finalmente, una de las consecuencias de llevar un estilo de vida poco saludable es el empobrecimiento de la calidad del sueño que puede provocar la somnolencia diurna. Esto puede resultar un grave peligro para la salud, por ejemplo si se produce mientras se está al volante. Para prevenir esto, se ha propuesto y validado un sistema de detección de somnolencia temprana a partir del análisis de la señal respiratoria basado en la variabilidad del ritmo respiratorio. El algoritmo diseñado ha sido validado con 15 sujetos y se ha obtenido una especificidad del 96.6% y una sensibilidad del 90.3%.Postprint (published version

Reconstructing Human Motion

This thesis presents methods for reconstructing human motion in a variety of applications and begins with an introduction to the general motion capture hardware and processing pipeline. Then, a data-driven method for the completion of corrupted marker-based motion capture data is presented. The approach is especially suitable for challenging cases, e.g., if complete marker sets of multiple body parts are missing over a long period of time. Using a large motion capture database and without the need for extensive preprocessing the method is able to fix missing markers across different actors and motion styles. The approach can be used for incrementally increasing prior-databases, as the underlying search technique for similar motions scales well to huge databases. The resulting clean motion database could then be used in the next application: a generic data-driven method for recognizing human full body actions from live motion capture data originating from various sources. The method queries an annotated motion capture database for similar motion segments, able to handle temporal deviations from the original motion. The approach is online-capable, works in realtime, requires virtually no preprocessing and is shown to work with a variety of feature sets extracted from input data including positional data, sparse accelerometer signals, skeletons extracted from depth sensors and even video data. Evaluation is done by comparing against a frame-based Support Vector Machine approach on a freely available motion database as well as a database containing Judo referee signal motions. In the last part, a method to indirectly reconstruct the effects of the human heart's pumping motion from video data of the face is applied in the context of epileptic seizures. These episodes usually feature interesting heart rate patterns like a significant increase at seizure start as well as seizure-type dependent drop-offs near the end. The pulse detection method is evaluated for applicability regarding seizure detection in a multitude of scenarios, ranging from videos recorded in a controlled clinical environment to patient supplied videos of seizures filmed with smartphones

Clinical Decision Support Systems with Game-based Environments, Monitoring Symptoms of Parkinson’s Disease with Exergames

Parkinson’s Disease (PD) is a malady caused by progressive neuronal degeneration, deriving in several physical and cognitive symptoms that worsen with time. Like many other chronic diseases, it requires constant monitoring to perform medication and therapeutic adjustments. This is due to the significant variability in PD symptomatology and progress between patients. At the moment, this monitoring requires substantial participation from caregivers and numerous clinic visits. Personal diaries and questionnaires are used as data sources for medication and therapeutic adjustments. The subjectivity in these data sources leads to suboptimal clinical decisions. Therefore, more objective data sources are required to better monitor the progress of individual PD patients. A potential contribution towards more objective monitoring of PD is clinical decision support systems. These systems employ sensors and classification techniques to provide caregivers with objective information for their decision-making. This leads to more objective assessments of patient improvement or deterioration, resulting in better adjusted medication and therapeutic plans. Hereby, the need to encourage patients to actively and regularly provide data for remote monitoring remains a significant challenge. To address this challenge, the goal of this thesis is to combine clinical decision support systems with game-based environments. More specifically, serious games in the form of exergames, active video games that involve physical exercise, shall be used to deliver objective data for PD monitoring and therapy. Exergames increase engagement while combining physical and cognitive tasks. This combination, known as dual-tasking, has been proven to improve rehabilitation outcomes in PD: recent randomized clinical trials on exergame-based rehabilitation in PD show improvements in clinical outcomes that are equal or superior to those of traditional rehabilitation. In this thesis, we present an exergame-based clinical decision support system model to monitor symptoms of PD. This model provides both objective information on PD symptoms and an engaging environment for the patients. The model is elaborated, prototypically implemented and validated in the context of two of the most prominent symptoms of PD: (1) balance and gait, as well as (2) hand tremor and slowness of movement (bradykinesia). While balance and gait affections increase the risk of falling, hand tremors and bradykinesia affect hand dexterity. We employ Wii Balance Boards and Leap Motion sensors, and digitalize aspects of current clinical standards used to assess PD symptoms. In addition, we present two dual-tasking exergames: PDDanceCity for balance and gait, and PDPuzzleTable for tremor and bradykinesia. We evaluate the capability of our system for assessing the risk of falling and the severity of tremor in comparison with clinical standards. We also explore the statistical significance and effect size of the data we collect from PD patients and healthy controls. We demonstrate that the presented approach can predict an increased risk of falling and estimate tremor severity. Also, the target population shows a good acceptance of PDDanceCity and PDPuzzleTable. In summary, our results indicate a clear feasibility to implement this system for PD. Nevertheless, long-term randomized clinical trials are required to evaluate the potential of PDDanceCity and PDPuzzleTable for physical and cognitive rehabilitation effects

Low-Cost Sensors and Biological Signals

Many sensors are currently available at prices lower than USD 100 and cover a wide range of biological signals: motion, muscle activity, heart rate, etc. Such low-cost sensors have metrological features allowing them to be used in everyday life and clinical applications, where gold-standard material is both too expensive and time-consuming to be used. The selected papers present current applications of low-cost sensors in domains such as physiotherapy, rehabilitation, and affective technologies. The results cover various aspects of low-cost sensor technology from hardware design to software optimization

Mobile Thermography-based Physiological Computing for Automatic Recognition of a Person’s Mental Stress

This thesis explores the use of Mobile Thermography1, a significantly less investigated sensing capability, with the aim of reliably extracting a person’s multiple physiological signatures and recognising mental stress in an automatic, contactless manner. Mobile thermography has greater potentials for real-world applications because of its light-weight, low computation-cost characteristics. In addition, thermography itself does not necessarily require the sensors to be worn directly on the skin. It raises less privacy concerns and is less sensitive to ambient lighting conditions. The work presented in this thesis is structured through a three-stage approach that aims to address the following challenges: i) thermal image processing for mobile thermography in variable thermal range scenes; ii) creation of rich and robust physiology measurements; and iii) automated stress recognition based on such measurements. Through the first stage (Chapter 4), this thesis contributes new processing techniques to address negative effects of environmental temperature changes upon automatic tracking of regions-of-interest and measuring of surface temperature patterns. In the second stage (Chapters 5,6,7), the main contributions are: robustness in tracking respiratory and cardiovascular thermal signatures both in constrained and unconstrained settings (e.g. respiration: strong correlation with ground truth, r=0.9987), and investigation of novel cortical thermal signatures associated with mental stress. The final stage (Chapters 8,9) contributes automatic stress inference systems that focus on capturing richer dynamic information of physiological variability: firstly, a novel respiration representation-based system (which has achieved state-of-the-art performance: 84.59% accuracy, two stress levels), and secondly, a novel cardiovascular representation-based system using short-term measurements of nasal thermal variability and heartrate variability from another sensing channel (78.33% accuracy achieved from 20seconds measurements). Finally, this thesis contributes software libraries and incrementally built labelled datasets of thermal images in both constrained and everyday ubiquitous settings. These are used to evaluate performance of our proposed computational methods across the three-stages

Pulse Rate Variability for the Assessment of Cardiovascular Changes

Pulse rate variability (PRV) describes the way pulse rate changes through time and is measured from pulsatile signals such as the photoplethysmogram (PPG). It has been proposed as a surrogate for heart rate variability (HRV). Nonetheless, the relationship between these variables is not entirely clear, probably due to both physiological and technical aspects involved in the extraction of PRV. Moreover, the effects of cardiovascular changes on PRV have not been elucidated. In this thesis, four studies were performed to (1) determine the best combination of some technical aspects for the extraction of PRV from PPG signals; (2) evaluate the relationship between PRV and HRV under different cardiovascular conditions; and (3) explore the effects of cardiovascular changes on PRV. First, PRV extraction gave lower errors when (1) signals were acquired for at least 120 s with a 256 Hz sampling rate and filtered with lower low cut-off frequencies and elliptic, equiripple or Parks-McClellan filter; (2) cardiac cycles were determined using the D2max algorithm and the a fiducial points; and (3) the Fast Fourier Transform was applied to obtain frequency spectra. Secondly, the relationship between HRV and PRV was found to be affected by cold exposure and changes in blood pressure, while PRV was found to be different at different body sites. Finally, PRV was affected by haemodynamic changes, such as target flow, stroke rate and blood pressure, both in an in-vitro model and in-vivo data. Additionally, PRV was found to be a potential tool for the estimation of blood pressure, with errors as low as 1:54 ± 0:17 mmHg, 1:07 ± 0:06 mmHg and 1:22 ± 0:09 mmHg for the estimation of systolic, diastolic and mean arterial pressure. Although more studies are needed to fully understand PRV and its clinical potential, PRV should not be regarded as the same as HRV, and it could be consider as a potential valuable biomarker for cardiovascular health