Predicting emotional states using behavioral markers derived from passively sensed data: Data-driven machine learning approach

Background: Mental health disorders affect multiple aspects of patients’ lives, including mood, cognition, and behavior. eHealth and mobile health (mHealth) technologies enable rich sets of information to be collected noninvasively, representing a promising opportunity to construct behavioral markers of mental health. Combining such data with self-reported information about psychological symptoms may provide a more comprehensive and contextualized view of a patient’s mental state than questionnaire data alone. However, mobile sensed data are usually noisy and incomplete, with significant amounts of missing observations. Therefore, recognizing the clinical potential of mHealth tools depends critically on developing methods to cope with such data issues. Objective: This study aims to present a machine learning–based approach for emotional state prediction that uses passively collected data from mobile phones and wearable devices and self-reported emotions. The proposed methods must cope with high-dimensional and heterogeneous time-series data with a large percentage of missing observations. Methods: Passively sensed behavior and self-reported emotional state data from a cohort of 943 individuals (outpatients recruited from community clinics) were available for analysis. All patients had at least 30 days’ worth of naturally occurring behavior observations, including information about physical activity, geolocation, sleep, and smartphone app use. These regularly sampled but frequently missing and heterogeneous time series were analyzed with the following probabilistic latent variable models for data averaging and feature extraction: mixture model (MM) and hidden Markov model (HMM). The extracted features were then combined with a classifier to predict emotional state. A variety of classical machine learning methods and recurrent neural networks were compared. Finally, a personalized Bayesian model was proposed to improve performance by considering the individual differences in the data and applying a different classifier bias term for each patient. Results: Probabilistic generative models proved to be good preprocessing and feature extractor tools for data with large percentages of missing observations. Models that took into account the posterior probabilities of the MM and HMM latent states outperformed those that did not by more than 20%, suggesting that the underlying behavioral patterns identified were meaningful for individuals’ overall emotional state. The best performing generalized models achieved a 0.81 area under the curve of the receiver operating characteristic and 0.71 area under the precision-recall curve when predicting self-reported emotional valence from behavior in held-out test data. Moreover, the proposed personalized models demonstrated that accounting for individual differences through a simple hierarchical model can substantially improve emotional state prediction performance without relying on previous days’ data. Conclusions: These findings demonstrate the feasibility of designing machine learning models for predicting emotional states from mobile sensing data capable of dealing with heterogeneous data with large numbers of missing observations. Such models may represent valuable tools for clinicians to monitor patients’ mood states.This project has received funding from the European Union's Horizon 2020 Research and Innovation Program under the Marie Sklodowska-Curie grant agreement number 813533. This work was partly supported by the Spanish government (Ministerio de Ciencia e Innovación) under grants TEC2017-92552-EXP and RTI2018-099655-B-100; the Comunidad de Madrid under grants IND2017/TIC-7618, IND2018/TIC-9649, IND2020/TIC-17372, and Y2018/TCS-4705; the BBVA Foundation under the Domain Alignment and Data Wrangling with Deep Generative Models (Deep-DARWiN) project; and the European Union (European Regional Development Fund and the European Research Council) through the European Union's Horizon 2020 Research and Innovation Program under grant 714161. The authors thank Enrique Baca-Garcia for providing demographic and clinical data and assisting in interpreting and summarizing the data

A Framework for Secure and Survivable Wireless Sensor Networks

Wireless sensor networks increasingly become viable solutions to many challenging problems and will successively be deployed in many areas in the future. A wireless sensor network (WSN) is vulnerable to security attacks due to the insecure communication channels, limited computational and communication capabilities and unattended nature of sensor node devices, limited energy resources and memory. Security and survivability of these systems are receiving increasing attention, particularly critical infrastructure protection. So we need to design a framework that provide both security and survivability for WSNs. To meet this goals, we propose a framework for secure and survivable WSNs and we present a key management scheme as a case study to prevent the sensor networks being compromised by an adversary. This paper also considers survivability strategies for the sensor network against a variety of threats that can lead to the failure of the base station, which represents a central point of failure.key management scheme, security, survivability, WSN

Passive mobile sensing and psychological traits for large scale mood prediction

Experience sampling has long been the established method to sample people’s mood in order to assess their mental state. Smartphones have started to be used as experience sampling tools for mental health state as they accompany individuals during their day and can therefore gather in-the-moment data. However, the granularity of the data needs to be traded off with the level of interruption these tools introduce on users’ activities. As a consequence the data collected with this technique is often sparse. This has been obviated by the use of passive sensing in addition to mood reports, however this adds additional noise. In this paper we show that psychological traits collected through one-off questionnaires combined with passively collected sensing data (movement from the accelerometer and noise levels from the microphone) can be used to detect individuals whose general mood deviates from the common relaxed characteristic of the general population. By using the reported mood as a classification target we show how to design models that depend only on passive sensors and one-off questionnaires, without bothering users with tedious experience sampling. We validate our approach by using a large dataset of mood reports and passive sensing data collected in the wild with tens of thousands of participants, finding that the combination of these modalities has the best classification performance, and that passive sensing yields a +5% boost in accuracy. We also show that sensor data collected for the duration of a week performs better than when only using data collected for single days for this task. We discuss feature extraction techniques and appropriate classifiers for this kind of multimodal data, as well as overfitting shortcomings of using deep learning to handle static and dynamic features. We believe these findings have significant implications for mobile health applications that can benefit from the correct modeling of passive sensing along with extra user metadata.This work was partially funded by the Embiricos Trust Scholarship of Jesus College, Cambridge and the EPSRC Doctoral Training Partnership (grant reference EP/N509620/1)

Happier People Live More Active Lives: Using Smartphones to Link Happiness and Physical Activity.

Physical activity, both exercise and non-exercise, has far-reaching benefits to physical health. Although exercise has also been linked to psychological health (e.g., happiness), little research has examined physical activity more broadly, taking into account non-exercise activity as well as exercise. We examined the relationship between physical activity (measured broadly) and happiness using a smartphone application. This app has collected self-reports of happiness and physical activity from over ten thousand participants, while passively gathering information about physical activity from the accelerometers on users' phones. The findings reveal that individuals who are more physically active are happier. Further, individuals are happier in the moments when they are more physically active. These results emerged when assessing activity subjectively, via self-report, or objectively, via participants' smartphone accelerometers. Overall, this research suggests that not only exercise but also non-exercise physical activity is related to happiness. This research further demonstrates how smartphones can be used to collect large-scale data to examine psychological, behavioral, and health-related phenomena as they naturally occur in everyday life.Engineering and Physical Sciences Research Council (UBhave project (Ubiquitous and Social Computing for Positive Behaviour Change, Grant ID: EP/I032673/1))This is the final version of the article. It first appeared from Public Library of Science via https://doi.org/http://dx.doi.org/10.1371/journal.pone.016058

A step towards Advancing Digital Phenotyping In Mental Healthcare

Smartphones and wrist-wearable devices have infiltrated our lives in recent years. According to published statistics, nearly 84% of the world’s population owns a smartphone, and almost 10% own a wearable device today (2022). These devices continuously generate various data sources from multiple sensors and apps, creating our digital phenotypes. This opens new research opportunities, particularly in mental health care, which has previously relied almost exclusively on self-reports of mental health symptoms. Unobtrusive monitoring using patients’ devices may result in clinically valuable markers that can improve diagnostic processes, tailor treatment choices, provide continuous insights into their condition for actionable outcomes, such as early signs of relapse, and develop new intervention models. However, these data sources must be translated into meaningful, actionable features related to mental health to achieve their full potential. In the mental health field, there is a great need and much to be gained from defining a way to continuously assess the evolution of patients’ mental states, ideally in their everyday environment, to support the monitoring and treatments by health care providers. A smartphone-based approach may be valuable in gathering long-term objective data, aside from the usually used self-ratings, to predict clinical state changes and investigate causal inferences about state changes in patients (e.g., those with affective disorders). Being objective does not imply that passive data collection is also perfect. It has several challenges: some sensors generate vast volumes of data, and others cause significant battery drain. Furthermore, the analysis of raw passive data is complicated, and collecting certain types of data may interfere with the phenotype of interest. Nonetheless, machine learning is predisposed to address these matters and advance psychiatry’s era of personalised medicine. This work aimed to advance the research efforts on mobile and wearable sensors for mental health monitoring. We applied supervised and unsupervised machine learning methods to model and understand mental disease evolution based on the digital phenotype of patients and clinician assessments at the follow-up visits, which provide ground truths. We needed to cope with regularly and irregularly sampled, high-dimensional, and heterogeneous time series data susceptible to distortion and missingness. Hence, the developed methods must be robust to these limitations and handle missing data properly. Throughout the various projects presented here, we used probabilistic latent variable models for data imputation and feature extraction, namely, mixture models (MM) and hidden Markov models (HMM). These unsupervised models can learn even in the presence of missing data by marginalising the missing values in the function of the present observations. Once the generative models are trained on the data set with missing values, they can be used to generate samples for imputation. First, the most probable component/state has to be found for each sample. Then, sampling from the most probable distribution yields valid and robust parameter estimates and explicit imputed values for variables that can be analysed as outcomes or predictors. The imputation process can be repeated several times, creating multiple datasets, thereby accounting for the uncertainty in the imputed values and implicitly augmenting the data. Moreover, they are robust to moderate deviations of the observed data from the assumed underlying distribution and provide accurate estimates even when missingness is high. Depending on the properties of the data at hand, we employed feature extraction methods combined with classical machine learning algorithms or deep learning-based techniques for temporal modelling to predict various mental health outcomes - emotional state, World Health Organisation Disability Assessment Schedule (WHODAS 2.0) functionality scores and Generalised Anxiety Disorder-7 (GAD-7) scores, of psychiatric outpatients. We mainly focused on one-size-fits-all models, as the labelled sample size per patient was limited; however, in the mood prediction case, it was possible to apply personalised models. Integrating machines and algorithms into the clinical workflow require interpretability to increase acceptance. Therefore, we also analysed feature importance by computing Shapley additive explanations (SHAP) values. SHAP values provide an overview of essential features in the machine learning models by designating the weight of predictability of each feature positively or negatively to the target variable. The provided solutions, as such, are proof of concept, which require further clinical validation to be deployable in the clinical workflow. Still, the results are promising and lay some foundations for future research and collaboration among clinicians, patients, and computer scientists. They set the paths to advance future research prospects in technology-based mental healthcare.En los últimos años, los smartphones y los dispositivos y pulseras inteligentes, comúnmente conocidos como wearables, se han infiltrado en nuestras vidas. Según las estadísticas publicadas a día de hoy (2022), cerca del 84% de la población tiene un smartphone y aproximadamente un 10% también posee un wearable. Estos dispositivos generan datos de forma continua en base a distintos sensores y aplicaciones, creando así nuestro fenotipo digital. Estos datos abren nuevas vías de investigación, particularmente en el área de salud mental, dónde las fuentes de datos han sido casi exclusivamente autoevaluaciones de síntomas de salud mental. Monitorizar de forma no intrusiva a los pacientes mediante sus dispositivos puede dar lugar a marcadores valiosos en aplicación clínica. Esto permite mejorar los procesos de diagnóstico, adaptar tratamientos, e incluso proporcionar información continua sobre el estado de los pacientes, como signos tempranos de recaída, y hasta desarrollar nuevos modelos de intervención. Aun así, estos datos en crudo han de ser traducidos a datos interpretables relacionados con la salud mental para conseguir un máximo rendimiento de los mismos. En salud mental existe una gran necesidad, y además hay mucho que ganar, de definir cómo evaluar de forma continuada la evolución del estado mental de los pacientes en su entorno cotidiano para ayudar en el tratamiento y seguimiento de los mismos por parte de los profesionales sanitarios. En este ámbito, un enfoque basado en datos recopilados desde sus smartphones puede ser valioso para recoger datos objetivos a largo plazo al mismo tiempo que se acompaña de las autoevaluaciones utilizadas habitualmente. La combinación de ambos tipos de datos puede ayudar a predecir los cambios en el estado clínico de estos pacientes e investigar las relaciones causales sobre estos cambios (por ejemplo, en aquellos que padecen trastornos afectivos). Aunque la recogida de datos de forma pasiva tiene la ventaja de ser objetiva, también implica varios retos. Por un lado, ciertos sensores generan grandes volúmenes de datos, provocando un importante consumo de batería. Además, el análisis de los datos pasivos en crudo es complicado, y la recogida de ciertos tipos de datos puede interferir con el fenotipo que se quiera analizar. No obstante, el machine learning o aprendizaje automático, está predispuesto a resolver estas cuestiones y aportar avances en la medicina personalizada aplicada a psiquiatría. Esta tesis tiene como objetivo avanzar en la investigación de los datos recogidos por sensores de smartphones y wearables para la monitorización en salud mental. Para ello, aplicamos métodos de aprendizaje automático supervisado y no supervisado para modelar y comprender la evolución de las enfermedades mentales basándonos en el fenotipo digital de los pacientes. Estos resultados se comparan con las evaluaciones de los médicos en las visitas de seguimiento, que proporcionan las etiquetas reales. Para aplicar estos métodos hemos lidiado con datos provenientes de series temporales con alta dimensionalidad, muestreados de forma regular e irregular, heterogéneos y, además, susceptibles a presentar patrones de datos perdidos y/o distorsionados. Por lo tanto, los métodos desarrollados deben ser resistentes a estas limitaciones y manejar adecuadamente los datos perdidos. A lo largo de los distintos proyectos presentados en este trabajo, hemos utilizado modelos probabilísticos de variables latentes para la imputación de datos y la extracción de características, como por ejemplo, Mixture Models (MM) y hidden Markov Models (HMM). Estos modelos no supervisados pueden aprender incluso en presencia de datos perdidos, marginalizando estos valores en función de las datos que sí han sido observados. Una vez entrenados los modelos generativos en el conjunto de datos con valores perdidos, pueden utilizarse para imputar dichos valores generando muestras. En primer lugar, hay que encontrar el componente/estado más probable para cada muestra. Luego, se muestrea de la distirbución más probable resultando en estimaciones de parámetros robustos y válidos. Además, genera imputaciones explícitas que pueden ser tratadas como resultados. Este proceso de imputación puede repetirse varias veces, creando múltiples conjuntos de datos, con lo que se tiene en cuenta la incertidumbre de los valores imputados y aumentándose así, implícitamente, los datos. Además, estas imputaciones son resistentes a desviaciones que puedan existir en los datos observados con respecto a la distribución subyacente asumida y proporcionan estimaciones precisas incluso cuando la falta de datos es elevada. Dependiendo de las propiedades de los datos en cuestión, hemos usado métodos de extracción de características combinados con algoritmos clásicos de aprendizaje automático o técnicas basadas en deep learning o aprendizaje profundo para el modelado temporal. La finalidad de ambas opciones es ser capaces de predecir varios resultados de salud mental/estado emocional, como la puntuación sobre el World Health Organisation Disability Assessment Schedule (WHODAS 2.0), o las puntuaciones del generalised anxiety disorder-7 (GAD-7) de pacientes psiquiátricos ambulatorios. Nos centramos principalmente en modelos generalizados, es decir, no personalizados para cada paciente sino explicativos para la mayoría, ya que el tamaño de muestras etiquetada por paciente es limitado; sin embargo, en el caso de la predicción del estado de ánimo, puidmos aplicar modelos personalizados. Para que la integración de las máquinas y algoritmos dentro del flujo de trabajo clínico sea aceptada, se requiere que los resultados sean interpretables. Por lo tanto, en este trabajo también analizamos la importancia de las características sacadas por cada algoritmo en base a los valores de las explicaciones aditivas de Shapley (SHAP). Estos valores proporcionan una visión general de las características esenciales en los modelos de aprendizaje automático designando el peso, positivo o negativo, de cada característica en su predictibilidad sobre la variable objetivo. Las soluciones aportadas en esta tesis, como tales, son pruebas de concepto, que requieren una mayor validación clínica para poder ser desplegadas en el flujo de trabajo clínico. Aun así, los resultados son prometedores y sientan base para futuras investigaciones y colaboraciones entre clínicos, pacientes y científicos de datos. Éstas establecen las guías para avanzar en las perspectivas de investigación futuras en la atención sanitaria mental basada en la tecnología.Programa de Doctorado en Multimedia y Comunicaciones por la Universidad Carlos III de Madrid y la Universidad Rey Juan CarlosPresidente: David Ramírez García.- Secretario: Alfredo Nazábal Rentería.- Vocal: María Luisa Barrigón Estéve

Crowd sensing and forecasting for Smart Cities

Dissertação de mestrado integrado em Engenharia InformáticaA utilização de inteligência sob forma de tecnologia no nosso dia-a-dia é uma realidade em crescimento e, portanto, devemos fazer uso da tecnologia disponível para melhorar várias áreas do nosso quotidiano. Por exemplo, a tecnologia atual permite a conceção de sensores inteligentes, mais especificamente sensores de multidão, para detetar passiva mente dispositivos como smartphones ou smartwatches através de probe requests emitidos por estes dispositivos que, por sua vez, fazem parte de um processo de comunicação que ocorre sempre que o Wi-Fi dos dispositivos está ativado. Adicionalmente, crowd sensing - uma solução de Ambient Intelligence (AmI) - é estudada hoje em dia em várias áreas com bons resultados. Portanto, esta dissertação visa investigar e utilizar sensores de multidão para capturar passivamente dados acerca da densidade de multidões, explorar as capacidades do sensor escolhido, analisar e processar os dados para obter melhores estimativas, e conceber e desenvolver modelos de Machine Learning (ML) para prever a densidade nas áreas sensorizadas. Áreas nas quais o sensor de multidão está inserido - AmI, Smart Cities, Wi-Fi Probing - são estudadas, juntamente com a análise de diferentes abordagens ao crowd sensing, assim como paradigmas e algoritmos de ML. Em seguida, é explicado como os dados foram capturados e analisados, seguido por uma experiência feita às capacidades do sensor. Além disso, é apresentado como os modelos de ML foram concebidos e otimizados. Finalmente, os resultados dos vários testes de ML são discutidos e o modelo com melhor desempenho é apresentado. A investigação e os resultados práticos abrem perspetivas importantes para a implementação deste tipo de soluções na nossa vida diária.Bringing intelligence to our everyday environments is a growing reality and therefore we should take advantage of the technology available to improve several areas of our daily life. For example, current technology allows the conception of smart scanners, more specifically crowd sensors, to passively detect devices such as smartphones or smartwatches through probe requests emitted by such devices, that, in turn, are part of a communication process that happens every time the devices’ Wi-Fi is enabled. Additionally, crowd sensing - an Ambient Intelligence (AmI) solution - is being studied nowadays in several areas with good results. Therefore, this dissertation aims to research and use crowd sensors to passively collect crowd density data, explore the capabilities of the chosen sensor, analyse and process the data to get better estimations and conceive and develop Machine Learning (ML) models to forecast the density of the sensed areas. Areas in which crowd sensing is inserted - AmI, Smart Cities, Wi-Fi probing - are studied, along with the analysis of different crowd sensing approaches and ML paradigms and algorithms. Then, it’s explained how the data was collected and analysed together with the insights obtained from it, followed by an experiment done on the crowd sensor capabilities. Moreover, it’s presented how the ML models were conceived and tuned. Finally, the results from the ML several tests are discussed and the best performing model is found. The investigation, together with practical results, opens important perspectives for the implementation of these kinds of solutions in our daily lives

Systematic review of smartphone-based passive sensing for health and wellbeing

OBJECTIVE: To review published empirical literature on the use of smartphone-based passive sensing for health and wellbeing. MATERIAL AND METHODS: A systematic review of the English language literature was performed following PRISMA guidelines. Papers indexed in computing, technology, and medical databases were included if they were empirical, focused on health and/or wellbeing, involved the collection of data via smartphones, and described the utilized technology as passive or requiring minimal user interaction. RESULTS: Thirty-five papers were included in the review. Studies were performed around the world, with samples of up to 171 (median n = 15) representing individuals with bipolar disorder, schizophrenia, depression, older adults, and the general population. The majority of studies used the Android operating system and an array of smartphone sensors, most frequently capturing accelerometry, location, audio, and usage data. Captured data were usually sent to a remote server for processing but were shared with participants in only 40% of studies. Reported benefits of passive sensing included accurately detecting changes in status, behavior change through feedback, and increased accountability in participants. Studies reported facing technical, methodological, and privacy challenges. DISCUSSION: Studies in the nascent area of smartphone-based passive sensing for health and wellbeing demonstrate promise and invite continued research and investment. Existing studies suffer from weaknesses in research design, lack of feedback and clinical integration, and inadequate attention to privacy issues. Key recommendations relate to developing passive sensing strategies matching the problem at hand, using personalized interventions, and addressing methodological and privacy challenges. CONCLUSION: As evolving passive sensing technology presents new possibilities for health and wellbeing, additional research must address methodological, clinical integration, and privacy issues. Doing so depends on interdisciplinary collaboration between informatics and clinical experts

Information-Centric Design and Implementation for Underwater Acoustic Networks

Over the past decade, Underwater Acoustic Networks (UANs) have received extensive attention due to their vast benefits in academia and industry alike. However, due to the overall magnitude and harsh characteristics of underwater environments, standard wireless network techniques will fail because current technology and energy restrictions limit underwater devices due to delayed acoustic communications. To help manage these limitations we utilize Information-Centric Networking (ICN). More importantly, we look at ICN\u27s paradigm shift from traditional TCP/IP architecture to improve data handling and enhance network efficiency. By utilizing some of ICN\u27s techniques, such as data naming hierarchy, we can reevaluate each component of the network\u27s protocol stack given current underwater limitations to study the vast solutions and perspectives Information-Centric architectures can provide to UANs. First, we propose a routing strategy used to manage and route large data files in a network prone to high mobility. Therefore, due to UANs limited transmitting capability, we passively store sensed data and adaptively find the best path. Furthermore, we introduce adapted Named Data Networking (NDN) components to improve upon routing robustness and adaptiveness. Beyond naming data, we use tracers to assist in tracking stored data locations without using other excess means such as flooding. By collaborating tracer consistency with routing path awareness our protocol can adaptively manage faulty or high mobility nodes. Through this incorporation of varied NDN techniques, we are able to see notable improvements in routing efficiency. Second, we analyze the effects of Denial of Service (DoS) attacks on upper layer protocols. Since UANs are typically resource restrained, malicious users can advantageously create fake traffic to burden the already constrained network. While ICN techniques only provide basic DoS restriction we must expand our detection and restriction technique to meet the unique demands of UANs. To provide enhanced security against DoS we construct an algorithm to detect and restrict against these types of attacks while adapting to meet acoustic characteristics. To better extend this work we incorporate three node behavior techniques using probabilistic, adaptive, and predictive approaches for detecting malicious traits. Thirdly, to depict and test protocols in UANs, simulators are commonly used due to their accessibility and controlled testing aspects. For this section, we review Aqua-Sim, a discrete event-driven open-source underwater simulator. To enhance the core aspect of this simulator we first rewrite the current architecture and transition Aqua-Sim to the newest core simulator, NS-3. Following this, we clean up redundant features spread out between the various underwater layers. Additionally, we fully integrate the diverse NS-3 API within our simulator. By revamping previous code layout we are able to improve architecture modularity and child class expandability. New features are also introduced including localization and synchronization support, busy terminal problem support, multi-channel support, transmission range uncertainty modules, external noise generators, channel trace-driven support, security module, and an adapted NDN module. Additionally, we provide extended documentation to assist in user development. Simulation testing shows improved memory management and continuous validity in comparison to other underwater simulators and past iterations of Aqua-Sim