Wearables, smartphones, and artificial intelligence for digital phenotyping and health

Ubiquitous progress in wearable sensing and mobile computing technologies, alongside growing diversity in sensor modalities, has created new pathways for the collection of health and well-being data outside of laboratory settings, in a longitudinal fashion. Wearable and mobile devices have the potential to provide low-cost, objective measures of physical activity, clinically relevant data for patient assessment, and scalable behavior monitoring in large populations. These data can be used in both interventional and observational studies to derive insights regarding the links between behavior, health. and disease, as well as to advance the personalization and effectiveness of commercial wellness applications. Today, over 400,000 participants have had their behavior tracked prospectively using accelerometers for epidemiological studies across the globe. Traditionally, epidemiologists and clinicians have relied upon self-report measures of physical activity and sleep which, while valuable in the absence of alternatives, are subject to bias and often provide partial, incomplete information Physical behavior data extracted from wearable devices are being used to derive sensor-assessed, objective measures of physical behaviors, overcoming the limitations of self-report with the aim of relating these to clinical endpoints and eventually applying the findings to preventive and predictive medicine. Moreover, the application of artificial intelligence (AI), sensor fusion, and signal processing to wearable sensor data has led to improved human activity recognition and behavioral phenotyping. Here, we review the state of the art in wearable and mobile sensing technology in epidemiology and clinical medicine and discuss how AI is changing the field

Feeling the Temperature of the Room: Unobtrusive Thermal Display of Engagement during Group Communication

Thermal signals have been explored in HCI for emotion-elicitation and enhancing two-person communication, showing that temperature invokes social and emotional signals in individuals. Yet, extending these findings to group communication is missing. We investigated how thermal signals can be used to communicate group affective states in a hybrid meeting scenario to help people feel connected over a distance. We conducted a lab study (N=20 participants) and explored wrist-worn thermal feedback to communicate audience emotions. Our results show that thermal feedback is an effective method of conveying audience engagement without increasing workload and can help a presenter feel more in tune with the audience. We outline design implications for real-world wearable social thermal feedback systems for both virtual and in-person communication that support group affect communication and social connectedness. Thermal feedback has the potential to connect people across distances and facilitate more effective and dynamic communication in multiple contexts.Comment: In IMWUT 202

Analysis and design of individual information systems to support health behavior change

As a wide-ranging socio-technical transformation, the digitalization has significantly influenced the world, bringing opportunities and challenges to our lives. Despite numerous benefits like the possibility to stay connected with people around the world, the increasing dispersion and use of digital technologies and media (DTM) pose risks to individuals’ well-being and health. Rising demands emerging from the digital world have been linked to digital stress, that is, stress directly or indirectly resulting from DTM (Ayyagari et al. 2011; Ragu-Nathan et al. 2008; Tarafdar et al. 2019; Weil and Rosen 1997), potentially intensifying individuals’ overall exposure to stress. Individuals experiencing this adverse consequence of digitalization are at elevated risk of developing severe mental health impairments (Alhassan et al. 2018; Haidt and Allen 2020; Scott et al. 2017), which is why various scholars emphasize that research should place a stronger focus on analyzing and shaping the role of the individual in a digital world, pursuing instrumental as well as humanistic objectives (Ameen et al. 2021; Baskerville 2011b). Information Systems (IS) research has long placed emphasis on the use of information and communication technology (ICT) in organizations, viewing an information system as the socio-technical system that emerges from individuals’ interaction with DTM in organizations. However, socio-technical information systems, as the essence of the IS discipline (Lee 2004; Sarker et al. 2019), are also present in different social contexts from private life. Acknowledging the increasing private use of DTM, such as smartphones and social networks, IS scholars have recently intensified their efforts to understand the human factor of IS (Avison and Fitzgerald 1991; Turel et al. 2021). A framework recently proposed by Matt et al. (2019) suggests three research angles: analyzing individuals’ behavior associated with their DTM use, analyzing what consequences arise from their DTM use behavior, and designing new technologies that promote positive or mitigate negative effects of individuals’ DTM use. Various recent studies suggest that individuals’ behavior seems to be an important lever influencing the outcomes of their DTM use (Salo et al. 2017; Salo et al. 2020; Weinstein et al. 2016). Therefore, this dissertation aims to contribute to IS research targeting the facilitation of a healthy DTM use behavior. It explores the use behavior, consequences, and design of DTM for individuals' use with the objective to deliver humanistic value by increasing individuals' health through supporting a behavior change related to their DTM use. The dissertation combines behavioral science and design science perspectives and applies pluralistic methodological approaches from qualitative (e.g., interviews, prototyping) and quantitative research (e.g., survey research, field studies), including mixed-methods approaches mixing both. Following the framework from Matt et al. (2019), the dissertation takes three perspectives therein: analyzing individuals’ behavior, analyzing individuals’ responses to consequences of DTM use, and designing information systems assisting DTM users. First, the dissertation presents new descriptive knowledge on individuals’ behavior related to their use of DTM. Specifically, it investigates how individuals behave when interacting with DTM, why they behave the way they do, and how their behavior can be influenced. Today, a variety of digital workplace technologies offer employees different ways of pursuing their goals or performing their tasks (Köffer 2015). As a result, individuals exhibit different behaviors when interacting with these technologies. The dissertation analyzes what interactional roles DTM users can take at the digital workplace and what may influence their behavior. It uses a mixed-methods approach and combines a quantitative study building on trace data from a popular digital workplace suite and qualitative interviews with users of this digital workplace suite. The empirical analysis yields eight user roles that advance the understanding of users’ behavior at the digital workplace and first insights into what factors may influence this behavior. A second study adds another perspective and investigates how habitual behavior can be changed by means of DTM design elements. Real-time feedback has been discussed as a promising way to do so (Schibuola et al. 2016; Weinmann et al. 2016). In a field experiment, employees working at the digital workplace are provided with an external display that presents real-time feedback on their office’s indoor environmental quality. The experiment examines if and to what extent the feedback influences their ventilation behavior to understand the effect of feedback as a means of influencing individuals’ behavior. The results suggest that real-time feedback can effectively alter individuals’ behavior, yet the feedback’s effectiveness reduces over time, possibly as a result of habituation to the feedback. Second, the dissertation presents new descriptive and prescriptive knowledge on individuals’ ways to mitigate adverse consequences arising from the digitalization of individuals. A frequently discussed consequence that digitalization has on individuals is digital stress. Although research efforts strive to determine what measures individuals can take to effectively cope with digital stress (Salo et al. 2017; Salo et al. 2020; Weinert 2018), further understanding of individuals’ coping behavior is needed (Weinert 2018). A group at high risk of suffering from the adverse effects of digital stress is adolescents because they grow up using DTM daily and are still developing their identity, acquiring mental strength, and adopting essential social skills. To facilitate a healthy DTM use, the dissertation explores what strategies adolescents use to cope with the demands of their DTM use. Combining a qualitative and a quantitative study, it presents 30 coping responses used by adolescents, develops five factors underlying adolescents’ activation of coping responses, and identifies gender- and age-related differences in their coping behavior. Third, the dissertation presents new prescriptive knowledge on the design of individual information systems supporting individuals in understanding and mitigating their perceived stress. Facilitated by the sensing capabilities of modern mobile devices, it explores the design and development of mobile systems that assess stress and support individuals in coping with stress by initiating a change of stress-related behavior. Since there is currently limited understanding of how to develop such systems, this dissertation explores various facets of their design and development. As a first step, it presents the development of a prototype aiming for life-integrated stress assessment, that is, the mobile sensor-based assessment of an individual’s stress without interfering with their daily routines. Data collected with the prototype yields a stress model relating sensor data to individuals’ perception of stress. To deliver a more generalized perspective on mobile stress assessment, the dissertation further presents a literature- and experience-based design theory comprising a design blueprint, design requirements, design principles, design features, and a discussion of potentially required trade-offs. Mobile stress assessment may be used for the development of mobile coping assistants. Aiming to assist individuals in effectively coping with stress and preventing future stress, a mobile coping assistant should recommend adequate coping strategies to the stressed individual in real-time or execute targeted actions within a defined scope of action automatically. While the implementation of a mobile coping assistant is yet up to future research, the dissertation presents an abstract design and algorithm for selecting appropriate coping strategies. To sum up, this dissertation contributes new knowledge on the digitalization of individuals to the IS knowledge bases, expanding both descriptive and prescriptive knowledge. Through the combination of diverse methodological approaches, it delivers knowledge on individuals’ behavior when using DTM, on the mitigation of consequences that may arise from individuals’ use of DTM, and on the design of individual information systems with the goal of facilitating a behavior change, specifically, regarding individuals’ coping with stress. Overall, the research contained in this dissertation may promote the development of digital assistants that support individuals’ in adopting a healthy DTM use behavior and thereby contribute to shaping a socio-technical environment that creates more benefit than harm for all individuals

Psychological research in the digital age

The smartphone has become an important personal companion in our daily lives. Each time we use the device, we generate data that provides information about ourselves. This data, in turn, is valuable to science because it objectively reflects our everyday behavior and experiences. In this way, smartphones enable research that is closer to everyday life than traditional laboratory experiments and questionnaire-based methods. While data collected with smartphones are increasingly being used in the field of personality psychology, new digital technologies can also be leveraged to collect and analyze large-scale unobtrusively sensed data in other areas of psychological research. This dissertation, therefore, explores the insights that smartphone sensing reveals for psychological research using two examples, situation and affect research, making a twofold research contribution. First, in two empirical studies, different data types of smartphone-sensed data, such as GPS or phone data, were combined with experience-sampled self-report, and classical questionnaire data to gain valuable insights into individual behavior, thinking, and feeling in everyday life. Second, predictive modeling techniques were applied to analyze the large, high-dimensional data sets collected by smartphones. To gain a deeper understanding of the smartphone data, interpretable variables were extracted from the raw sensing data, and the predictive performance of various machine learning algorithms was compared. In summary, the empirical findings suggest that smartphone data can effectively capture certain situational and behavioral indicators of psychological phenomena in everyday life. However, in certain research areas such as affect research, smartphone data should only complement, but not completely replace, traditional questionnaire-based data as well as other data sources such as neurophysiological indicators. The dissertation also concludes that the use of smartphone sensor data introduces new difficulties and challenges for psychological research and that traditional methods and perspectives are reaching their limits. The complexity of data collection, processing, and analysis requires established guidelines for study design, interdisciplinary collaboration, and theory-driven research that integrates explanatory and predictive approaches. Accordingly, further research is needed on how machine learning models and other big data methods in psychology can be reconciled with traditional theoretical approaches. Only in this way can we move closer to the ultimate goal of psychology to better understand, explain, and predict human behavior and experiences and their interplay with everyday situations

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject

Approaches, applications, and challenges in physiological emotion recognition — a tutorial overview

An automatic emotion recognition system can serve as a fundamental framework for various applications in daily life from monitoring emotional well-being to improving the quality of life through better emotion regulation. Understanding the process of emotion manifestation becomes crucial for building emotion recognition systems. An emotional experience results in changes not only in interpersonal behavior but also in physiological responses. Physiological signals are one of the most reliable means for recognizing emotions since individuals cannot consciously manipulate them for a long duration. These signals can be captured by medical-grade wearable devices, as well as commercial smart watches and smart bands. With the shift in research direction from laboratory to unrestricted daily life, commercial devices have been employed ubiquitously. However, this shift has introduced several challenges, such as low data quality, dependency on subjective self-reports, unlimited movement-related changes, and artifacts in physiological signals. This tutorial provides an overview of practical aspects of emotion recognition, such as experiment design, properties of different physiological modalities, existing datasets, suitable machine learning algorithms for physiological data, and several applications. It aims to provide the necessary psychological and physiological backgrounds through various emotion theories and the physiological manifestation of emotions, thereby laying a foundation for emotion recognition. Finally, the tutorial discusses open research directions and possible solutions

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject