Annotating Affect in the Field: A Case Study on the Usability of a Minimalist Smartwatch User Interface for Affect Annotation

Successful empathetic interaction requires an accurate understanding of the interaction partner\u27s affect dynamics. Self-reported annotations provide a way to better understand affect and empathy in real-life; however, the necessary user interactions for collecting such data must be designed to be as unobtrusive as possible. To address this challenge, we explore the potential of a smartwatch annotation application for affect that aims to minimize user interaction effort while maximizing usability. In a field study conducted as part of a student career fair (N=9), we evaluated the feasibility and usability of our app. Participants reported high usability scores and our data collection successfully captured self-reported affect labels at a high temporal resolution. Our work contributes to the challenge of providing minimal obtrusive applications for the collection of self-reported labels of affective states

From the Inside Out: A Literature Review on Possibilities of Mobile Emotion Measurement and Recognition

Information systems are becoming increasingly intelligent and emotion artificial intelligence is an important component for the future. Therefore, the measurement and recognition of emotions is necessary and crucial. This paper presents a state of the art in the research field of mobile emotion measurement and recognition. The aim of this structured literature analysis using the PRISMA statement is to collect and classify the relevant literature and to provide an overview of the current status of mobile emotion recording and its future trends. A total of 59 articles were identified in the relevant literature databases, which can be divided into four main categories of emotion measurement. There was an increase of publications over the years in all four categories, but with a particularly strong increase in the areas of optical and vital-data-based recording. Over time, both the speed as well as the accuracy of the measurement has improved considerably in all four categories

Digital Transformation in Retail: Can Customer Value Services enhance the Experience?

The brick-and-mortar retail is struggling with the digital transformation and the shift to e-commerce. Likewise, technological developments in retail service delivery raise new questions concerning the nature of relationships between retailers and customers. To secure a strong customer relationship and support satisfaction, retailers have to transform their real-world advantages into digital goods and offer new value services to the customer. With an iterative process based on design science research (DSR), we want to explore the impact of different combinations of technology-mediated value services in customer-retail-relationships. Therefore, we want to evaluate, compare and classify a combination of emotional and context-aware approaches as well as services which link online and offline advantages. This research is aimed to identify services to support and lead the brick-and-mortar retailers through the digital transformation

Feel the Moosic: Emotion-based Music Selection and Recommendation

Digital transformation has changed all aspects of life, including the music market and listening habits. The spread of mobile devices and music streaming services has enabled the possibility to access a huge selection of music regardless of time or place. However, this access leads to the customer\u27s problem of choosing the right music for a certain situation or mood. The user is often overwhelmed while choosing music. Context information, especially the emotional state of the user, can help within this process. The possibilities of an emotional music selection are currently limited. The providers rely on predefined playlists for different situations or moods. However, the problem with these lists is, that they do not adapt to new user conditions. A simple, intuitive and automatic emotion-based music selection has so far been poorly investigated in IS practice and research. This paper describes the IS music research project Moosic , which investigates and iteratively implements an intuitive emotion-based music recommendation application. In addition, an initial evaluation of the prototype will be discussed and an outlook on further development will be given

Transformation im stationären Einzelhandel: Emotionen und digitale Kundenbeziehungen

Das signifikante und kontinuierliche Wachstum des E-Commerce sowie die digitale Transformation selbst sind verantwortlich für einen notwendigen Transformationsprozess des Einzelhandels (Dennis, Jayawardhena, & Papamatthaiou, 2010; Doherty & Ellis-Chadwick, 2010; Hagberg, Sundstrom, & Egels-Zandén, 2016; Sands, Ferraro, & Luxton, 2010). Das aktuelle Phänomen der Digitalisierung im Einzelhandel ist bereits Gegenstand verschiedener Forschungsprojekte (Hagberg et al., 2016; Keeling, Keeling, & McGoldrick, 2013). Gleichzeitig ermöglichen mobile Geräte - insbesondere Smartphones und seit kurzem auch Smartwatches - diese digitale Transformation und führen zu einem veränderten Kundenverhalten (Blázquez, 2014). Mobile Endgeräte sind in der Lage dem Kunden spezifischere und situationsbezogene Informationen zu liefern (Rohm & Sultan, 2006). Mit diesen mobilen und intelligenten Technologien können Kunden jederzeit und überall auf das Internet zugreifen. So können produktspezifische Informationen wie Preise, Produktbilder und Kundenbewertungen schneller und einfacher erhalten werden (Spaid & Flint, 2014). Weiterhin ändert sich nicht nur das Verhalten der Kunden, sondern auch ihre Erwartungen. Durch die Integration mobiler Geräte in den Alltag erwarten Kunden eine bessere Erreichbarkeit der Händler in Online- und Offline-Kanälen (Fulgoni, 2014). Diese neue digitale Handelssituation bietet viele Herausforderungen, aber auch zahlreiche Chancen für den stationären Handel (Härtfelder & Winkelmann, 2016). Die Branche hat einige dieser Probleme bereits erkannt, aber bisher nur in geringem Maße reagiert. [Aus der Einleitung.

Ambulatory assessment for physical activity research. State of the science, best practices and future directions

Technological and digital progress benefits physical activity (PA) research. Here we compiled expert knowledge on how Ambulatory Assessment (AA) is utilized to advance PA research, i.e., we present results of the 2nd International CAPA Workshop 2019 "Physical Activity Assessment - State of the Science, Best Practices, Future Directions" where invited researchers with experience in PA assessment, evaluation, technology and application participated. First, we provide readers with the state of the AA science, then we give best practice recommendations on how to measure PA via AA and shed light on methodological frontiers, and we furthermore discuss future directions. AA encompasses a class of methods that allows the study of PA and its behavioral, biological and physiological correlates as they unfold in everyday life. AA includes monitoring of movement (e.g., via accelerometry), physiological function (e.g., via mobile electrocardiogram), contextual information (e.g., via geolocation-tracking), and ecological momentary assessment (EMA; e.g., electronic diaries) to capture self-reported information. The strengths of AA are data assessment that near real-time, which minimizes retrospective biases in real-world settings, consequentially enabling ecological valid findings. Importantly, AA enables multiple assessments across time within subjects resulting in intensive longitudinal data (ILD), which allows unraveling within-person determinants of PA in everyday life. In this paper, we show how AA methods such as triggered e-diaries and geolocation-tracking can be used to measure PA and its correlates, and furthermore how these findings may translate into real-life interventions. In sum, AA provides numerous possibilities for PA research, especially the opportunity to tackle within-subject antecedents, concomitants, and consequences of PA as they unfold in everyday life. In-depth insights on determinants of PA could help us design and deliver impactful interventions in real-world contexts, thus enabling us to solve critical health issues in the 21st century such as insufficient PA and high levels of sedentary behavior. (DIPF/Orig.

PhysioVR: a novel mobile virtual reality framework for physiological computing

Virtual Reality (VR) is morphing into a ubiquitous technology by leveraging of smartphones and screenless cases in order to provide highly immersive experiences at a low price point. The result of this shift in paradigm is now known as mobile VR (mVR). Although mVR offers numerous advantages over conventional immersive VR methods, one of the biggest limitations is related with the interaction pathways available for the mVR experiences. Using physiological computing principles, we created the PhysioVR framework, an Open-Source software tool developed to facilitate the integration of physiological signals measured through wearable devices in mVR applications. PhysioVR includes heart rate (HR) signals from Android wearables, electroencephalography (EEG) signals from a low cost brain computer interface and electromyography (EMG) signals from a wireless armband. The physiological sensors are connected with a smartphone via Bluetooth and the PhysioVR facilitates the streaming of the data using UDP communication protocol, thus allowing a multicast transmission for a third party application such as the Unity3D game engine. Furthermore, the framework provides a bidirectional communication with the VR content allowing an external event triggering using a real-time control as well as data recording options. We developed a demo game project called EmoCat Rescue which encourage players to modulate HR levels in order to successfully complete the in-game mission. EmoCat Rescue is included in the PhysioVR project which can be freely downloaded. This framework simplifies the acquisition, streaming and recording of multiple physiological signals and parameters from wearable consumer devices providing a single and efficient interface to create novel physiologically-responsive mVR applications.info:eu-repo/semantics/publishedVersio

DESIGNING AND IMPLEMENTING ACCESSIBLE WEARABLE INTERACTIONS FOR PEOPLE WITH MOTOR IMPAIRMENTS

Emerging wearable technologies like fitness bands, smartwatches, and head-mounted displays (HMDs) are entering the mainstream market. Unlike smartphones and tablets, these wearables, worn on the body or clothing, are always available and have the potential to provide quick access to information [7]. For instance, HMDs can provide relatively hands-free interaction compared to smartphones, and smartwatches and activity trackers can collect continuous health and fitness-related information of their wearer. However, there are over 20 million people in the U.S. with upper body motor impairments [133], who may not be able to gain from the potential benefits of these wearables. For example, the small interaction spaces of smartwatches may present accessibility challenges. Yet, few studies have explored the potential impacts or evaluated the accessibility of these wearables or investigated ways to design accessible wearable interactions for people with motor impairments. To inform the design of future wearable technologies, my dissertation investigates three threads of research: (1) assessing the accessibility of wearable technologies like HMDs, smartwatches and fitness trackers; (2) understanding the potential impacts of sharing automatically tracked fitness-related information for people with mobility impairments; and (3) implementing and evaluating accessible interactions for HMDs and smartwatches. As part of my first research thread, I conducted two formative studies investigating the accessibility of HMDs and fitness trackers and found that people with motor impairments experienced accessibility challenges like problematic form factors, irrelevant data tracking and difficulty with existing input. For my second research thread, I investigated the potential impacts of sharing automatically tracked data from fitness trackers with peers with similar impairments and therapists and presented design opportunities to build tools to support sharing. Towards my third research thread, I addressed the earlier issues identified with HMD accessibility by building custom wearable touchpads to control a commercial HMD. Next, I explored the touchscreen and non-touchscreen areas (bezel, wristband and user’s body) of smartwatches for accessible interaction. And, lastly, I built and compared bezel input with touchscreen input for accessible smartwatch interaction. The techniques implemented and evaluated in this dissertation will enable more equitable and independent use of wearable technologies for people with motor impairments

From Personalized Medicine to Population Health: A Survey of mHealth Sensing Techniques

Mobile Sensing Apps have been widely used as a practical approach to collect behavioral and health-related information from individuals and provide timely intervention to promote health and well-beings, such as mental health and chronic cares. As the objectives of mobile sensing could be either \emph{(a) personalized medicine for individuals} or \emph{(b) public health for populations}, in this work we review the design of these mobile sensing apps, and propose to categorize the design of these apps/systems in two paradigms -- \emph{(i) Personal Sensing} and \emph{(ii) Crowd Sensing} paradigms. While both sensing paradigms might incorporate with common ubiquitous sensing technologies, such as wearable sensors, mobility monitoring, mobile data offloading, and/or cloud-based data analytics to collect and process sensing data from individuals, we present a novel taxonomy system with two major components that can specify and classify apps/systems from aspects of the life-cycle of mHealth Sensing: \emph{(1) Sensing Task Creation \& Participation}, \emph{(2) Health Surveillance \& Data Collection}, and \emph{(3) Data Analysis \& Knowledge Discovery}. With respect to different goals of the two paradigms, this work systematically reviews this field, and summarizes the design of typical apps/systems in the view of the configurations and interactions between these two components. In addition to summarization, the proposed taxonomy system also helps figure out the potential directions of mobile sensing for health from both personalized medicines and population health perspectives.Comment: Submitted to a journal for revie

Digital phenotyping through multimodal, unobtrusive sensing

The growing adoption of multimodal wearable and mobile devices, such as smartphones and wrist-worn watches has generated an increase in the collection of physiological and behavioural data at scale. This digital phenotyping data enables researchers to make inferences regarding users’ physical and mental health at scale, for the first time. However, translating this data into actionable insights requires computational approaches that turn unlabelled, multimodal time-series sensor data into validated measures that can be interpreted at scale. This thesis describes the derivation of novel computational methods that leverage digital phenotyping data from wearable devices in large-scale populations to infer physical behaviours. These methods combine insights from signal processing, data mining and machine learning alongside domain knowledge in physical activity and sleep epidemiology. First, the inference of sleeping windows in free-living conditions through a heart rate sensing approach is explored. This algorithm is particularly valuable in the absence of ground truth or sleep diaries given its simplicity, adaptability and capacity for personalization. I then explore multistage sleep classification through combined movement and cardiac wearable sensing and machine learning. Further, I demonstrate that postural changes detected through wrist accelerometers can inform habitual behaviours and are valuable complements to traditional, intensity-based physical activity metrics. I then leverage the concomitant responses of heart rate to physical activity that can be captured through multimodal wearable sensors through a self-supervised training task. The resulting embeddings from this task are shown to be useful for the downstream classification of demographic factors, BMI, energy expenditure and cardiorespiratory fitness. Finally, I describe a deep learning model for the adaptive inference of cardiorespiratory fitness (VO2max) using wearable data in free living conditions. I demonstrate the robustness of the model in a large UK population and show the models’ adaptability by evaluating its performance in a subset of the population with repeated measures ~6 years after the original recordings. Together, this work increases the potential of multimodal wearable and mobile sensors for physical activity and behavioural inferences in population studies. In particular, this thesis showcases the potential of using wearable devices to make valuable physical activity, sleep and fitness inferences in large cohort studies. Given the nature of the data collected and the fact that most of this data is currently generated by commercial providers and not research institutes, laying the foundations for responsible data governance and ethical use of these technologies will be critical to building trust and enabling the development of the field of digital phenotyping.I was funded by GlaxoSmithKline and the Engineering and Physical Sciences Research Council. I was also supported by the Alan Turing Institute through their Enrichment Scheme