Logging Stress and Anxiety Using a Gamified Mobile-based EMA Application, and Emotion Recognition Using a Personalized Machine Learning Approach

According to American Psychological Association (APA) more than 9 in 10 (94 percent) adults believe that stress can contribute to the development of major health problems, such as heart disease, depression, and obesity. Due to the subjective nature of stress, and anxiety, it has been demanding to measure these psychological issues accurately by only relying on objective means. In recent years, researchers have increasingly utilized computer vision techniques and machine learning algorithms to develop scalable and accessible solutions for remote mental health monitoring via web and mobile applications. To further enhance accuracy in the field of digital health and precision diagnostics, there is a need for personalized machine-learning approaches that focus on recognizing mental states based on individual characteristics, rather than relying solely on general-purpose solutions. This thesis focuses on conducting experiments aimed at recognizing and assessing levels of stress and anxiety in participants. In the initial phase of the study, a mobile application with broad applicability (compatible with both Android and iPhone platforms) is introduced (we called it STAND). This application serves the purpose of Ecological Momentary Assessment (EMA). Participants receive daily notifications through this smartphone-based app, which redirects them to a screen consisting of three components. These components include a question that prompts participants to indicate their current levels of stress and anxiety, a rating scale ranging from 1 to 10 for quantifying their response, and the ability to capture a selfie. The responses to the stress and anxiety questions, along with the corresponding selfie photographs, are then analyzed on an individual basis. This analysis focuses on exploring the relationships between self-reported stress and anxiety levels and potential facial expressions indicative of stress and anxiety, eye features such as pupil size variation and eye closure, and specific action units (AUs) observed in the frames over time. In addition to its primary functions, the mobile app also gathers sensor data, including accelerometer and gyroscope readings, on a daily basis. This data holds potential for further analysis related to stress and anxiety. Furthermore, apart from capturing selfie photographs, participants have the option to upload video recordings of themselves while engaging in two neuropsychological games. These recorded videos are then subjected to analysis in order to extract pertinent features that can be utilized for binary classification of stress and anxiety (i.e., stress and anxiety recognition). The participants that will be selected for this phase are students aged between 18 and 38, who have received recent clinical diagnoses indicating specific stress and anxiety levels. In order to enhance user engagement in the intervention, gamified elements - an emerging trend to influence user behavior and lifestyle - has been utilized. Incorporating gamified elements into non-game contexts (e.g., health-related) has gained overwhelming popularity during the last few years which has made the interventions more delightful, engaging, and motivating. In the subsequent phase of this research, we conducted an AI experiment employing a personalized machine learning approach to perform emotion recognition on an established dataset called Emognition. This experiment served as a simulation of the future analysis that will be conducted as part of a more comprehensive study focusing on stress and anxiety recognition. The outcomes of the emotion recognition experiment in this study highlight the effectiveness of personalized machine learning techniques and bear significance for the development of future diagnostic endeavors. For training purposes, we selected three models, namely KNN, Random Forest, and MLP. The preliminary performance accuracy results for the experiment were 93%, 95%, and 87% respectively for these models

Self-Supervised Learning for Audio-Based Emotion Recognition

Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU- MOSEI's acoustic modality. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data. Our model is first pretrained to uncover the randomly-masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via several evaluation metrics against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics. This work shows the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small, and that the effect is most pronounced for emotions which are easier to classify such as happy, sad and anger. This work further demonstrates that self-supervised learning works when applied to embedded feature representations rather than the traditional approach of pre-training on the raw input space.Comment: 8 pages, 9 figures, submitted to IEEE Transactions on Affective Computin

Gamifying Software Testing – A Focus on Strategy & Tools Development

This study aims to introduce new software testing strategies and tools with the aim of creating a more engaging and rewarding environment for software testers. For this purpose, gamification has been selected as a potential solution to raise the performances of testers. Empirical experiments were conducted to validate key factors and metrics influencing the design and development of a gamified software testing system

Supporting users' influence in gamification settings and game live-streams

Playing games has long been important to mankind. One reason for this is the associated autonomy, as players can decide on many aspects on their own and can shape the experience. Game-related sub-fields have appeared in Human-Computer Interaction where this autonomy is questionable: in this thesis, we consider gamification and game live-streams and here, we support the users' influence at runtime. We hypothesize that this should affect the perception of autonomy and should lead to positive effects overall. Our contribution is three-fold: first, we investigate crowd-based, self-sustaining systems in which the user's influence directly impacts the outcome of the system's service. We show that users are willing to expend effort in such systems even without additional motivation, but that gamification is still beneficial here. Second, we introduce "bottom-up" gamification, i.e., the idea of self-tailored gamification. Here, users have full control over the gamification used in a system, i.e., they can set it up as they see fit at the system's runtime. Through user studies, we show that this has positive behavioral effects and thus adds to the ongoing efforts to move away from "one-size-fits-all" solutions. Third, we investigate how to make gaming live-streams more interactive, and how viewers perceive this. We also consider shared game control settings in live-streams, in which viewers have full control, and we contribute options to support viewers' self-administration here.Seit jeher nehmen Spiele im Leben der Menschen eine wichtige Rolle ein. Ein Grund hierfür ist die damit einhergehende Autonomie, mit der Spielende Aspekte des Spielerlebnisses gestalten können. Spiele-bezogene Teilbereiche werden innerhalb der Mensch-Maschine-Interaktion untersucht, bei denen dieser Aspekt jedoch diskutabel ist: In dieser Arbeit betrachten wir Gamification und Spiele Live-Streams und geben Anwendern mehr Einfluss. Wir stellen die Hypothese auf, dass sich dies auf die Autonomie auswirkt und zu positiven Effekten führt. Der Beitrag dieser Dissertation ist dreistufig: Wir untersuchen crowdbasierte, selbsterhaltende Systeme, bei denen die Einflussnahme des Einzelnen sich auf das Systemergebnis auswirkt. Wir zeigen, dass Nutzer aus eigenem Antrieb bereit sind, sich hier einzubringen, der Einfluss von Gamification sich aber förderlich auswirkt. Im zweiten Schritt führen wir "bottom-up" Gamification ein. Hier hat der Nutzer die volle Kontrolle über die Gamification und kann sie nach eigenem Ermessen zur Laufzeit einstellen. An Hand von Nutzerstudien belegen wir daraus resultierende positive Verhaltenseffekte, was die anhaltenden Bemühungen bestärkt, individuelle Gamification-Konzepte anzubieten. Im dritten Schritt untersuchen wir, wie typische Spiele Live-Streams für Zuschauer interaktiver gestaltet werden können. Zudem betrachten wir Fälle, in denen Zuschauer die gemeinsame Kontrolle über ein Spiel ausüben und wie dies technologisch unterstützt werden kann

Exploring the Impacts, Setbacks and Potentials of Gamification in Promoting Brand Value Co-Creation

Business-to-consumer platforms are increasingly employing gamification – which refers to the use of game design elements in non-game context – to motivate their online users’ involvement in brand development. However, little is known so far about the process through which gamification promotes brand value co-creation. This PhD project is set to unravel this process, alongside addressing its major setbacks and potentials via three consecutive studies. First, a systematic literature review study is conducted, leading to the development of an advanced framework labelled Mechanics - Dynamics - Psychological Triggers - Motivational Effects, which outlines the key stages of the designated process. Second, a content analysis study of selected social threads in the online gamified community of the British mobile network operator Giffgaff is pursued. The study investigates the impact of gamification on promoting an underexamined type of brand value co-creation, associated with online users’ contribution to social activities. Correspondingly, a new theoretical model titled Motivational Drivers in Gamified Social Programs is developed, unveiling a range of social values that are demonstrably found driving online users’ engagement in this overlooked type of brand value co-creation in a gamified environment. Third, a sequential mixed-method study is carried out to address gamification’s failure in persuading a large segment of online users – so-called lurkers – to engage in brand value co-creation. The study comprises a series of focus group discussions, followed by a cross-sectional survey with lurkers of the global gamified travel review platform TripAdvisor. An original theoretical framework entitled Lurkers’ Rational in Gamified CoCreative Platforms is thereby generated, demonstrating the reasons of lurkers’ stance. Additionally, a cluster of potential measures designated to practically address their disengagement is constructively developed. This thesis offers a compound of theoretical contributions to the areas of gamification and brand value co-creation, and provides evidence-informed recommendations to practitioners, in addition to unveiling a set of promising future research directions

From libertarian paternalism to liberalism: behavioural science and policy in an age of new technology

Behavioural science has been effectively used by policy makers in various domains, from health to savings. However, interventions that behavioural scientists typically employ to change behaviour have been at the centre of an ethical debate, given that they include elements of paternalism that have implications for people’s freedom of choice. In the present article, we argue that this ethical debate could be resolved in the future through implementation and advancement of new technologies. We propose that several technologies which are currently available and are rapidly evolving (i.e., virtual and augmented reality, social robotics, gamification, self-quantification, and behavioural informatics) have a potential to be integrated with various behavioural interventions in a non-paternalistic way. More specifically, people would decide themselves which behaviours they want to change and select the technologies they want to use for this purpose, and the role of policy makers would be to develop transparent behavioural interventions for these technologies. In that sense, behavioural science would move from libertarian paternalism to liberalism, given that people would freely choose how they want to change, and policy makers would create technological interventions that make this change possible