BigEAR: Inferring the Ambient and Emotional Correlates from Smartphone-based Acoustic Big Data

This paper presents a novel BigEAR big data framework that employs psychological audio processing chain (PAPC) to process smartphone-based acoustic big data collected when the user performs social conversations in naturalistic scenarios. The overarching goal of BigEAR is to identify moods of the wearer from various activities such as laughing, singing, crying, arguing, and sighing. These annotations are based on ground truth relevant for psychologists who intend to monitor/infer the social context of individuals coping with breast cancer. We pursued a case study on couples coping with breast cancer to know how the conversations affect emotional and social well being. In the state-of-the-art methods, psychologists and their team have to hear the audio recordings for making these inferences by subjective evaluations that not only are time-consuming and costly, but also demand manual data coding for thousands of audio files. The BigEAR framework automates the audio analysis. We computed the accuracy of BigEAR with respect to the ground truth obtained from a human rater. Our approach yielded overall average accuracy of 88.76% on real-world data from couples coping with breast cancer.Comment: 6 pages, 10 equations, 1 Table, 5 Figures, IEEE International Workshop on Big Data Analytics for Smart and Connected Health 2016, June 27, 2016, Washington DC, US

The ethics of forgetting in an age of pervasive computing

In this paper, we examine the potential of pervasive computing to create widespread sousveillance, that will complement surveillance, through the development of lifelogs; socio-spatial archives that document every action, every event, every conversation, and every material expression of an individual’s life. Examining lifelog projects and artistic critiques of sousveillance we detail the projected mechanics of life-logging and explore their potential implications. We suggest, given that lifelogs have the potential to convert exterior generated oligopticons to an interior panopticon, that an ethics of forgetting needs to be developed and built into the development of life-logging technologies. Rather than seeing forgetting as a weakness or a fallibility we argue that it is an emancipatory process that will free pervasive computing from burdensome and pernicious disciplinary effects

Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Pervasive sensing has opened up new opportunities for measuring our feelings and understanding our behavior by monitoring our affective states while mobile. This review paper surveys pervasive affect sensing by examining and considering three major elements of affective pervasive systems, namely; “sensing”, “analysis”, and “application”. Sensing investigates the different sensing modalities that are used in existing real-time affective applications, Analysis explores different approaches to emotion recognition and visualization based on different types of collected data, and Application investigates different leading areas of affective applications. For each of the three aspects, the paper includes an extensive survey of the literature and finally outlines some of challenges and future research opportunities of affective sensing in the context of pervasive computing

Monitoring and detection of agitation in dementia: towards real-time and big-data solutions

The changing demographic profile of the population has potentially challenging social, geopolitical, and financial consequences for individuals, families, the wider society, and governments globally. The demographic change will result in a rapidly growing elderly population with healthcare implications which importantly include Alzheimer type conditions (a leading cause of dementia). Dementia requires long term care to manage the negative behavioral symptoms which are primarily exhibited in terms of agitation and aggression as the condition develops. This paper considers the nature of dementia along with the issues and challenges implicit in its management. The Behavioral and Psychological Symptoms of Dementia (BPSD) are introduced with factors (precursors) to the onset of agitation and aggression. Independent living is considered, health monitoring and implementation in context-aware decision-support systems is discussed with consideration of data analytics. Implicit in health monitoring are technical and ethical constraints, we briefly consider these constraints with the ability to generalize to a range of medical conditions. We postulate that health monitoring offers exciting potential opportunities however the challenges lie in the effective realization of independent assisted living while meeting the ethical challenges, achieving this remains an open research question remains.Peer ReviewedPostprint (author's final draft

I hear you eat and speak: automatic recognition of eating condition and food type, use-cases, and impact on ASR performance

We propose a new recognition task in the area of computational paralinguistics: automatic recognition of eating conditions in speech, i. e., whether people are eating while speaking, and what they are eating. To this end, we introduce the audio-visual iHEARu-EAT database featuring 1.6 k utterances of 30 subjects (mean age: 26.1 years, standard deviation: 2.66 years, gender balanced, German speakers), six types of food (Apple, Nectarine, Banana, Haribo Smurfs, Biscuit, and Crisps), and read as well as spontaneous speech, which is made publicly available for research purposes. We start with demonstrating that for automatic speech recognition (ASR), it pays off to know whether speakers are eating or not. We also propose automatic classification both by brute-forcing of low-level acoustic features as well as higher-level features related to intelligibility, obtained from an Automatic Speech Recogniser. Prediction of the eating condition was performed with a Support Vector Machine (SVM) classifier employed in a leave-one-speaker-out evaluation framework. Results show that the binary prediction of eating condition (i. e., eating or not eating) can be easily solved independently of the speaking condition; the obtained average recalls are all above 90%. Low-level acoustic features provide the best performance on spontaneous speech, which reaches up to 62.3% average recall for multi-way classification of the eating condition, i. e., discriminating the six types of food, as well as not eating. The early fusion of features related to intelligibility with the brute-forced acoustic feature set improves the performance on read speech, reaching a 66.4% average recall for the multi-way classification task. Analysing features and classifier errors leads to a suitable ordinal scale for eating conditions, on which automatic regression can be performed with up to 56.2% determination coefficient

The Effects of Displayed Violence and Game Speed in First-Person Shooters on Physiological Arousal and Aggressive Behavior

Many studies have been conducted to examine the effects of displayed violence in digital games on outcomes like aggressive behavior and physiological arousal. However, they often lack a proper manipulation of the relevant factors and control of confounding variables. In this study, the displayed violence and game speed of a recent first-person shooter game were varied systematically using the technique of modding, so that effects could be explained properly by the respective manipulations. Aggressive behavior was measured with the standardized version of the Competitive Reaction Time Task or CRTT (Ferguson et al., 2008}. Physiological arousal was operationalized with four measurements: galvanic skin response (GSR), heart rate (HR), body movement, force on mouse and keyboard. A total of N = 87 participants played in one of four game conditions (low- vs. high-violence, normal- vs. high speed) while physiological measurements were taken with finger clips, force sensors on input devices (mouse and keyboard), and a Nintendo Wii balance board on the chair they sat on. After play, their aggressive behavior was measured with the CRTT. The results of the study do not support the hypothesis that playing digital games increases aggressive behavior. There were no significant differences in GSR and HR, but with a higher game speed, participants showed less overall body movement, most likely to meet the game’s higher demands on cognitive and motor capacities. Also, higher game speed and displayed violence caused an increase in applied force on mouse and keyboard. Previous experience with digital games did not moderate any of these findings. Moreover, it provides further evidence that the CRTT should only be used in a standardized way as a measurement for aggression, if at all. Using all 7 different published (though not validated) ways to calculate levels of aggression from the raw data, “evidence” was found that playing a violent digital game increases, decreases, or does not change aggression at all. Thus, the present study does extend previous research. Firstly, it shows the methodological advantages of modding in digital game research to accomplish the principles of psychological (laboratory) experiments by manipulating relevant variables and controlling all others. It also demonstrates the test-theoretical problems of the highly diverse use of the CRTT. It provides evidence that for a meaningful interpretation of effects of displayed violence in digital games, there are other game characteristics that should be controlled for since they might have an effect on relevant outcome variables. Further research needs to identify more of those game features, and it should also improve the understanding of the different measures for physiological arousal and their interrelatedness

Psychological research in the digital age

The smartphone has become an important personal companion in our daily lives. Each time we use the device, we generate data that provides information about ourselves. This data, in turn, is valuable to science because it objectively reflects our everyday behavior and experiences. In this way, smartphones enable research that is closer to everyday life than traditional laboratory experiments and questionnaire-based methods. While data collected with smartphones are increasingly being used in the field of personality psychology, new digital technologies can also be leveraged to collect and analyze large-scale unobtrusively sensed data in other areas of psychological research. This dissertation, therefore, explores the insights that smartphone sensing reveals for psychological research using two examples, situation and affect research, making a twofold research contribution. First, in two empirical studies, different data types of smartphone-sensed data, such as GPS or phone data, were combined with experience-sampled self-report, and classical questionnaire data to gain valuable insights into individual behavior, thinking, and feeling in everyday life. Second, predictive modeling techniques were applied to analyze the large, high-dimensional data sets collected by smartphones. To gain a deeper understanding of the smartphone data, interpretable variables were extracted from the raw sensing data, and the predictive performance of various machine learning algorithms was compared. In summary, the empirical findings suggest that smartphone data can effectively capture certain situational and behavioral indicators of psychological phenomena in everyday life. However, in certain research areas such as affect research, smartphone data should only complement, but not completely replace, traditional questionnaire-based data as well as other data sources such as neurophysiological indicators. The dissertation also concludes that the use of smartphone sensor data introduces new difficulties and challenges for psychological research and that traditional methods and perspectives are reaching their limits. The complexity of data collection, processing, and analysis requires established guidelines for study design, interdisciplinary collaboration, and theory-driven research that integrates explanatory and predictive approaches. Accordingly, further research is needed on how machine learning models and other big data methods in psychology can be reconciled with traditional theoretical approaches. Only in this way can we move closer to the ultimate goal of psychology to better understand, explain, and predict human behavior and experiences and their interplay with everyday situations