SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.Comment: 14 pages, 11 figure

Privacy-sensitive recognition of group conversational context with sociometers

Recognizing the conversational context in which group interactions unfold has applications in machines that support collaborative work and perform automatic social inference using contextual knowledge. This paper addresses the task of discriminating one conversational context from another, specifically brainstorming from decision-making interactions, using easily computable nonverbal behavioral cues. Privacy-sensitive mobile sociometers are used to record the interaction data. We hypothesize that the difference in the conversational dynamics between brainstorming and decision-making discussions is significant and measurable using speaking activity-based nonverbal cues. We characterize the communication patterns of the entire group by the aggregation (both temporal and person-wise) of their nonverbal behavior. The results on our interaction data set show that the floor-occupation patterns in a brainstorming interaction are different from a decision-making interaction, and our method can obtain a classification accuracy as high as 87.5%

Learning when out and about

[Introduction] Mobile digital devices such as tablet computers and smartphones (mobile phones that can run apps and access the Internet), enable learners to access learning materials while out and about during their daily lives. This enables networked digital learning to move from beyond the classroom and to become part of everyday routines. Smartphones and tablets are increasingly likely to be the first devices a wide range of people will turn to for technology enhanced learning, incorporated into their everyday activities and carried with them. Learning becomes ubiquitous, making use of ‘domesticated’ technologies that serve a variety of purposes in daily life. In this chapter, we consider how smartphones can trigger location specific learning resources to support adults learning languages when out and about, and consider two projects, MASELTOV and SALSA. There has been increased interest in exploring the potential of ‘smart cities’ – urban environments with high-tech infrastructures – to support learning. We look at an example, the SALSA project, where a combination of smartphones, location-based technologies and learning resources has been used to prompt language learners, and to understand whether this motivates them to extend their learning

Sensing, Understanding, and Shaping Social Behavior

The ability to understand social systems through the aid of computational tools is central to the emerging field of computational social systems. Such understanding can answer epistemological questions on human behavior in a data-driven manner, and provide prescriptive guidelines for persuading humans to undertake certain actions in real-world social scenarios. The growing number of works in this subfield has the potential to impact multiple walks of human life including health, wellness, productivity, mobility, transportation, education, shopping, and sustenance. The contribution of this paper is twofold. First, we provide a functional survey of recent advances in sensing, understanding, and shaping human behavior, focusing on real-world behavior of users as measured using passive sensors. Second, we present a case study on how trust, which is an important building block of computational social systems, can be quantified, sensed, and applied to shape human behavior. Our findings suggest that:1) trust can be operationalized and predicted via computational methods (passive sensing and network analysis) and 2) trust has a significant impact on social persuasion; in fact, it was found to be significantly more effective than the closeness of ties in determining the amount of behavior change.U.S. Army Research Laboratory (Cooperative Agreement W911NF-09-2-0053

Exploring the Affective Loop

Research in psychology and neurology shows that both body and mind are involved when experiencing emotions (Damasio 1994, Davidson et al. 2003). People are also very physical when they try to communicate their emotions. Somewhere in between beings consciously and unconsciously aware of it ourselves, we produce both verbal and physical signs to make other people understand how we feel. Simultaneously, this production of signs involves us in a stronger personal experience of the emotions we express. Emotions are also communicated in the digital world, but there is little focus on users' personal as well as physical experience of emotions in the available digital media. In order to explore whether and how we can expand existing media, we have designed, implemented and evaluated /eMoto/, a mobile service for sending affective messages to others. With eMoto, we explicitly aim to address both cognitive and physical experiences of human emotions. Through combining affective gestures for input with affective expressions that make use of colors, shapes and animations for the background of messages, the interaction "pulls" the user into an /affective loop/. In this thesis we define what we mean by affective loop and present a user-centered design approach expressed through four design principles inspired by previous work within Human Computer Interaction (HCI) but adjusted to our purposes; /embodiment/ (Dourish 2001) as a means to address how people communicate emotions in real life, /flow/ (Csikszentmihalyi 1990) to reach a state of involvement that goes further than the current context, /ambiguity/ of the designed expressions (Gaver et al. 2003) to allow for open-ended interpretation by the end-users instead of simplistic, one-emotion one-expression pairs and /natural but designed expressions/ to address people's natural couplings between cognitively and physically experienced emotions. We also present results from an end-user study of eMoto that indicates that subjects got both physically and emotionally involved in the interaction and that the designed "openness" and ambiguity of the expressions, was appreciated and understood by our subjects. Through the user study, we identified four potential design problems that have to be tackled in order to achieve an affective loop effect; the extent to which users' /feel in control/ of the interaction, /harmony and coherence/ between cognitive and physical expressions/,/ /timing/ of expressions and feedback in a communicational setting, and effects of users' /personality/ on their emotional expressions and experiences of the interaction

Intelligent environments: a manifesto

We explain basic features of an emerging area called Intelligent Environments. We give a short overview on how it has developed, what is the current state of the art and what are the challenges laying ahead. The aim of the article is to make aware the Computer Science community of this new development, the differences with previous dominant paradigms and the opportunities that this area offers to the scientific community and society

Use of nonintrusive sensor-based information and communication technology for real-world evidence for clinical trials in dementia

Cognitive function is an important end point of treatments in dementia clinical trials. Measuring cognitive function by standardized tests, however, is biased toward highly constrained environments (such as hospitals) in selected samples. Patient-powered real-world evidence using information and communication technology devices, including environmental and wearable sensors, may help to overcome these limitations. This position paper describes current and novel information and communication technology devices and algorithms to monitor behavior and function in people with prodromal and manifest stages of dementia continuously, and discusses clinical, technological, ethical, regulatory, and user-centered requirements for collecting real-world evidence in future randomized controlled trials. Challenges of data safety, quality, and privacy and regulatory requirements need to be addressed by future smart sensor technologies. When these requirements are satisfied, these technologies will provide access to truly user relevant outcomes and broader cohorts of participants than currently sampled in clinical trials

How to Support Domestic Violence Survivors with Conversational Agents: Meta Requirements and Design Principles

Domestic violence is a prevalent and complicated issue that can have detrimental effects on the survivors, their families, and communities. Survivors are often reluctant to divulge their experiences to others in person for social, emotional, privacy, or cultural reasons. Consequently, many are not actively seeking support that meets their needs. Conversational agents, a form of technology support, hold great promise for facilitating counseling and support by promoting self-disclosure and enhancing user engagement. To address the knowledge gaps in design principles for conversational agents for DV survivors, we conducted in-depth interviews with 11 professionals working with domestic violence survivors. After analyzing the interview transcripts and related literature, we identified several meta-requirements and categorized them into four categories —conversation, language, support, and trust. We further grouped these meta-requirements into several design principles. Our work lays the foundation for design science research in designing and developing conversational agents to support domestic violence survivors

Value Co-Creation in Smart Services: A Functional Affordances Perspective on Smart Personal Assistants

In the realm of smart services, smart personal assistants (SPAs) have become a popular medium for value co-creation between service providers and users. The market success of SPAs is largely based on their innovative material properties, such as natural language user interfaces, machine learning-powered request handling and service provision, and anthropomorphism. In different combinations, these properties offer users entirely new ways to intuitively and interactively achieve their goals and thus co-create value with service providers. But how does the nature of the SPA shape value co-creation processes? In this paper, we look through a functional affordances lens to theorize about the effects of different types of SPAs (i.e., with different combinations of material properties) on users’ value co-creation processes. Specifically, we collected SPAs from research and practice by reviewing scientific literature and web resources, developed a taxonomy of SPAs’ material properties, and performed a cluster analysis to group SPAs of a similar nature. We then derived 2 general and 11 cluster-specific propositions on how different material properties of SPAs can yield different affordances for value co-creation. With our work, we point out that smart services require researchers and practitioners to fundamentally rethink value co-creation as well as revise affordances theory to address the dynamic nature of smart technology as a service counterpart