Towards responsive Sensitive Artificial Listeners

This paper describes work in the recently started project SEMAINE, which aims to build a set of Sensitive Artificial Listeners – conversational agents designed to sustain an interaction with a human user despite limited verbal skills, through robust recognition and generation of non-verbal behaviour in real-time, both when the agent is speaking and listening. We report on data collection and on the design of a system architecture in view of real-time responsiveness

Emotional Biosensing: Exploring Critical Alternatives

Emotional biosensing is rising in daily life: Data and categories claim to know how people feel and suggest what they should do about it, while CSCW explores new biosensing possibilities. Prevalent approaches to emotional biosensing are too limited, focusing on the individual, optimization, and normative categorization. Conceptual shifts can help explore alternatives: toward materiality, from representation toward performativity, inter-action to intra-action, shifting biopolitics, and shifting affect/desire. We contribute (1) synthesizing wide-ranging conceptual lenses, providing analysis connecting them to emotional biosensing design, (2) analyzing selected design exemplars to apply these lenses to design research, and (3) offering our own recommendations for designers and design researchers. In particular we suggest humility in knowledge claims with emotional biosensing, prioritizing care and affirmation over self- improvement, and exploring alternative desires. We call for critically questioning and generatively re- imagining the role of data in configuring sensing, feeling, ‘the good life,’ and everyday experience

Detecting depression in dyadic conversations with multimodal narratives and visualizations

Conversations contain a wide spectrum of multimodal information that gives us hints about the emotions and moods of the speaker. In this paper, we developed a system that supports humans to analyze conversations. Our main contribution is the identification of appropriate multimodal features and the integration of such features into verbatim conversation transcripts. We demonstrate the ability of our system to take in a wide range of multimodal information and automatically generated a prediction score for the depression state of the individual. Our experiments showed that this approach yielded better performance than the baseline model. Furthermore, the multimodal narrative approach makes it easy to integrate learnings from other disciplines, such as conversational analysis and psychology. Lastly, this interdisciplinary and automated approach is a step towards emulating how practitioners record the course of treatment as well as emulating how conversational analysts have been analyzing conversations by hand.Comment: 12 page

Interpersonal affect in groupwork: a comparative case study of two small groups with contrasting group dynamics outcomes

Teamwork capabilities are essential for 21st century life, with groupwork emerging as a fruitful context to develop these skills. Case studies that explore interpersonal affect dynamics in authentic higher education groupwork settings can highlight collaborative skills development needs. This comparative case-study traced the sociodynamic evolution of two groups of first-year university students to investigate the high collaborative variance outcomes of the two groups, which reported starkly contrasting group dynamics (negative and dysfunctional or positive and collaborative). Mixed-methods (video-recorded observations of five groupwork labs over one semester, and group interviews) provided interpersonal affect data as real-time visible behaviours, and the felt experiences and perceptions of  participants. The study traced interpersonal affect dynamics in the natural fluctuation of not just task-focused (on-task), but also explicitly relational (off-task) interactions, which revealed their function in both task participation and group dynamics. Findings illustrate visible interpersonal affect behaviours that manifested and evolved over time as interactive patterns, and group dynamics outcomes. Fine-grained analysis of interactions unveiled interpersonal affect as a collective, evolving process, and the mechanism through which one group started and stayed highly positive and collaborative over the semester. The other group showed a tendency towards splitting to undertake tasks early, leading to low group-level interpersonal attentiveness, and over time, subgroups emerged through interactions both off-task and on-task. The study made visible the pervasive nature of interpersonal affect as enacted through seemingly inconsequential everyday behaviours that supported the relational and task-based needs of groupwork, and those behaviours which impeded collaboration.

Interpersonal affect in groupwork: A comparative case study of two small groups with contrasting group dynamics outcomes

Teamwork capabilities are essential for 21st century life, with groupwork emerging as a fruitful context to develop these skills. Case studies that explore interpersonal affect dynamics in authentic higher education groupwork settings can highlight collaborative skills development needs. This comparative case-study traced the sociodynamic evolution of two groups of first-year university students to investigate the high collaborative variance outcomes of the two groups, which reported starkly contrasting group dynamics (negative and dysfunctional or positive and collaborative). Mixed-methods (video-recorded observations of five groupwork labs over one semester, and group interviews) provided interpersonal affect data as real-time visible behaviours, and the felt experiences and perceptions of participants. The study traced interpersonal affect dynamics in the natural fluctuation of not just task-focused (on-task), but also explicitly relational (off-task) interactions, which revealed their function in both task participation and group dynamics. Findings illustrate visible interpersonal affect behaviours that manifested and evolved over time as interactive patterns, and group dynamics outcomes. Fine-grained analysis of interactions unveiled interpersonal affect as a collective, evolving process, and the mechanism through which one group started and stayed highly positive and collaborative over the semester. The other group showed a tendency towards splitting to undertake tasks early, leading to low group-level interpersonal attentiveness, and over time, subgroups emerged through interactions both off-task and on-task. The study made visible the pervasive nature of interpersonal affect as enacted through seemingly inconsequential everyday behaviours that supported the relational and task-based needs of groupwork, and those behaviours which impeded collaboration

A Framework for Situation-based Social Interaction

This paper presents a theoretical framework for computationally representing social situations in a robot. This work is based on interdependence theory, a social psychological theory of interaction and social situation analysis. We use interdependence theory to garner information about the social situations involving a human and a robot. We also quantify the gain in outcome resulting from situation analysis. Experiments demonstrate the utility of social situation information and of our situation-based framework as a method for guiding robot interaction. We conclude that this framework offers a principled, general approach for studying interactive robotics problems

Visitor engagement and learning behaviour in science centres, zoos and aquaria

The purpose of this research was to devise an assessment tool to effectively capture the nature of visitors' learning experiences with live animal exhibits in zoos and aquaria. A comprehensive learning framework was developed and field-tested with a total of 900 visitor. The resulting framework provides researchers and practitioners in zoos and aquaria with a valuable tool to assess the learning impact of exhibits through observable behavioural indicators

Improvable objects and attached dialogue: new literacy practices employed by learners to build knowledge together in asynchronous settings

Asynchronous online dialogue offers advantages to learners, but has appeared to involve only limited use of new literacy practices. To investigate this, a multimodal approach was applied to asynchronous dialogue. The study analysed the online discussions of small groups of university students as they developed collaboratively authored documents. Sociocultural discourse analysis of the dialogue was combined with visual analysis of its structural elements. The groups were found to employ new literacies that supported the joint construction of knowledge. The documents on which they worked together functioned as ‘improvable objects’ and the development of these was associated with engagement in ‘attached dialogue’. By investigating a wider range of conference dialogue than has previously been explored, it was found that engaging in attached dialogue associated with collaborative authorship of improvable objects prompts groups of online learners to share knowledge, challenge ideas, justify opinions, evaluate evidence and consider options