Measuring, analysing and artificially generating head nodding signals in dyadic social interaction

Social interaction involves rich and complex behaviours where verbal and non-verbal signals are exchanged in dynamic patterns. The aim of this thesis is to explore new ways of measuring and analysing interpersonal coordination as it naturally occurs in social interactions. Specifically, we want to understand what different types of head nods mean in different social contexts, how they are used during face-to-face dyadic conversation, and if they relate to memory and learning. Many current methods are limited by time-consuming and low-resolution data, which cannot capture the full richness of a dyadic social interaction. This thesis explores ways to demonstrate how high-resolution data in this area can give new insights into the study of social interaction. Furthermore, we also want to demonstrate the benefit of using virtual reality to artificially generate interpersonal coordination to test our hypotheses about the meaning of head nodding as a communicative signal. The first study aims to capture two patterns of head nodding signals – fast nods and slow nods – and determine what they mean and how they are used across different conversational contexts. We find that fast nodding signals receiving new information and has a different meaning than slow nods. The second study aims to investigate a link between memory and head nodding behaviour. This exploratory study provided initial hints that there might be a relationship, though further analyses were less clear. In the third study, we aim to test if interactive head nodding in virtual agents can be used to measure how much we like the virtual agent, and whether we learn better from virtual agents that we like. We find no causal link between memory performance and interactivity. In the fourth study, we perform a cross-experimental analysis of how the level of interactivity in different contexts (i.e., real, virtual, and video), impacts on memory and find clear differences between them

Life is an Adventure! An agent-based reconciliation of narrative and scientific worldviews\ud

The scientific worldview is based on laws, which are supposed to be certain, objective, and independent of time and context. The narrative worldview found in literature, myth and religion, is based on stories, which relate the events experienced by a subject in a particular context with an uncertain outcome. This paper argues that the concept of “agent”, supported by the theories of evolution, cybernetics and complex adaptive systems, allows us to reconcile scientific and narrative perspectives. An agent follows a course of action through its environment with the aim of maximizing its fitness. Navigation along that course combines the strategies of regulation, exploitation and exploration, but needs to cope with often-unforeseen diversions. These can be positive (affordances, opportunities), negative (disturbances, dangers) or neutral (surprises). The resulting sequence of encounters and actions can be conceptualized as an adventure. Thus, the agent appears to play the role of the hero in a tale of challenge and mystery that is very similar to the "monomyth", the basic storyline that underlies all myths and fairy tales according to Campbell [1949]. This narrative dynamics is driven forward in particular by the alternation between prospect (the ability to foresee diversions) and mystery (the possibility of achieving an as yet absent prospect), two aspects of the environment that are particularly attractive to agents. This dynamics generalizes the scientific notion of a deterministic trajectory by introducing a variable “horizon of knowability”: the agent is never fully certain of its further course, but can anticipate depending on its degree of prospect

Visual perspective and the characteristics of mind wandering

Peer reviewedPublisher PD

Audiovisual Learning in Dyslexic and Typical Adults: Modulating Influences of Location and Context Consistency

Learning to read involves efficient binding of visual to auditory information. Aberrant cross-modal binding skill has been observed in both children and adults with developmental dyslexia. Here, we examine the contribution of episodic memory to acquisition of novel cross-modal bindings in typical and dyslexic adult readers. Participants gradually learned arbitrary associations between unfamiliar Mandarin Chinese characters and English-like pseudowords over multiple exposures, simulating the early stages of letter-to-letter sound mapping. The novel cross-modal bindings were presented in consistent or varied locations (i.e., screen positions), and within consistent or varied contexts (i.e., co-occurring distractor items). Our goal was to examine the contribution, if any, of these episodic memory cues (i.e., the contextual and spatial properties of the stimuli) to binding acquisition, and investigate the extent to which readers with and without dyslexia would differ in their reliance on episodic memory during the learning process. Participants were tested on their ability to recognize and recall the bindings both during training and then in post-training tasks. We tracked participants’ eye movements remotely with their personal webcams to assess whether they would re-fixate relevant empty screen locations upon hearing an auditory cue—indicative of episodic memory retrieval—and the extent to which the so-called “looking-at-nothing behavior” would modulate recognition of the novel bindings. Readers with dyslexia both recognized and recalled significantly fewer bindings than typical readers, providing further evidence of their persistent difficulties with cross-modal binding. Looking-at-nothing behavior was generally associated with higher recognition error rates for both groups, a pattern that was particularly more evident in later blocks for bindings encoded in the inconsistent location condition. Our findings also show that whilst readers with and without dyslexia are capable of using stimulus consistencies in the input—both location and context—to assist in audiovisual learning, readers with dyslexia appear particularly reliant on consistent contextual information. Taken together, our results suggest that whilst readers with dyslexia fail to efficiently learn audiovisual binding as a function of stimulus frequency, they are able to use stimulus consistency—aided by episodic recall—to assist in the learning process

Improving the Security of Mobile Devices Through Multi-Dimensional and Analog Authentication

Mobile devices are ubiquitous in today\u27s society, and the usage of these devices for secure tasks like corporate email, banking, and stock trading grows by the day. The first, and often only, defense against attackers who get physical access to the device is the lock screen: the authentication task required to gain access to the device. To date mobile devices have languished under insecure authentication scheme offerings like PINs, Pattern Unlock, and biometrics-- or slow offerings like alphanumeric passwords. This work addresses the design and creation of five proof-of-concept authentication schemes that seek to increase the security of mobile authentication without compromising memorability or usability. These proof-of-concept schemes demonstrate the concept of Multi-Dimensional Authentication, a method of using data from unrelated dimensions of information, and the concept of Analog Authentication, a method utilizing continuous rather than discrete information. Security analysis will show that these schemes can be designed to exceed the security strength of alphanumeric passwords, resist shoulder-surfing in all but the worst-case scenarios, and offer significantly fewer hotspots than existing approaches. Usability analysis, including data collected from user studies in each of the five schemes, will show promising results for entry times, in some cases on-par with existing PIN or Pattern Unlock approaches, and comparable qualitative ratings with existing approaches. Memorability results will demonstrate that the psychological advantages utilized by these schemes can lead to real-world improvements in recall, in some instances leading to near-perfect recall after two weeks, significantly exceeding the recall rates of similarly secure alphanumeric passwords

Making Them Remember-Emotional Virtual Characters with Memory

The search for the perfect virtual character is on, but the moment users interact with characters, any illusion that we’ve found it is broken. Adding memory capabilities to models of human emotions, personality, and behavior traits is a step toward a more natural interaction style

Lifelong Augmentation of Multi-Modal Streaming Autobiographical Memories

Robot systems that interact with humans over extended periods of time will benefit from storing and recalling large amounts of accumulated sensorimotor and interaction data. We provide a principled framework for the cumulative organisation of streaming autobiographical data so that data can be continuously processed and augmented as the processing and reasoning abilities of the agent develop and further interactions with humans take place. As an example, we show how a kinematic structure learning algorithm reasons a-posteriori about the skeleton of a human hand. A partner can be asked to provide feedback about the augmented memories, which can in turn be supplied to the reasoning processes in order to adapt their parameters. We employ active, multi-modal remembering, so the robot as well as humans can gain insights of both the original and augmented memories. Our framework is capable of storing discrete and continuous data in real-time. The data can cover multiple modalities and several layers of abstraction (e.g. from raw sound signals over sentences to extracted meanings). We show a typical interaction with a human partner using an iCub humanoid robot. The framework is implemented in a platform-independent manner. In particular, we validate its multi platform capabilities using the iCub, Baxter and NAO robots. We also provide an interface to cloud based services, which allow automatic annotation of episodes. Our framework is geared towards the developmental robotics community, as it 1) provides a variety of interfaces for other modules, 2) unifies previous works on autobiographical memory, and 3) is licensed as open source software

Points of Reflection: A Case for Moral Engagement Across Video Game Time and Space

In the field of video game studies, meaningful action and flow are upheld as primary targets of game design, and key factors in many ontological definitions of what games can and should be. Yet, games are not all action. Within most games one encounters numerous pauses and interruptions of various kinds, including the much-maligned “cut-scenes” that lead or force the player out of an active role at certain moments. Furthermore, not all actions are goal-directed. If they are, they are not necessarily pragmatic. These pauses, interruptions, and nuanced goals are often overlooked, if not actively derided, in the field of game studies. In short, ideas about how players stop and reflect, how their goals and experiences take on emotional and/or moral valences, are under-represented. My work argues that moral reflection does occur even in mainstream games, and that it tends to happen in connection with the very moments game scholars often overlook—in the pauses before or after actions, in the moments of awe or realization, when the controller has been set down or the keyboard pushed away, and yes, even during cut-scenes. Such moments may invite the player into a state of moral reflection, but for this state of moral reflection to be poignant and memorable to the player these moments must also involve a consideration of differing values. Finally, how a game structures the player’s experience of time, from receiving quests to setting out into the game world, from pauses to demanding challenges, and even through the layout of video game spaces lends these points of reflection their crucial impact